I am trying to generate several snakemake rules. However, I would like to include some restrictions due to storage limitations
I first would like to copy the the bamfiles, 1 for each sample, to a staging area. However, while I have over 50 samples, I can only process three samples at a time.
Then I want to compare the bam file to chunked candidate files, 1 for each chromosome. This however, I would like to run on 23 cores per sample so that once each bam file is copied 69 cores are working.
finally, if all chunks for a given sample have been processed I want to clean up the staging area, copy the bamfile for the next sample and process the chunks for that sample.
My current implementation, see below, just continues copying bamfiles and crashes when I run out of strorage;
rule copy_bam:
input:
bamfile=os.path.join(config['bam_folder'], "{sample}.gatk.sorted.MD.bam"),
bamindex=os.path.join(config['bam_folder'], "{sample}.gatk.sorted.MD.bam.bai")
output:
staging_bamfile="{staging_area}/{sample}.bam",
staging_bamindex="{staging_area}/{sample}.bam.bai"
wildcard_constraints:
sample="[A-Za-z0-9_]+"
resources:
bam_copy=1
shell:
"""
cp {input.bamfile} {output.staging_bamfile}
cp {input.bamindex} {output.staging_bamindex}
"""
rule process_chunks:
input:
chunkfile="{staging_area}/chunks/{chunk}_with_header.bcf.gz",
bamfile=rules.copy_bam.output.staging_bamfile
output:
dynamic("{staging_area}/calls/{sample}/{chunk}.bcf")
threads: 23
shell:
"""
process chunks
"""
rule cleanup:
input:
bamfile="{staging_area}/{sample}.bam",
bamindex="{staging_area}/{sample}.bam.bai",
processed=lambda wildcards: expand("{staging_area}/calls/{sample}/{chunk}.bcf", chunk=get_chunks()[wildcards.sample], staging_area=config['staging_area'], sample=wildcards.sample)
output:
touch("{staging_area}/{sample}/cleanup_done.txt")
shell:
"""
rm {input.bamfile}
rm {input.bamindex}
rm {input.processed}
"""
run with: snakemake --resources bam_copy=3 --cores 70
temp()
should help, just note that I don't think the resource tracking even matters in that case (the resource is only "in use" whilecopy_bam
is actively running.) $\endgroup$