I'm trying to run a command using snakemake and a TSV config file. Here is my sample config file:
sample_id FQ1_id FQ2_id
MyCleanName-1 MyFqName-1_R1.fastq.gz MyFqName-1_R2.fastq.gz
MyCleanName-2 MyFqName-2_R1.fastq.gz MyFqName-2_R2.fastq.gz
Here is my snakefile:
import pandas as pd
samples = pd.read_table("samples.tsv").set_index("sample_id", drop = False)
rule all:
input:
expand("{sample}_{num}.fastq.gz", sample = samples.sample_id, num = [1,2])
rule my_job:
input:
fq1 = expand("{fq1_id}", fq1_id = samples.FQ1_id),
fq2 = expand("{fq2_id}", fq2_id = samples.FQ2_id),
logdir = expand("{sample}.logs", sample = samples.sample_id)
output:
"{sample}_1.fastq.gz",
"{sample}_2.fastq.gz",
message:
"Running MyCmd on {wildcards.sample}"
shell:
"""
MyCmd --sample-name {wildcards.sample} -i1 {input.fq1} -i2 {input.fq2}
"""
rule create_logdir:
output:
directory("{sample}.logs")
message:
"Creating log directories"
shell:
"mkdir -pv {wildcards.sample}.logs/{{MyCmd,MyOtherCmd}}"
When I run snakemake -np MyCleanName-1.fastq.gz
, I get the following:
Building DAG of jobs...
Job counts:
count jobs
2 create_logdir
1 my_job
3
[Fri Dec 24 16:38:54 2021]
Job 1: Creating log directories
mkdir -pv MyCleanName-1.logs/{MyCmd,MyOtherCmd}
[Fri Dec 24 16:38:54 2021]
Job 2: Creating log directories
mkdir -pv MyCleanName-2.logs/{MyCmd,MyOtherCmd}
[Fri Dec 24 16:38:54 2021]
Job 0: Running MyCmd on MDA-IBC-3
MyCmd --sample-name MyCleanName-1 -i1 MyFqName-1_R1.fastq.gz MyFqName-2_R1.fastq.gz -i2 MyFqName-1_R2.fastq.gz MyFqName-2_R2.fastq.gz
Job counts:
count jobs
2 create_logdir
1 my_job
3
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
How can I restrict this so the run only creates log files for the Clean Name being used and uses only the corresponding FASTQs? I feel like I'm missing something trivial here. Should I not be using the config file variable samples
directly in the rule my_job
, but somehow wiring up the corresponding sample_id
and FQ_id
s on the all
rule?
expand
is a Snakemake help function that tends to create confusion for beginners. The goal is to generate lists of file names from patterns, usually to generate the input of a "gathering" / "summarizing" rule. If you want a single file name, I would recommend to use standard Python (>=3.6) f-strings. $\endgroup$