# Prevent Snakemake from changing wildcards: sample for every rule

EDIT: I posted this as a bug, perhaps wait for the result of that? github

All,

I am using Snakemake to build a pipeline. To define input and output names, I constructed them based on a filename base (from the samples table), a read indicator (like R1, R2) and the fastq file extension, like so:

samples = pd.read_csv('samples.tsv', sep='\t').set_index('samples', drop=False)
fastq_file_extension = _001.fastq.gz

rule fastqc:
input:
output:
wrapper:
"0.49.0/bio/fastqc"


However, during pipeline execution already I can see strange things happening, for this rule it reports:

[Fri Feb 21 12:39:57 2020]
rule fastqc:
input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz
output: qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.html, qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.zip
jobid: 10


As you can see, Snakemake pulled the R part of the read to the sample and set the read to 1 (should be R1). But this still works because it produces output files as Snakemake expects.

But the next rule, trim_galore_pe:

rule trim_galore_pe:
input:
output:
params:
extra="--retain_unpaired"
wrapper:
"0.49.0/bio/trim_galore/pe"


This goes wrong, during pipeline execution Snakemake reports:

[Fri Feb 21 12:40:37 2020]
Error in rule trim_galore_pe:
jobid: 7
output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
conda-env: /home/nlv24077/projects/snaqs/snakemake_envs/1b94430b
cluster_jobid: 11904662.osios
Error executing rule trim_galore_pe on cluster (jobid: 7, external: 11904662.osios, jobscript: /home/nlv24077/temp/RNASeqPairedEnd/.snakemake/tmp.ooypuklc/snakejob.trim_galore_pe.7.sh).
For error details see the cluster log and the log files of the involved rule(s).


This is understandeble because the file fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz does not exist, it should be fastq_trimmed/0053_P2017BB3S20R_S2.1_val_1.fq.gz, somehow Snakemake added an extra underscore, which you can see in the execution output, under "wildcards" (0053_P2017BB3S20R_S2_).

[Fri Feb 21 12:39:57 2020]
rule trim_galore_pe:
input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz, fastq_raw/0053_P2017BB3S20R_S2_R2_001.fastq.gz
output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
jobid: 7
wildcards: sample=0053_P2017BB3S20R_S2_


So how can I prevent Snakemake from changing the samples and read wildcard in seemingly arbitrary ways?

Placing two wildcards directly behind each other will make it impossible for snakemake to determine where one wildcard ends and the other begins. So it is simply taking the longest possible match for the first wildcard (sample) and then one character for the second (read). This will also cause the _ to be included in the wildcard value sample.

I'm assuming that this same pattern is used for the inputs of another rule to refer to the trim_galore_pe output. Hence why the _ is included in the wildcards there.

You might be able to fix this, simply by adding a _ in-between {sample} and {readgroup} in your first rule (and any other rules using this same pattern):

rule fastqc:
input:
output:
wrapper:
"0.49.0/bio/fastqc"

• I did this but Snakemake conjures up a new _ out of nowhere: Building DAG of jobs... MissingInputException in line 2 of /rst1/temp//RNASeqPairedEnd/rules/trim_galore_pe.smk: Missing input files for rule trim_galore_pe: fastq_raw/0053_P2017BB3S19R_S1__R1_001.fastq.gz fastq_raw/0053_P2017BB3S19R_S1__R2_001.fastq.gz Feb 26 '20 at 7:37
• But why is is impossible for Snakemake to separate them? They are not in {} for nothing right? I don't understand why Snakemake would want to change my wildcards from anything it gets from the samples table. What would be the use? It seems to me that this is just a bug? Or am I really fundamentally misunderstanding Snakemake? Feb 26 '20 at 7:40
• Regarding my first comment, one might think that Snakemake now added and extra _ so I can remove the existing one perhaps? But no, it then removes both: Building DAG of jobs... MissingInputException in line 2 of /rst1/temp/RNASeqPairedEnd/rules/trim_galore_pe.smk: Missing input files for rule trim_galore_pe: fastq_raw/0053_P2017BB3S19R_S1R2_001.fastq.gz fastq_raw/0053_P2017BB3S19R_S1R1_001.fastq.gz So it's either 0 or 2, not 1, as one would expect. Feb 26 '20 at 7:46
• So I guess I am misunderstanding this completely, When I change the input of trim_galore_pe, it complains of missing files, but when I remove the input entirely it complains that the fastqc rule is missing input files, even though I did not touch that rule. Is there any way to more clearly see where errors originate exactly? I feel completely lost now. Feb 26 '20 at 8:01
• @Freek Could you share the entire Snakefile? It's quite hard to see what's going on exactly without it. Feb 26 '20 at 8:18

I think you want to prevent snakemake from interpreting your wildcard strings as regular expressions. For this, escape each wildcard value using wildcard_constraints. E.g., towards the top of your Snakefile add:

wildcard_constraints:

(Personally, I use the wildcard_constraints directive as above quite liberally to constrain every wildcard)