0
$\begingroup$

EDIT: I posted this as a bug, perhaps wait for the result of that? github

All,

I am using Snakemake to build a pipeline. To define input and output names, I constructed them based on a filename base (from the samples table), a read indicator (like R1, R2) and the fastq file extension, like so:

samples = pd.read_csv('samples.tsv', sep='\t').set_index('samples', drop=False)
reads = ['R1', 'R2']
fastq_file_extension = _001.fastq.gz
rule fastqc:
    input:
        os.path.join(config['Directories']['fastq_raw_dir'], '{sample}{read}' + fastq_file_extension)
    output:
        html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
        zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
    wrapper:
        "0.49.0/bio/fastqc"

However, during pipeline execution already I can see strange things happening, for this rule it reports:

[Fri Feb 21 12:39:57 2020]
rule fastqc:
input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz
output: qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.html, qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.zip
jobid: 10
wildcards: sample=0053_P2017BB3S20R_S2_R, read=1

As you can see, Snakemake pulled the R part of the read to the sample and set the read to 1 (should be R1). But this still works because it produces output files as Snakemake expects.

But the next rule, trim_galore_pe:

rule trim_galore_pe:
    input:
        [os.path.join(config['Directories']['fastq_raw_dir'], '{sample}' + read + fastq_file_extension) for read in reads]
    output:
        fastq_read1 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.1_val_1.fq.gz'),
        fastq_read2 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.2_val_2.fq.gz'),
        report_read1 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.1.fastq.gz_trimming_report.txt'),
        report_read2 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.2.fastq.gz_trimming_report.txt')
    params:
        extra="--retain_unpaired"
    wrapper:
        "0.49.0/bio/trim_galore/pe"

This goes wrong, during pipeline execution Snakemake reports:

[Fri Feb 21 12:40:37 2020]
Error in rule trim_galore_pe:
jobid: 7
output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
conda-env: /home/nlv24077/projects/snaqs/snakemake_envs/1b94430b
cluster_jobid: 11904662.osios
Error executing rule trim_galore_pe on cluster (jobid: 7, external: 11904662.osios, jobscript: /home/nlv24077/temp/RNASeqPairedEnd/.snakemake/tmp.ooypuklc/snakejob.trim_galore_pe.7.sh).
For error details see the cluster log and the log files of the involved rule(s).

This is understandeble because the file fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz does not exist, it should be fastq_trimmed/0053_P2017BB3S20R_S2.1_val_1.fq.gz, somehow Snakemake added an extra underscore, which you can see in the execution output, under "wildcards" (0053_P2017BB3S20R_S2_).

[Fri Feb 21 12:39:57 2020]
rule trim_galore_pe:
input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz, fastq_raw/0053_P2017BB3S20R_S2_R2_001.fastq.gz
output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
jobid: 7
wildcards: sample=0053_P2017BB3S20R_S2_

So how can I prevent Snakemake from changing the samples and read wildcard in seemingly arbitrary ways?

$\endgroup$
0
$\begingroup$

Placing two wildcards directly behind each other will make it impossible for snakemake to determine where one wildcard ends and the other begins. So it is simply taking the longest possible match for the first wildcard (sample) and then one character for the second (read). This will also cause the _ to be included in the wildcard value sample.

I'm assuming that this same pattern is used for the inputs of another rule to refer to the trim_galore_pe output. Hence why the _ is included in the wildcards there.

You might be able to fix this, simply by adding a _ in-between {sample} and {readgroup} in your first rule (and any other rules using this same pattern):

rule fastqc:
    input:
        os.path.join(config['Directories']['fastq_raw_dir'], '{sample}_{read}' + fastq_file_extension)
    output:
        html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
        zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
    wrapper:
        "0.49.0/bio/fastqc"
| improve this answer | |
$\endgroup$
  • $\begingroup$ I did this but Snakemake conjures up a new _ out of nowhere: Building DAG of jobs... MissingInputException in line 2 of /rst1/temp//RNASeqPairedEnd/rules/trim_galore_pe.smk: Missing input files for rule trim_galore_pe: fastq_raw/0053_P2017BB3S19R_S1__R1_001.fastq.gz fastq_raw/0053_P2017BB3S19R_S1__R2_001.fastq.gz $\endgroup$ – Freek Feb 26 at 7:37
  • $\begingroup$ But why is is impossible for Snakemake to separate them? They are not in {} for nothing right? I don't understand why Snakemake would want to change my wildcards from anything it gets from the samples table. What would be the use? It seems to me that this is just a bug? Or am I really fundamentally misunderstanding Snakemake? $\endgroup$ – Freek Feb 26 at 7:40
  • $\begingroup$ Regarding my first comment, one might think that Snakemake now added and extra _ so I can remove the existing one perhaps? But no, it then removes both: Building DAG of jobs... MissingInputException in line 2 of /rst1/temp/RNASeqPairedEnd/rules/trim_galore_pe.smk: Missing input files for rule trim_galore_pe: fastq_raw/0053_P2017BB3S19R_S1R2_001.fastq.gz fastq_raw/0053_P2017BB3S19R_S1R1_001.fastq.gz So it's either 0 or 2, not 1, as one would expect. $\endgroup$ – Freek Feb 26 at 7:46
  • $\begingroup$ So I guess I am misunderstanding this completely, When I change the input of trim_galore_pe, it complains of missing files, but when I remove the input entirely it complains that the fastqc rule is missing input files, even though I did not touch that rule. Is there any way to more clearly see where errors originate exactly? I feel completely lost now. $\endgroup$ – Freek Feb 26 at 8:01
  • $\begingroup$ @Freek Could you share the entire Snakefile? It's quite hard to see what's going on exactly without it. $\endgroup$ – DavyCats Feb 26 at 8:18
0
$\begingroup$

I think you want to prevent snakemake from interpreting your wildcard strings as regular expressions. For this, escape each wildcard value using wildcard_constraints. E.g., towards the top of your Snakefile add:

wildcard_constraints:
    read= '|'.join([re.escape(x) for x in reads]),
    sample= '|'.join([re.escape(x) for x in samples])

(Personally, I use the wildcard_constraints directive as above quite liberally to constrain every wildcard)

| improve this answer | |
$\endgroup$
  • $\begingroup$ Does this mess with how you can use rules.fastqc.output.html as the input for other rules? It gives me a lot of errors for a downstream step (multiqc). $\endgroup$ – Freek Feb 26 at 7:39
  • $\begingroup$ As far as I know, this doesn't cause problems. If you don't show the error it's difficult to tell what's going on. $\endgroup$ – dariober Feb 26 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.