3
$\begingroup$

I'm using Snakemake and I'm trying to get all my QC reports into multiqc but I get the following error:

WorkflowError in line 120 of /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full/Snakefile:
'Wildcards' object has no attribute 'sample'

Rule 120 is the start of my multiqc rule which looks like this:

rule multiqc:
    input:
        expand([os.path.join(analyzed_dir, '{sample}.stat', '{sample}.cnt'),
                os.path.join(rseqc_dir, '{sample}.bam_stat.txt'),
                os.path.join(rseqc_dir, '{sample}.clipping_profile.xls'),
                os.path.join(rseqc_dir, '{sample}.deletion_profile.txt'),
                os.path.join(rseqc_dir, '{sample}.infer_experiment.txt'),
                os.path.join(rseqc_dir, '{sample}.geneBodyCoverage.txt'),
                os.path.join(rseqc_dir, '{sample}.inner_distance.txt'),
                os.path.join(rseqc_dir, '{sample}.insertion_profile.xls'),
                os.path.join(rseqc_dir, '{sample}.junction.xls'),
                os.path.join(rseqc_dir, '{sample}.junctionSaturation_plot.r'),
                os.path.join(rseqc_dir, '{sample}.mismatch_profile.xls'),
                os.path.join(rseqc_dir, '{sample}.read_distribution.txt'),
                os.path.join(rseqc_dir, '{sample}.pos.DupRate.xls'),
                os.path.join(rseqc_dir, '{sample}.seq.DupRate.xls'),
                os.path.join(rseqc_dir, '{sample}.GC.xls'),
                os.path.join(rseqc_dir, '{sample}.NVC.xls'),
                os.path.join(rseqc_dir, '{sample}.qual.r'),
                os.path.join(rseqc_dir, '{sample}.RNA_fragment_size.txt'),
                os.path.join(rseqc_dir, '{sample}.STAR.genome.sorted.summary.txt'),
                os.path.join(fastqc_dir, '{sample}_R{read}_001_fastqc.html')],
                sample=samples['samples'], read=reads)
    output:
        os.path.join(qc_dir, 'multiqc_report.html') 
    conda:
        "envs/multiqc.yaml"
    shell:
        '''
        #!/bin/bash
        multiqc . --outdir qc --ignore .snakemake --force
        '''

My "rule all looks like this:"

rule all:
    input:
        expand([os.path.join(analyzed_dir, '{sample}.genes.results'),
                os.path.join(qc_dir, 'multiqc_report.html')],
                sample=samples['samples'])

I don't see anything wrong with it, in fact, it is very similar to my "rule all" I used before and which worked. Can anyone see an error?

EDIT

I limited the rule to only one input, I still have the same error:

rule multiqc:
    input:
        expand(os.path.join(rseqc_dir, '{sample}.geneBodyCoverage.txt'),
                sample=samples['samples'])
    output:
        os.path.join(qc_dir, 'multiqc_report.html') 
    conda:
        "envs/multiqc.yaml"
    shell:
        '''
        #!/bin/bash
        multiqc . --outdir qc --ignore .snakemake --force
        '''

This is the ouput, you can see it read the samples correctly:

run_snakemake.sh 
/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/site-packages/snakemake/workflow.py:12: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  from collections import OrderedDict
Building DAG of jobs...
/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/site-packages/snakemake/workflow.py:12: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  from collections import OrderedDict
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 256
Job counts:
    count   jobs
    1   all
    1   multiqc
    2

[Mon Jan 14 17:46:40 2019]
rule multiqc:
    input: qc/rseqc/0067_P2018SEQE27R01_S1.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R02_S2.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R03_S3.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R04_S4.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R05_S5.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R06_S6.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R08_S8.geneBodyCoverage.txt
    output: qc/multiqc_report.html
    jobid: 2

WorkflowError in line 119 of /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full/Snakefile:
'Wildcards' object has no attribute 'sample'
17:46 nlv24077@kiato /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full > ll qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt
-rw-rw-r-- 1 nlv24077 2017-0205_illuminaseq 303 Jan  7 16:00 qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt

EDIT 2:

This is how I start Snakemake:

> cat run_snakemake.sh
source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
snakemake --dag | dot -T svg > dag.svg
snakemake --cluster-config cluster.json --cluster "qsub -l nodes={cluster.nodes}:ppn={cluster.ppn} -N {wildcards.sample}_{rule}" --jobs 256 --use-conda --rerun-incomplete

EDIT 3:

Ok, that last remark (by @finswimmer) triggered me, I removed the "{wildcards.sample}_" from the above command and then it works. Or, works, I get a very different error, it is multiqc specific and solved as specified here: https://multiqc.info/docs/#locale-error-messages. When adding the two lines as suggested this error disappears and everything works as expected, even with the long inputs list.

RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/en/7.x/python3/ for mitigation steps.

Anyway, I'm guessing Snakemake does not understand how to name this task when it uses an entire list of {sample}'s... Is this a bug? What should be used? I guess it is impossible to use {sample} for naming for this rule. It's a shame I have to kill the naming for all rules because using sample_rule is very nice for keeping track.

$\endgroup$
9
  • 2
    $\begingroup$ Can you start by limiting the input to that rule to a couple files? That will at least help debug this. $\endgroup$
    – Devon Ryan
    Jan 14 '19 at 0:09
  • 1
    $\begingroup$ I don't know anything about snakemake but the problem seems to come from not declaring what a sample should be $\endgroup$
    – llrs
    Jan 14 '19 at 9:03
  • 1
    $\begingroup$ @llrs sample is defined in the expand() function. Oddly, there's no direct use of wildcards outside of that, which is usually where this sort of thing crops up. $\endgroup$
    – Devon Ryan
    Jan 14 '19 at 9:21
  • 3
    $\begingroup$ Is samples['samples'] defined somewhere? $\endgroup$
    – finswimmer
    Jan 14 '19 at 14:45
  • 2
    $\begingroup$ And how does your command look like to start the workflow? $\endgroup$
    – finswimmer
    Jan 14 '19 at 17:32
5
$\begingroup$

Snakemake gets wildcards from the parsing the input/output file names. For example, if you had

rule sort:
    input: "{file}.bed"
    output: "{file}.sorted.bed"
    shell: "sort -k1,1 -k2,2n {input} > {output}"

then the wildcards variable would be set as wildcards = {"file": ...}.

In your multiqc rule, you have the {sample} variable within your expand(), but this variable doesn't exist outside of expand's local context. You don't actually have any wildcards there, nor in your output file (qc_dir and rseqc_dir are variables you define elsewhere, but not "wildcards" as Snakemake interprets them).

So you're getting the error 'Wildcards' object has no attribute 'sample' because there are no wildcards for that rule.

Your rule is currently set up to process every sample that you have listed in samples['sample']. If you want your multiqc rule to work for a single sample, simply remove the sample=samples['sample'] and double brace the {sample} wildcard.

rule multiqc:
    input:
        expand([os.path.join(analyzed_dir, '{{sample}}.stat', '{{sample}}.cnt'),
                os.path.join(rseqc_dir, '{{sample}}.bam_stat.txt'),
                os.path.join(rseqc_dir, '{{sample}}.clipping_profile.xls'),
                os.path.join(rseqc_dir, '{{sample}}.deletion_profile.txt'),
                os.path.join(rseqc_dir, '{{sample}}.infer_experiment.txt'),
                os.path.join(rseqc_dir, '{{sample}}.geneBodyCoverage.txt'),
                os.path.join(rseqc_dir, '{{sample}}.inner_distance.txt'),
                os.path.join(rseqc_dir, '{{sample}}.insertion_profile.xls'),
                os.path.join(rseqc_dir, '{{sample}}.junction.xls'),
                os.path.join(rseqc_dir, '{{sample}}.junctionSaturation_plot.r'),
                os.path.join(rseqc_dir, '{{sample}}.mismatch_profile.xls'),
                os.path.join(rseqc_dir, '{{sample}}.read_distribution.txt'),
                os.path.join(rseqc_dir, '{{sample}}.pos.DupRate.xls'),
                os.path.join(rseqc_dir, '{{sample}}.seq.DupRate.xls'),
                os.path.join(rseqc_dir, '{{sample}}.GC.xls'),
                os.path.join(rseqc_dir, '{{sample}}.NVC.xls'),
                os.path.join(rseqc_dir, '{{sample}}.qual.r'),
                os.path.join(rseqc_dir, '{{sample}}.RNA_fragment_size.txt'),
                os.path.join(rseqc_dir, '{{sample}}.STAR.genome.sorted.summary.txt'),
                os.path.join(fastqc_dir, '{{sample}}_R{read}_001_fastqc.html')],
                read=reads)
    output:
        # this {sample} is in single {} and is recognized as a wildcard
        os.path.join(qc_dir, '{sample}', 'multiqc_report.html') 
    conda:
        "envs/multiqc.yaml"
    shell:
        '''
        #!/bin/bash
        # using {wildcards.sample} here since MultiQC likes directories
        multiqc . --outdir qc/{wildcards.sample} --ignore .snakemake --force
        '''

When wildcards are recognized, you'll see them in the job list.

Running snakemake 0067_P2018SEQE27R01_S1/multiqc_report.html, for example, should give you something like:

rule multiqc:
    input: qc/rseqc/0067_P2018SEQE27R01_S1.geneBodyCoverage.txt, ...
    output: qc/0067_P2018SEQE27R01_S1/multiqc_report.html
    jobid: 2
    wildcards: sample=0067_P2018SEQE27R01_S1
$\endgroup$
5
  • $\begingroup$ Thank you, nicely explained, I'll accept this answer. But, do you know of any way to temporarily set the wildcards.sample variable? So that I can use it for naming my multiqc job? I find it a shame that because of this one rule I can't have the {sample}_{rule} naming scheme anymore... $\endgroup$
    – Freek
    Jan 15 '19 at 20:02
  • $\begingroup$ Maybe you could try using multiple bash commands? Something like multiqc . --outdir qc --ignore .snakemake --force && mv multiqc_report.html {wildcards.sample}_multiqc.html? $\endgroup$ Jan 15 '19 at 22:59
  • 1
    $\begingroup$ But I think that because multiqc scans the entire directory, all your files are going to get scanned anyway, and end up in each report. So I don't know if they're actually going to be any different, or if you're just going to end up with the same file under multiple names $\endgroup$ Jan 15 '19 at 23:00
  • $\begingroup$ @JamesHawley, I thought the OP wanted many inputs (defined by the expand() call) to a single output (qc/multiqc_report.html), rather than your solution, which if I understand results in an output for every input. It should be possible to achieve what the OP asked for, and the multiqc wrapper seems to suggest so. In which case, what was wrong with the OP's code? (I'm having the same error in a similar situation, thus my comment) $\endgroup$
    – hugot
    Jan 13 '20 at 16:39
  • $\begingroup$ I got it now: the problem wasn't with the rule's code itself, it was to do with the call to --cluster. So, there is no known solution for this problem of wanting to use the wildcards.sample in the call to qsub when there is one, but otherwise not? This would be particularly useful for log files. $\endgroup$
    – hugot
    Jan 13 '20 at 16:58
0
$\begingroup$

Although @JamesHawley answer was informative, I don't think it solved the initial issue requested by the OP. (edit: or maybe it partially did, I'm not sure as I got a bit confused with the wildcards on the initial question/answer. I'm anyway leaving this answer here, in case another way of explaining it is useful for others)

Solution

I think you only need to change the output of your rule multiqc from:

os.path.join(qc_dir, 'multiqc_report.html')

to

os.path.join(qc_dir, '{sample}.html')

This seemed counter-intuitive to me initially, but I think what's happening is that snakemake will look at rule all and see that you want an output called "qc/multiqc_report.html". And the only place where it can find something of that sort is in rule multiqc, by replacing the wildcard {sample} by the string multiqc_report.

It felt confusing to me, because the same {sample} wildcard is used in other rules (namely the rules you use to generate the outputs in your analysed_dir). But snakemake will be smart to replace {sample} with whatever it needs to recreate the output files you're requesting.


Longer explanation

Here is a (hopefully) reproducible example, if it helps.

First create some fake directory structure with some files:

mkdir inputs
mkdir analysed_dir
mkdir qc_dir

touch inputs/file1.txt
touch inputs/file2.txt

Then, this would be the snakefile:

# Define your variables, fetch things from config, or csv, etc...
input_dir = "inputs"
output_dir = "outputs"
samples = ["file1.txt", "file2.txt"]


# Rule all defining all the output files required
rule all:
  input:
    expand("analysed_dir/{samples}", samples = samples),
    "qc_dir/multi_qc_report.html"

# this rule is equivalent to whichever rules process your data individually
rule analyse_each_file:
  input:
    "inputs/{sample}.txt"
  output:
    "analysed_dir/{sample}.txt"
  shell:
    "cat {input} > {output}"

# This rule is equivalent to your multiqc step where multiple files result in a single output
rule multiqc:
  input:
    expand("analysed_dir/{samples}", samples = samples)
  output:
    "qc_dir/{sample}.html"
  shell:
    "cat {input} > {output}"

to generate this pipeline:

dag

And then issuing:

snakemake --cluster "qsub -N {wildcards.sample}-{rule}" --jobs 10

Generates the following job names:

  • file1-analyse_each_file
  • file2-analyse_each_file
  • multi_qc_report-multiqc

Edit with further explanation:

Notice in the DAG how the sample wildcard takes different values for the different rules. I think the snakemake logic is as follows:

  • Starting with rule all I see that you want three files as input (the first two come from the expand() call):
    • analysed_dir/file1.txt
    • analysed_dir/file2.txt
    • qc_dir/multi_qc_report.html
  • Let me start with the first one analysed_dir/file1.txt. The file doesn't exist, so let me look in the other rules and see if I can find it in their output. Actually in rule analyse_each_file I see the output is set to "analysed_dir/{sample}.txt", so if I replace {sample} by file1.txt I can recreate your file from it. Therefore, the input for that rule will be "inputs/file1.txt" (and this file already exists, so I can issue this job).
  • For the second file analysed_dir/file2.txt, well it's a similar case as with the previous one, so I'll issue a job with the same rule analyse_each_file, but this time {sample} = file2.txt.
  • Finally, you want a file called qc_dir/multi_qc_report.html. Where can I create such a file from? Well, the output of rule multiqc is set to "qc_dir/{sample}.html", so if I make {sample} = multi_qc_report, then I can issue the job and produce your file of interest. In this case, the rule takes as input two files "analysed_dir/file1.txt" and "analysed_dir/file2.txt", which I already know how to create from another rule.

The key thing to realise is that snakemake defines the wildcard {sample} for each rule separately. So, in this example, wildcard.sample in rule analyse_each_file does not take on the same values as in rule multiqc.

$\endgroup$
2
  • $\begingroup$ Thank you for your comment, I still don't have a satisfying solution (meaning I now just leave out the sample wildcard from the qsub job names). For your solution I wonder, will Snakemake not complain about "qc_dir/{sample}.html" never being generated? I think the core issue is that the Multiqc rule takes in all samples so it is correct that there is no single "sample" wildcard. I was just looking for a way to set the sample wildcards to something like "all_samples" for the purpose of naming the qsubbed jobs. Am I making any sense? $\endgroup$
    – Freek
    Jan 15 '20 at 13:08
  • $\begingroup$ @Freek, when you say "I still don't have a satisfying solution", do you mean you tried my solution and it didn't work? (note, I've edited my solution to more correctly match your posted code using os.path.join() - see the Solution section at the top). I don't think snakemake will get confused, because it seems to me that wildcards are defined within each rule individually. See my further edit with expanded explanation at the end of the answer (below the DAG). I hope it helps. $\endgroup$
    – hugot
    Jan 16 '20 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.