'Wildcards' object has no attribute 'sample'

Question

I'm using Snakemake and I'm trying to get all my QC reports into multiqc but I get the following error:

WorkflowError in line 120 of /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full/Snakefile:
'Wildcards' object has no attribute 'sample'

Rule 120 is the start of my multiqc rule which looks like this:

rule multiqc:
    input:
        expand([os.path.join(analyzed_dir, '{sample}.stat', '{sample}.cnt'),
                os.path.join(rseqc_dir, '{sample}.bam_stat.txt'),
                os.path.join(rseqc_dir, '{sample}.clipping_profile.xls'),
                os.path.join(rseqc_dir, '{sample}.deletion_profile.txt'),
                os.path.join(rseqc_dir, '{sample}.infer_experiment.txt'),
                os.path.join(rseqc_dir, '{sample}.geneBodyCoverage.txt'),
                os.path.join(rseqc_dir, '{sample}.inner_distance.txt'),
                os.path.join(rseqc_dir, '{sample}.insertion_profile.xls'),
                os.path.join(rseqc_dir, '{sample}.junction.xls'),
                os.path.join(rseqc_dir, '{sample}.junctionSaturation_plot.r'),
                os.path.join(rseqc_dir, '{sample}.mismatch_profile.xls'),
                os.path.join(rseqc_dir, '{sample}.read_distribution.txt'),
                os.path.join(rseqc_dir, '{sample}.pos.DupRate.xls'),
                os.path.join(rseqc_dir, '{sample}.seq.DupRate.xls'),
                os.path.join(rseqc_dir, '{sample}.GC.xls'),
                os.path.join(rseqc_dir, '{sample}.NVC.xls'),
                os.path.join(rseqc_dir, '{sample}.qual.r'),
                os.path.join(rseqc_dir, '{sample}.RNA_fragment_size.txt'),
                os.path.join(rseqc_dir, '{sample}.STAR.genome.sorted.summary.txt'),
                os.path.join(fastqc_dir, '{sample}_R{read}_001_fastqc.html')],
                sample=samples['samples'], read=reads)
    output:
        os.path.join(qc_dir, 'multiqc_report.html') 
    conda:
        "envs/multiqc.yaml"
    shell:
        '''
        #!/bin/bash
        multiqc . --outdir qc --ignore .snakemake --force
        '''

My "rule all looks like this:"

rule all:
    input:
        expand([os.path.join(analyzed_dir, '{sample}.genes.results'),
                os.path.join(qc_dir, 'multiqc_report.html')],
                sample=samples['samples'])

I don't see anything wrong with it, in fact, it is very similar to my "rule all" I used before and which worked. Can anyone see an error?

EDIT

I limited the rule to only one input, I still have the same error:

rule multiqc:
    input:
        expand(os.path.join(rseqc_dir, '{sample}.geneBodyCoverage.txt'),
                sample=samples['samples'])
    output:
        os.path.join(qc_dir, 'multiqc_report.html') 
    conda:
        "envs/multiqc.yaml"
    shell:
        '''
        #!/bin/bash
        multiqc . --outdir qc --ignore .snakemake --force
        '''

This is the ouput, you can see it read the samples correctly:

run_snakemake.sh 
/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/site-packages/snakemake/workflow.py:12: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  from collections import OrderedDict
Building DAG of jobs...
/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/site-packages/snakemake/workflow.py:12: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  from collections import OrderedDict
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 256
Job counts:
    count   jobs
    1   all
    1   multiqc
    2

[Mon Jan 14 17:46:40 2019]
rule multiqc:
    input: qc/rseqc/0067_P2018SEQE27R01_S1.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R02_S2.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R03_S3.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R04_S4.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R05_S5.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R06_S6.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R08_S8.geneBodyCoverage.txt
    output: qc/multiqc_report.html
    jobid: 2

WorkflowError in line 119 of /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full/Snakefile:
'Wildcards' object has no attribute 'sample'
17:46 nlv24077@kiato /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full > ll qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt
-rw-rw-r-- 1 nlv24077 2017-0205_illuminaseq 303 Jan  7 16:00 qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt

EDIT 2:

This is how I start Snakemake:

> cat run_snakemake.sh
source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
snakemake --dag | dot -T svg > dag.svg
snakemake --cluster-config cluster.json --cluster "qsub -l nodes={cluster.nodes}:ppn={cluster.ppn} -N {wildcards.sample}_{rule}" --jobs 256 --use-conda --rerun-incomplete

EDIT 3:

Ok, that last remark (by @finswimmer) triggered me, I removed the "{wildcards.sample}_" from the above command and then it works. Or, works, I get a very different error, it is multiqc specific and solved as specified here: https://multiqc.info/docs/#locale-error-messages. When adding the two lines as suggested this error disappears and everything works as expected, even with the long inputs list.

RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/en/7.x/python3/ for mitigation steps.

Anyway, I'm guessing Snakemake does not understand how to name this task when it uses an entire list of {sample}'s... Is this a bug? What should be used? I guess it is impossible to use {sample} for naming for this rule. It's a shame I have to kill the naming for all rules because using sample_rule is very nice for keeping track.

Answer 1

Snakemake gets wildcards from the parsing the input/output file names. For example, if you had

rule sort:
    input: "{file}.bed"
    output: "{file}.sorted.bed"
    shell: "sort -k1,1 -k2,2n {input} > {output}"

then the wildcards variable would be set as wildcards = {"file": ...}.

In your multiqc rule, you have the {sample} variable within your expand(), but this variable doesn't exist outside of expand's local context. You don't actually have any wildcards there, nor in your output file (qc_dir and rseqc_dir are variables you define elsewhere, but not "wildcards" as Snakemake interprets them).

So you're getting the error 'Wildcards' object has no attribute 'sample' because there are no wildcards for that rule.

Your rule is currently set up to process every sample that you have listed in samples['sample']. If you want your multiqc rule to work for a single sample, simply remove the sample=samples['sample'] and double brace the {sample} wildcard.

rule multiqc:
    input:
        expand([os.path.join(analyzed_dir, '{{sample}}.stat', '{{sample}}.cnt'),
                os.path.join(rseqc_dir, '{{sample}}.bam_stat.txt'),
                os.path.join(rseqc_dir, '{{sample}}.clipping_profile.xls'),
                os.path.join(rseqc_dir, '{{sample}}.deletion_profile.txt'),
                os.path.join(rseqc_dir, '{{sample}}.infer_experiment.txt'),
                os.path.join(rseqc_dir, '{{sample}}.geneBodyCoverage.txt'),
                os.path.join(rseqc_dir, '{{sample}}.inner_distance.txt'),
                os.path.join(rseqc_dir, '{{sample}}.insertion_profile.xls'),
                os.path.join(rseqc_dir, '{{sample}}.junction.xls'),
                os.path.join(rseqc_dir, '{{sample}}.junctionSaturation_plot.r'),
                os.path.join(rseqc_dir, '{{sample}}.mismatch_profile.xls'),
                os.path.join(rseqc_dir, '{{sample}}.read_distribution.txt'),
                os.path.join(rseqc_dir, '{{sample}}.pos.DupRate.xls'),
                os.path.join(rseqc_dir, '{{sample}}.seq.DupRate.xls'),
                os.path.join(rseqc_dir, '{{sample}}.GC.xls'),
                os.path.join(rseqc_dir, '{{sample}}.NVC.xls'),
                os.path.join(rseqc_dir, '{{sample}}.qual.r'),
                os.path.join(rseqc_dir, '{{sample}}.RNA_fragment_size.txt'),
                os.path.join(rseqc_dir, '{{sample}}.STAR.genome.sorted.summary.txt'),
                os.path.join(fastqc_dir, '{{sample}}_R{read}_001_fastqc.html')],
                read=reads)
    output:
        # this {sample} is in single {} and is recognized as a wildcard
        os.path.join(qc_dir, '{sample}', 'multiqc_report.html') 
    conda:
        "envs/multiqc.yaml"
    shell:
        '''
        #!/bin/bash
        # using {wildcards.sample} here since MultiQC likes directories
        multiqc . --outdir qc/{wildcards.sample} --ignore .snakemake --force
        '''

When wildcards are recognized, you'll see them in the job list.

Running snakemake 0067_P2018SEQE27R01_S1/multiqc_report.html, for example, should give you something like:

rule multiqc:
    input: qc/rseqc/0067_P2018SEQE27R01_S1.geneBodyCoverage.txt, ...
    output: qc/0067_P2018SEQE27R01_S1/multiqc_report.html
    jobid: 2
    wildcards: sample=0067_P2018SEQE27R01_S1

Answer 2

Although @JamesHawley answer was informative, I don't think it solved the initial issue requested by the OP. (edit: or maybe it partially did, I'm not sure as I got a bit confused with the wildcards on the initial question/answer. I'm anyway leaving this answer here, in case another way of explaining it is useful for others)

Solution

I think you only need to change the output of your rule multiqc from:

os.path.join(qc_dir, 'multiqc_report.html')

to

os.path.join(qc_dir, '{sample}.html')

This seemed counter-intuitive to me initially, but I think what's happening is that snakemake will look at rule all and see that you want an output called "qc/multiqc_report.html". And the only place where it can find something of that sort is in rule multiqc, by replacing the wildcard {sample} by the string multiqc_report.

It felt confusing to me, because the same {sample} wildcard is used in other rules (namely the rules you use to generate the outputs in your analysed_dir). But snakemake will be smart to replace {sample} with whatever it needs to recreate the output files you're requesting.

---

Longer explanation

Here is a (hopefully) reproducible example, if it helps.

First create some fake directory structure with some files:

mkdir inputs
mkdir analysed_dir
mkdir qc_dir

touch inputs/file1.txt
touch inputs/file2.txt

Then, this would be the snakefile:

# Define your variables, fetch things from config, or csv, etc...
input_dir = "inputs"
output_dir = "outputs"
samples = ["file1.txt", "file2.txt"]


# Rule all defining all the output files required
rule all:
  input:
    expand("analysed_dir/{samples}", samples = samples),
    "qc_dir/multi_qc_report.html"

# this rule is equivalent to whichever rules process your data individually
rule analyse_each_file:
  input:
    "inputs/{sample}.txt"
  output:
    "analysed_dir/{sample}.txt"
  shell:
    "cat {input} > {output}"

# This rule is equivalent to your multiqc step where multiple files result in a single output
rule multiqc:
  input:
    expand("analysed_dir/{samples}", samples = samples)
  output:
    "qc_dir/{sample}.html"
  shell:
    "cat {input} > {output}"

to generate this pipeline:

And then issuing:

snakemake --cluster "qsub -N {wildcards.sample}-{rule}" --jobs 10

Generates the following job names:

• file1-analyse_each_file
• file2-analyse_each_file
• multi_qc_report-multiqc

Edit with further explanation:

Notice in the DAG how the sample wildcard takes different values for the different rules. I think the snakemake logic is as follows:

• Starting with rule all I see that you want three files as input (the first two come from the expand() call):
  • analysed_dir/file1.txt
  • analysed_dir/file2.txt
  • qc_dir/multi_qc_report.html
• Let me start with the first one analysed_dir/file1.txt. The file doesn't exist, so let me look in the other rules and see if I can find it in their output. Actually in rule analyse_each_file I see the output is set to "analysed_dir/{sample}.txt", so if I replace {sample} by file1.txt I can recreate your file from it. Therefore, the input for that rule will be "inputs/file1.txt" (and this file already exists, so I can issue this job).
• For the second file analysed_dir/file2.txt, well it's a similar case as with the previous one, so I'll issue a job with the same rule analyse_each_file, but this time {sample} = file2.txt.
• Finally, you want a file called qc_dir/multi_qc_report.html. Where can I create such a file from? Well, the output of rule multiqc is set to "qc_dir/{sample}.html", so if I make {sample} = multi_qc_report, then I can issue the job and produce your file of interest. In this case, the rule takes as input two files "analysed_dir/file1.txt" and "analysed_dir/file2.txt", which I already know how to create from another rule.

The key thing to realise is that snakemake defines the wildcard {sample} for each rule separately. So, in this example, wildcard.sample in rule analyse_each_file does not take on the same values as in rule multiqc.