1
$\begingroup$

I am trying to run Fastqc on multiple files using Snakemake but am receiving an ambiguous SyntaxError past the last line of code in the Snakefile.

Snakefile:

import pandas as pd

SAMPLE_TABLE=pd.read_csv('./SampleTable.txt',sep='\t')
SAMPLES=SAMPLE_TABLE["sample_name"]
SAMPLES=list(SAMPLES)
DIRECTORY=list(SAMPLE_TABLE["FolderName"])

rule run_fastqc:
    input:
        expand("/PathToInputsFolder/{dir}/{sample}_{{Rnumber}}_001.fastq.gz", zip, dir=DIRECTORY, sample=SAMPLES)
    output:
        expand("{sample}_{{Rnumber}}_001.fastqc.html", sample=SAMPLES)
        expand("{sample}_{{Rnumber}}_001.fastqc.zip", sample=SAMPLES)
    shell:
        "fastqc {input} --outdir=./qc/"

Code:

snakemake --cores 1 {.+}.fastqc.html {.+}.fastqc.zip

Output:

SyntaxError in line 16 of /PathToSnakefile/Snakefile:
invalid syntax (Snakefile, line 16)

How should I figure out where the error is? Thank you for your time!

$\endgroup$
2
$\begingroup$

How to figure out what error messages are trying to tell you is one of the great mysteries… :-)

What version of Snakemake are you using? I get slightly different messages. With an ancient version (3.13.3) on one machine:

SyntaxError in line 14 of /path/to/Snakefile:
invalid syntax

With a more up-to-date version (6.9.1, running on Python 3.9):

SyntaxError in line 16 of /path/to/Snakefile:
invalid syntax. Perhaps you forgot a comma? (Snakefile, line 16)

Both of these are useful hints. So one piece of advice would be to update your Snakemake and/or Python if it is old.

Another useful technique when faced with a mysterious error like this is to temporarily remove parts of your Snakefile until the error goes away. The complicated part of your script towards the end of the file is the output: stanza, so you might try removing one of the expand(…) lines and seeing what happens. If the error goes away, then you have localised the problem and can put the line back and carefully check the syntax in that particular part of the Snakefile.

$\endgroup$
1
$\begingroup$

You're missing a comma between the two output lines. (John Marshall's more detailed syntax error message was spot-on.)

With a simplified version of your Snakefile:

SAMPLES=["s1", "s2", "s3"]
rule run_fastqc:
    output:
        expand("{sample}_{{Rnumber}}_001.fastqc.html", sample=SAMPLES)
        expand("{sample}_{{Rnumber}}_001.fastqc.zip", sample=SAMPLES)
    shell:
        "echo {output}"

You get the syntax error:

$ snakemake -nrp s1_R1_001.fastqc.html
SyntaxError in line 6 of /data/home/jesse/Snakefile:
invalid syntax (Snakefile, line 6)

With the comma:

SAMPLES=["s1", "s2", "s3"]
rule run_fastqc:
    output:
        expand("{sample}_{{Rnumber}}_001.fastqc.html", sample=SAMPLES), # <-- here
        expand("{sample}_{{Rnumber}}_001.fastqc.zip", sample=SAMPLES)
    shell:
        "echo {output}"

Now it's OK:

$ snakemake -nrp s1_R1_001.fastqc.html
Building DAG of jobs...
Job stats:
job           count    min threads    max threads
----------  -------  -------------  -------------
run_fastqc        1              1              1
total             1              1              1


[Fri Nov 12 08:54:54 2021]
rule run_fastqc:
    output: s1_R1_001.fastqc.html, s2_R1_001.fastqc.html, s3_R1_001.fastqc.html, s1_R1_001.fastqc.zip, s2_R1_001.fastqc.zip, s3_R1_001.fastqc.zip
    jobid: 0
    reason: Missing output files: s1_R1_001.fastqc.html
    wildcards: Rnumber=R1
    resources: tmpdir=/tmp

echo s1_R1_001.fastqc.html s2_R1_001.fastqc.html s3_R1_001.fastqc.html s1_R1_001.fastqc.zip s2_R1_001.fastqc.zip s3_R1_001.fastqc.zip
Job stats:
job           count    min threads    max threads
----------  -------  -------------  -------------
run_fastqc        1              1              1
total             1              1              1

This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

Each expand() makes a list, so at first Snakemake sees two lists back-to-back and doesn't know what to do with them. If you use a comma and give output names you could handle the two lists separately in your rule, but if you don't need to, you could also just us a single expand call, like:

output:
    expand("{sample}_{{Rnumber}}_001.fastqc.{ext}", sample=SAMPLES, ext=["html", "zip"])

Other suggestions for troubleshooting:

  • If the above simplified Snakefile works but the real one still doesn't, edit them incrementally to be more and more similar and you'll hone in on whatever problems are left.
  • I find trimming away complexity until the problem reveals itself is often easiest, so I'd keep simplifying the real case until that happens.
  • expand is just a Python function that Snakemake provides you, so you can call it yourself outside of a rule definition and check what it returns.
  • You can even comment out all the Snakemake-specific syntax and leave just Python code when debugging, to sanity-check the rest of the code (e.g. your inputs and outputs from expand).
$\endgroup$
3
  • $\begingroup$ You can give someone a fish [a comma] today… or you can teach them to fish [to decrypt their own error messages]… $\endgroup$ Nov 12 at 14:31
  • $\begingroup$ With the suggested comma, I get the message: "WorkflowError: Duplicate output file pattern in rule run_fastqc. First two duplicate for entries 0 and 1 File "/PathToSnakefile/Snakefile", line 22, in <module>". I'm not sure how the output files are duplicate if one is .html and the other is .zip. I also added 'READ=list(SAMPLE_TABLE["read"])' and changed '{{Rnumber}}' to 'R{val}, val=READ' so the wildcard is defined, and it still leads to the duplicate output error :/ $\endgroup$
    – Swietenia
    Nov 12 at 23:13
  • $\begingroup$ I added a few bullets on the end of my answer that might help you. To John Marshall's point, Snakemake is correctly telling you the problem here, but you're assuming the output files are what you expect them to be. If you investigate the expand behavior in your new version you'll see the issue. $\endgroup$
    – Jesse
    Nov 16 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.