As the first step in my Snakemake pipeline, I generate small fastq files from all input fastq.gz files. This is to run a quick round of "infer_experiment.py" (from RSeQC, to get the kit strandedness). However, Snakemake says this step fails. I think I know why, it is because a command like this

zcat fastq_raw/20190011-043_S43_R1_001.fastq.gz | head -n 400000 > infer_strandedness/20190011-043_S43_subset_R1.fastq

Ends with an error: gzip: stdout: Broken pipe and Snakemake uses Bash strict mode. (Snakemake reports: (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)). So my question is:

How to avoid this, or, are there other truncating methods that do finish cleanly? Preferebly they are also efficient (seqtk offers subsampling but this parses the entire fastq file!).

  • 1
    $\begingroup$ You're better off using seqtk even if it's slower since you're then not performing a highly biased sampling from the beginning of the file, where the quality tends to be worse. $\endgroup$ – Devon Ryan Mar 31 at 17:23
  • $\begingroup$ But the library is randomized during prep and flowing into the chip, why would the first part of the fastq file (first scanned tile of the chip) be biased? A bit worse quality would not be a problem if I only want to detect the strandedness... $\endgroup$ – Freek Apr 2 at 7:19
  • $\begingroup$ The beginning of the files represent the edge of the flow cell where the quality is lower. $\endgroup$ – Devon Ryan Apr 2 at 7:20

Snakemake is using bash strict mode to ensure best practice error reporting in shell commands. Unfortunately some tools violate bash strict mode.

More details of the error, here. Alternatively, you can use if you don't want disable the strict mode.

zcat file.fastq.gz  | awk '(NR<=400000)' > subset.fastq
| improve this answer | |
  • $\begingroup$ This solves my problem exactly (the awk solution), it is fast and does what my original command does but without pipefail error. Thank you. $\endgroup$ – Freek Apr 2 at 7:42
  • $\begingroup$ @Freek another option would be to add || true to give a 0 exit status $\endgroup$ – Chris_Rands Apr 2 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.