3
$\begingroup$

All,

As the first step in my Snakemake pipeline, I generate small fastq files from all input fastq.gz files. This is to run a quick round of "infer_experiment.py" (from RSeQC, to get the kit strandedness). However, Snakemake says this step fails. I think I know why, it is because a command like this

zcat fastq_raw/20190011-043_S43_R1_001.fastq.gz | head -n 400000 > infer_strandedness/20190011-043_S43_subset_R1.fastq

Ends with an error: gzip: stdout: Broken pipe and Snakemake uses Bash strict mode. (Snakemake reports: (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)). So my question is:

How to avoid this, or, are there other truncating methods that do finish cleanly? Preferebly they are also efficient (seqtk offers subsampling but this parses the entire fastq file!).

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ You're better off using seqtk even if it's slower since you're then not performing a highly biased sampling from the beginning of the file, where the quality tends to be worse. $\endgroup$
    – Devon Ryan
    Mar 31, 2020 at 17:23
  • $\begingroup$ But the library is randomized during prep and flowing into the chip, why would the first part of the fastq file (first scanned tile of the chip) be biased? A bit worse quality would not be a problem if I only want to detect the strandedness... $\endgroup$
    – Freek
    Apr 2, 2020 at 7:19
  • $\begingroup$ The beginning of the files represent the edge of the flow cell where the quality is lower. $\endgroup$
    – Devon Ryan
    Apr 2, 2020 at 7:20

1 Answer 1

Reset to default
1
$\begingroup$

Snakemake is using bash strict mode to ensure best practice error reporting in shell commands. Unfortunately some tools violate bash strict mode.

More details of the error, here. Alternatively, you can use if you don't want disable the strict mode.

zcat file.fastq.gz  | awk '(NR<=400000)' > subset.fastq
Improve this answer
$\endgroup$
2
  • $\begingroup$ This solves my problem exactly (the awk solution), it is fast and does what my original command does but without pipefail error. Thank you. $\endgroup$
    – Freek
    Apr 2, 2020 at 7:42
  • $\begingroup$ @Freek another option would be to add || true to give a 0 exit status $\endgroup$
    – Chris_Rands
    Apr 2, 2020 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.