4
$\begingroup$

I have generated several FASTQ files and I would like to know the amount of reads for each of them. I am planning to run FastQC on the files which I know would give me the number of reads per sample but it would be a lot easier for me if I could get one text file with the file name and number of reads for all samples together.

When I check for one file I normally do zcat filename_R1.fastq.gz | wc -l and then divide by 4. Is there a way to do it for all my files at once and get a text file with the file name and reads at the end?

For example I will have 71 R1.fastq.gz files and I would like to output the amount of reads to a text file. I just meant that I would prefer to use one command instead of having to use the command I mentioned above for each file.

$\endgroup$
2
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Feb 3 at 22:26
  • $\begingroup$ Note that fastq files can have more than 1 line of sequence per entry so dividing by 4 is not robust. Make sure to check your file and see if you have cases of multi-line sequences. $\endgroup$
    – terdon
    Feb 6 at 10:26

4 Answers 4

5
$\begingroup$

One way would be to use seqkit stats, which can be installed using conda:

conda install -c bioconda seqkit

It has a number of useful options, including ways to parallelize the counting, using -j, --threads (default 4), and to skip errors using -e, --skip-err. The latter can be especially useful if you have a larger number of FASTQ files with millions of reads and you'd like to run the command overnight for example. There's also an option to produce a machine-friendly tabular output, using -T, --tabular which would allow you to parse the output without much effort. I'd be tempted to just write the output to a file using the -o, --out-file option:

seqkit stats -To stats.tsv *.fastq.gz

Example results:

$ cat stats.tsv 
file    format  type    num_seqs    sum_len min_len avg_len max_len
PHB_Rep1_R1.fastq.gz    FASTQ   RNA 118571  11857100    100 100.0   100
PHB_Rep1_R2.fastq.gz    FASTQ   RNA 118571  11857100    100 100.0   100
PHB_Rep2_R1.fastq.gz    FASTQ   RNA 144826  14482600    100 100.0   100
PHB_Rep2_R2.fastq.gz    FASTQ   RNA 144826  14482600    100 100.0   100
PHB_Rep3_R1.fastq.gz    FASTQ   RNA 129786  12978600    100 100.0   100
PHB_Rep3_R2.fastq.gz    FASTQ   RNA 129786  12978600    100 100.0   100
PUH_Rep1_R1.fastq.gz    FASTQ   RNA 227392  22739200    100 100.0   100
PUH_Rep1_R2.fastq.gz    FASTQ   RNA 227392  22739200    100 100.0   100
PUH_Rep2_R1.fastq.gz    FASTQ   RNA 162373  16237300    100 100.0   100
PUH_Rep2_R2.fastq.gz    FASTQ   RNA 162373  16237300    100 100.0   100
PUH_Rep3_R1.fastq.gz    FASTQ   RNA 185442  18544200    100 100.0   100
PUH_Rep3_R2.fastq.gz    FASTQ   RNA 185442  18544200    100 100.0   100

And then just parse the output to get what you need, if or when you need it. For your specific requirements, you could just use AWK to skip the header line and print the first and fourth columns:

awk 'BEGIN {FS=OFS="\t" } NR>1 { print $1, $4 }' stats.tsv

Results:

PHB_Rep1_R1.fastq.gz    118571
PHB_Rep1_R2.fastq.gz    118571
PHB_Rep2_R1.fastq.gz    144826
PHB_Rep2_R2.fastq.gz    144826
PHB_Rep3_R1.fastq.gz    129786
PHB_Rep3_R2.fastq.gz    129786
PUH_Rep1_R1.fastq.gz    227392
PUH_Rep1_R2.fastq.gz    227392
PUH_Rep2_R1.fastq.gz    162373
PUH_Rep2_R2.fastq.gz    162373
PUH_Rep3_R1.fastq.gz    185442
PUH_Rep3_R2.fastq.gz    185442
$\endgroup$
1
$\begingroup$

You can replace gunzip -c with zcat.

#USAGE: script.sh /runFolder
for i in $1/*.1.fastq.gz
do
   echo "$i" $(gunzip -c $i | echo `wc -l`/4 | bc -l)
done

#OR in the current folder, do this:
for i in ./*.1.fastq.gz
do
   echo "$i" $(gunzip -c $i | echo `wc -l`/4 | bc -l)
done
$\endgroup$
2
  • 1
    $\begingroup$ Backticks are legacy syntax. Subshells can be nested for example using: echo "${i}: $(echo "scale=2; $(zcat "${i}" | wc -l)/4" | bc)" $\endgroup$
    – Steve
    Feb 4 at 2:53
  • $\begingroup$ Also, I think the OP is looking for a summary text file, so you could just have your loop append to an output file (i.e. ... >> counts.txt) $\endgroup$
    – Steve
    Feb 4 at 2:59
0
$\begingroup$

zcat, like cat, works with multiple file names as arguments, up to the argument limit (on Linux, that's usually at least a thousand), so counting multiple lines in multiple files still works, assuming the fastq files are formatted in the expected way with only four lines per record:

zcat filename_R1.fastq.gz filename_R2.fastq.gz ... | wc -l

If there's a consistent file pattern, that can be used with file name glob wildcards:

zcat *.fastq.gz | wc -l

If you have a file containing a list of files, this can be processed as shell arguments as well:

zcat $(cat list_of_files.txt) | wc -l

For an unlimited number of files, something like GNU parallel can be used to feed file names from piped input. It's also a bit safer for file names containing weird characters in them:

cat list_of_files.txt | parallel zcat {} | wc -l

... but [after checking the question again] given that you want counts for individual files, the file name needs to be added as well:

cat list_of_files.txt | parallel echo {} $(zcat {} | wc -l)

The read function provides an alternate way to do the same thing:

cat list_of_files.txt | while read fileName; do echo ${fileName} $(zcat ${fileName} | wc -l); done

Assuming you are using bash, dividing by 4 can be done via the arithmetic substitution operator, $((X)):

cat list_of_files.txt | parallel echo {} $(( $(zcat {} | wc -l) / 4 ))

Or with read:

cat list_of_files.txt | while read fileName; do echo ${fileName} $(( $(zcat ${fileName} | wc -l) / 4 )); done

And, finally, this can be output to a file using >:

(cat list_of_files.txt | while read fileName; do echo ${fileName} $(( $(zcat ${fileName} | wc -l) / 4 )); done) > fastq_record_counts.txt
$\endgroup$
3
  • 1
    $\begingroup$ You'll need to wrap your commands in single quotes to prevent the dollar variables from expanding. You can also avoid the Useless Use of Cat (UUoC) by redirecting stdin or using the :::: argfiles syntax with GNU parallel. To be safe, remember to always quote your variables. $\endgroup$
    – Steve
    Feb 6 at 3:17
  • $\begingroup$ For echo? I don't think it'll make any difference $\endgroup$
    – gringer
    Feb 6 at 11:48
  • $\begingroup$ Sorry, I mean the parallel commands need to be singled quoted: parallel 'echo {} $(( $(zcat {} | wc -l) / 4 ))' :::: list_of_files.txt $\endgroup$
    – Steve
    Feb 6 at 12:31
-4
$\begingroup$

If you're looking for a one-liner,

for i in $(ls *.fastq.gz); do count=$(zcat $i | grep "^@" | wc -l); printf "%b\t%b" $i $count; done > counts.tsv
$\endgroup$
2
  • 3
    $\begingroup$ Note that fastq files can include '@' in the first character of the quality string, so this may produce incorrect results. $\endgroup$
    – gringer
    Feb 5 at 19:58
  • 1
    $\begingroup$ Also, NEVER do for i in $(ls *), that is very bad practice, will break on even slightly odd file names and is really not needed since a simple for i in *.fastq.gz; is safer, more robust and shorter to write. $\endgroup$
    – terdon
    Feb 6 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.