How to get a file with the number of reads for several fastq.gz files?

Question

I have generated several FASTQ files and I would like to know the amount of reads for each of them. I am planning to run FastQC on the files which I know would give me the number of reads per sample but it would be a lot easier for me if I could get one text file with the file name and number of reads for all samples together.

When I check for one file I normally do zcat filename_R1.fastq.gz | wc -l and then divide by 4. Is there a way to do it for all my files at once and get a text file with the file name and reads at the end?

For example I will have 71 R1.fastq.gz files and I would like to output the amount of reads to a text file. I just meant that I would prefer to use one command instead of having to use the command I mentioned above for each file.

Answer 1

One way would be to use seqkit stats, which can be installed using conda:

conda install -c bioconda seqkit

It has a number of useful options, including ways to parallelize the counting, using -j, --threads (default 4), and to skip errors using -e, --skip-err. The latter can be especially useful if you have a larger number of FASTQ files with millions of reads and you'd like to run the command overnight for example. There's also an option to produce a machine-friendly tabular output, using -T, --tabular which would allow you to parse the output without much effort. I'd be tempted to just write the output to a file using the -o, --out-file option:

seqkit stats -To stats.tsv *.fastq.gz

Example results:

$ cat stats.tsv 
file    format  type    num_seqs    sum_len min_len avg_len max_len
PHB_Rep1_R1.fastq.gz    FASTQ   RNA 118571  11857100    100 100.0   100
PHB_Rep1_R2.fastq.gz    FASTQ   RNA 118571  11857100    100 100.0   100
PHB_Rep2_R1.fastq.gz    FASTQ   RNA 144826  14482600    100 100.0   100
PHB_Rep2_R2.fastq.gz    FASTQ   RNA 144826  14482600    100 100.0   100
PHB_Rep3_R1.fastq.gz    FASTQ   RNA 129786  12978600    100 100.0   100
PHB_Rep3_R2.fastq.gz    FASTQ   RNA 129786  12978600    100 100.0   100
PUH_Rep1_R1.fastq.gz    FASTQ   RNA 227392  22739200    100 100.0   100
PUH_Rep1_R2.fastq.gz    FASTQ   RNA 227392  22739200    100 100.0   100
PUH_Rep2_R1.fastq.gz    FASTQ   RNA 162373  16237300    100 100.0   100
PUH_Rep2_R2.fastq.gz    FASTQ   RNA 162373  16237300    100 100.0   100
PUH_Rep3_R1.fastq.gz    FASTQ   RNA 185442  18544200    100 100.0   100
PUH_Rep3_R2.fastq.gz    FASTQ   RNA 185442  18544200    100 100.0   100

And then just parse the output to get what you need, if or when you need it. For your specific requirements, you could just use AWK to skip the header line and print the first and fourth columns:

awk 'BEGIN {FS=OFS="\t" } NR>1 { print $1, $4 }' stats.tsv

Results:

PHB_Rep1_R1.fastq.gz    118571
PHB_Rep1_R2.fastq.gz    118571
PHB_Rep2_R1.fastq.gz    144826
PHB_Rep2_R2.fastq.gz    144826
PHB_Rep3_R1.fastq.gz    129786
PHB_Rep3_R2.fastq.gz    129786
PUH_Rep1_R1.fastq.gz    227392
PUH_Rep1_R2.fastq.gz    227392
PUH_Rep2_R1.fastq.gz    162373
PUH_Rep2_R2.fastq.gz    162373
PUH_Rep3_R1.fastq.gz    185442
PUH_Rep3_R2.fastq.gz    185442

Answer 2

You can replace gunzip -c with zcat.

#USAGE: script.sh /runFolder
for i in $1/*.1.fastq.gz
do
   echo "$i" $(gunzip -c $i | echo `wc -l`/4 | bc -l)
done

#OR in the current folder, do this:
for i in ./*.1.fastq.gz
do
   echo "$i" $(gunzip -c $i | echo `wc -l`/4 | bc -l)
done

Answer 3

zcat, like cat, works with multiple file names as arguments, up to the argument limit (on Linux, that's usually at least a thousand), so counting multiple lines in multiple files still works, assuming the fastq files are formatted in the expected way with only four lines per record:

zcat filename_R1.fastq.gz filename_R2.fastq.gz ... | wc -l

If there's a consistent file pattern, that can be used with file name glob wildcards:

zcat *.fastq.gz | wc -l

If you have a file containing a list of files, this can be processed as shell arguments as well:

zcat $(cat list_of_files.txt) | wc -l

For an unlimited number of files, something like GNU parallel can be used to feed file names from piped input. It's also a bit safer for file names containing weird characters in them:

cat list_of_files.txt | parallel zcat {} | wc -l

... but [after checking the question again] given that you want counts for individual files, the file name needs to be added as well:

cat list_of_files.txt | parallel echo {} $(zcat {} | wc -l)

The read function provides an alternate way to do the same thing:

cat list_of_files.txt | while read fileName; do echo ${fileName} $(zcat ${fileName} | wc -l); done

Assuming you are using bash, dividing by 4 can be done via the arithmetic substitution operator, $((X)):

cat list_of_files.txt | parallel echo {} $(( $(zcat {} | wc -l) / 4 ))

Or with read:

cat list_of_files.txt | while read fileName; do echo ${fileName} $(( $(zcat ${fileName} | wc -l) / 4 )); done

And, finally, this can be output to a file using >:

(cat list_of_files.txt | while read fileName; do echo ${fileName} $(( $(zcat ${fileName} | wc -l) / 4 )); done) > fastq_record_counts.txt

Answer 4

If you're looking for a one-liner,

for i in $(ls *.fastq.gz); do count=$(zcat $i | grep "^@" | wc -l); printf "%b\t%b" $i $count; done > counts.tsv