If we take real world gzipped FASTQ (which as the OP suggested would be beneficial) rather than trivial FASTA as the starting point then the real issue is actually decompressing the file not counting the Ns and in this case the C program count-N is no longer the fastest solution.
Additionally it would good to use the a specific file for benchmarking which actually has Ns, because you'll get some quite interesting execution time differences with some methods counting the more frequently occurring As rather than Ns.
One such file is:
http://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/HG00096/sequence_read/SRR077487_2.filt.fastq.gz
It's also worth checking the various solutions return the correct answer there should be 306072 Ns in the above file.
Next note that decompression of this file redirected to /dev/null
is slower with zcat
and gzip
(which are both gzip 1.6 on my system) than say a parallel implementation of gzip like Mark Adler's pigz
, which appears to use 4 threads for decompression. All timings represent an average of 10 runs reporting real (wall clock time).
time pigz -dc SRR077487_2.filt.fastq.gz > /dev/null
real 0m29.0132s
time gzip -dc SRR077487_2.filt.fastq.gz > /dev/null
real 0m40.6996s
There is an ~11.7 second difference between the two. Next if I then try to benchmark a one-liner which performs on FASTQ and gives the correct answer (Note I've yet to encounter FASTQ which is not 4 line, and seriously who generates these files!)
time pigz -dc SRR077487_2.filt.fastq.gz | awk 'NR%4==2{print $1}' | tr -cd N | wc -c
306072
real 0m34.793s
As you can see the counting adds ~5.8 second to the total run time versus pigz
based decompression. Additionally this time delta is higher when using gzip
~6.7 seconds above gzip decompression alone.
time gzip -dc SRR077487_2.filt.fastq.gz | awk 'NR%4==2{print $1}' | tr -cd N | wc -c
306072
real 0m44.399s
The pigz awk tr wc
based solution is however ~4.5 seconds faster than the count-N
based C code solution:
time count-N SRR077487_2.filt.fastq.gz
2385855128 306072 0
real 0m39.266s
This difference appears to be robust to re-running as many times as I like. I expect if you could use pthread in the C based solution or alter it to take the standard out from pigz it would also show an increase in performance.
Benchmarking another alternative pigz grep
variant appears to take more or less the same time as the tr
based variant:
time pigz -dc SRR077487_2.filt.fastq.gz | awk 'NR%4==2{print $1}' | grep -o N | wc -l
306072
real 0m34.869s
Note that the seqtk
based solution discussed above is noticeably slower,
time seqtk comp SRR077487_2.filt.fastq.gz | awk '{x+=$9}END{print x}'
306072
real 1m42.062s
However it's worth noting seqtk comp
is doing a bit more than the other solutions.