26
votes
Accepted
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
For FASTQ:
seqtk fqchk in.fq | head -2
It gives you percentage of "N" bases, not the exact count, though.
For FASTA:
...
- 6,605
23
votes
Accepted
What is the fastest way to get the reverse complement of a DNA sequence in python?
I don't know if it's the fastest, but the following provides an approximately 10x speed up over your functions:
...
- 19.8k
18
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
5 hours and no benchmarks posted? I am sorely disappointed.
I'll restrict the comparison to just be fasta files, since fastq will end up being the same. So far, the contenders are:
R with the ...
- 19.8k
14
votes
Accepted
Fast way to count number of reads and number of bases in a fastq file?
It's difficult to get this to go massively quicker I think - as with this question working with large gzipped FASTQ files is mostly IO-bound. We could instead focus on making sure we are getting the ...
- 861
11
votes
Random access on a FASTQ file
Arbitrary record access in constant time
To get a random record in constant time, it is sufficient to get an arbitrary record in constant time.
I have two solutions here: One with ...
- 2,286
10
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
I think that this should be pretty fast:
FASTA:
grep -v "^>" seqs.fa | tr -cd N | wc -c
FASTQ:
...
- 1,939
9
votes
What is the fastest way to get the reverse complement of a DNA sequence in python?
Here's a Cython approach that might suggest a generic approach to speeding up Python work.
If you're manipulating (ASCII) character strings and performance is a design consideration, then C or Perl ...
- 3,184
8
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
FASTQ
As it was pointed out, fastq can be complicated. But in a simple case when you have four lines per record, one possible solution in bash is:
...
- 2,735
8
votes
Accepted
Random access on a FASTQ file
As wkretzsch suggested this was worthy of an actual answer, I feel the obvious solution is missing here; index the FASTQ.
Index it
As much as I typically hesitate ...
- 812
8
votes
Samtools sort: most efficient memory and thread settings for many samples on a cluster
I usually use samtools sort -@4 -m4g. Samtools will be faster with more threads and more RAM, but I feel the gain is not significant. I haven't done a proper ...
- 6,605
8
votes
Accepted
Samtools sort: most efficient memory and thread settings for many samples on a cluster
Update 2
Ok, for what I expect will be my final update for this answer, I compared the following:
samtools 1.14 + zlib
...
- 1,414
7
votes
Fast way to count number of reads and number of bases in a fastq file?
The following is more than twice as fast; however, wc counts newline characters as well. We thus need to subtract the line count from the base count (using Bash):
<...
- 4,895
7
votes
Fast way to count number of reads and number of bases in a fastq file?
pigz | awk | wc is the fastest method
First off for benchmarks with FASTQ it's best to use a specific real-world example with a known answer. I've chosen this file:
ftp://ftp.1000genomes.ebi.ac.uk/...
- 1,069
7
votes
How do I efficiently subset a very large line-based file?
Turns out, simply keeping track of the next candidate line (after sorting the sample line numbers) fixes the performance issue, and most of the remaining slowness seems to be due to the overhead of ...
- 4,895
7
votes
What is the fastest way to get the reverse complement of a DNA sequence in python?
The most reliable and simplest way is probably using Biopython:
...
- 3,968
7
votes
Remove/delete sequences by ID from multifasta
Suppose you keep sequence names in ids.txt and sequences in seq.fa:
...
- 6,605
7
votes
Remove/delete sequences by ID from multifasta
Just today I wrote a script to do exactly this using Biopython. I also add a warning If any the headers I wanted to filter was not present in the original fasta. So ...
- 5,652
6
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
Decompression of gzipped FASTQ is the main issue
If we take real world gzipped FASTQ (which as the OP suggested would be beneficial) rather than trivial FASTA as the starting point then the real ...
- 1,069
6
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
Using bioawk
With bioawk (counting "A" in the C. elegans genome, because there seem to be no "N" in this file), on my computer:
...
- 3,195
6
votes
Remove/delete sequences by ID from multifasta
If you want to learn how to do things with command-line tools, you can linearize the FASTA with awk, pipe to grep to filter for ...
- 3,184
5
votes
Fast way to count number of reads and number of bases in a fastq file?
I get fairly quick results with my fastx-length.pl script, with the added bonus of being able to handle multi-line FASTQ files and displaying additional read-length QC statistics:
...
- 15.1k
5
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
If it is raw speed you're after, then writing an own little C/C++ program is probably what you need to do.
Fortunately, the worst part (a fast and reliable parser) has already been tackled: the ...
- 734
5
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
Honestly, the easiest way (especially for FASTQ) is probably to use a dedicated parsing library, such as R/Bioconductor:
...
- 4,895
5
votes
Random access on a FASTQ file
You could shuffle the FASTQ once and then read sequences off the top of the file as you need them:
...
- 2,286
5
votes
How do I efficiently subset a very large line-based file?
Perl should be fairly fast with this when using a hash set to store the list of lines. A structure like this also works for subsetting based on a field value, where the comparison would be with the ...
- 15.1k
5
votes
What is the fastest way to get the reverse complement of a DNA sequence in python?
Another python extension but without cython. The source code is available at the bottom of this answer or from this gist. On Mac with Python3:
...
- 6,605
4
votes
How do I efficiently subset a very large line-based file?
Some related questions appear in other sites, with potentially interesting solutions, which I report here:
To sample approximately 1% of the non-empty lines:
...
- 3,195
3
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
For FASTA files, I've implemented a relatively efficient method in pyfaidx v0.4.9.1. This post made me realize that my previous code was quite slow and easy to replace:
...
- 141
3
votes
Random access on a FASTQ file
One possibility is to:
reformat the data such that each record is a single line containing the read description, bases, and quality scores
pad out each record to a maximum length in each field such ...
- 31
3
votes
What is the fastest way to get the reverse complement of a DNA sequence in python?
Use a bytearray instead of a string and then employ maketrans to translate
You do not need ...
- 131
Only top scored, non community-wiki answers of a minimum length are eligible