Skip to main content
26 votes
Accepted

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

For FASTQ: seqtk fqchk in.fq | head -2 It gives you percentage of "N" bases, not the exact count, though. For FASTA: ...
user172818's user avatar
  • 6,545
23 votes
Accepted

What is the fastest way to get the reverse complement of a DNA sequence in python?

I don't know if it's the fastest, but the following provides an approximately 10x speed up over your functions: ...
Devon Ryan's user avatar
  • 19.7k
18 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

5 hours and no benchmarks posted? I am sorely disappointed. I'll restrict the comparison to just be fasta files, since fastq will end up being the same. So far, the contenders are: R with the ...
Devon Ryan's user avatar
  • 19.7k
14 votes
Accepted

Fast way to count number of reads and number of bases in a fastq file?

It's difficult to get this to go massively quicker I think - as with this question working with large gzipped FASTQ files is mostly IO-bound. We could instead focus on making sure we are getting the ...
sjcockell's user avatar
  • 861
11 votes

Random access on a FASTQ file

Arbitrary record access in constant time To get a random record in constant time, it is sufficient to get an arbitrary record in constant time. I have two solutions here: One with ...
winni2k's user avatar
  • 2,286
10 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

I think that this should be pretty fast: FASTA: grep -v "^>" seqs.fa | tr -cd N | wc -c FASTQ: ...
Karel Břinda's user avatar
9 votes

What is the fastest way to get the reverse complement of a DNA sequence in python?

Here's a Cython approach that might suggest a generic approach to speeding up Python work. If you're manipulating (ASCII) character strings and performance is a design consideration, then C or Perl ...
Alex Reynolds's user avatar
8 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

FASTQ As it was pointed out, fastq can be complicated. But in a simple case when you have four lines per record, one possible solution in bash is: ...
Iakov Davydov's user avatar
8 votes
Accepted

Random access on a FASTQ file

As wkretzsch suggested this was worthy of an actual answer, I feel the obvious solution is missing here; index the FASTQ. Index it As much as I typically hesitate ...
Sam Nicholls's user avatar
8 votes

Samtools sort: most efficient memory and thread settings for many samples on a cluster

I usually use samtools sort -@4 -m4g. Samtools will be faster with more threads and more RAM, but I feel the gain is not significant. I haven't done a proper ...
user172818's user avatar
  • 6,545
7 votes

Fast way to count number of reads and number of bases in a fastq file?

The following is more than twice as fast; however, wc counts newline characters as well. We thus need to subtract the line count from the base count (using Bash): <...
Konrad Rudolph's user avatar
7 votes

Fast way to count number of reads and number of bases in a fastq file?

pigz | awk | wc is the fastest method First off for benchmarks with FASTQ it's best to use a specific real-world example with a known answer. I've chosen this file: ftp://ftp.1000genomes.ebi.ac.uk/...
Matt Bashton's user avatar
  • 1,069
7 votes

How do I efficiently subset a very large line-based file?

Turns out, simply keeping track of the next candidate line (after sorting the sample line numbers) fixes the performance issue, and most of the remaining slowness seems to be due to the overhead of ...
Konrad Rudolph's user avatar
7 votes

What is the fastest way to get the reverse complement of a DNA sequence in python?

The most reliable and simplest way is probably using Biopython: ...
Chris_Rands's user avatar
  • 3,948
7 votes

Remove/delete sequences by ID from multifasta

Suppose you keep sequence names in ids.txt and sequences in seq.fa: ...
user172818's user avatar
  • 6,545
7 votes

Remove/delete sequences by ID from multifasta

Just today I wrote a script to do exactly this using Biopython. I also add a warning If any the headers I wanted to filter was not present in the original fasta. So ...
Kamil S Jaron's user avatar
6 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

Decompression of gzipped FASTQ is the main issue If we take real world gzipped FASTQ (which as the OP suggested would be beneficial) rather than trivial FASTA as the starting point then the real ...
Matt Bashton's user avatar
  • 1,069
6 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

Using bioawk With bioawk (counting "A" in the C. elegans genome, because there seem to be no "N" in this file), on my computer: ...
bli's user avatar
  • 3,130
6 votes

Remove/delete sequences by ID from multifasta

If you want to learn how to do things with command-line tools, you can linearize the FASTA with awk, pipe to grep to filter for ...
Alex Reynolds's user avatar
6 votes
Accepted

Samtools sort: most efficient memory and thread settings for many samples on a cluster

Update 2 Ok, for what I expect will be my final update for this answer, I compared the following: samtools 1.14 + zlib ...
Mark Ebbert's user avatar
  • 1,354
5 votes

Fast way to count number of reads and number of bases in a fastq file?

I get fairly quick results with my fastx-length.pl script, with the added bonus of being able to handle multi-line FASTQ files and displaying additional read-length QC statistics: ...
gringer's user avatar
  • 14.4k
5 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

If it is raw speed you're after, then writing an own little C/C++ program is probably what you need to do. Fortunately, the worst part (a fast and reliable parser) has already been tackled: the ...
BaCh's user avatar
  • 734
5 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

Honestly, the easiest way (especially for FASTQ) is probably to use a dedicated parsing library, such as R/Bioconductor: ...
Konrad Rudolph's user avatar
5 votes

Random access on a FASTQ file

You could shuffle the FASTQ once and then read sequences off the top of the file as you need them: ...
winni2k's user avatar
  • 2,286
5 votes

How do I efficiently subset a very large line-based file?

Perl should be fairly fast with this when using a hash set to store the list of lines. A structure like this also works for subsetting based on a field value, where the comparison would be with the ...
gringer's user avatar
  • 14.4k
5 votes

What is the fastest way to get the reverse complement of a DNA sequence in python?

Another python extension but without cython. The source code is available at the bottom of this answer or from this gist. On Mac with Python3: ...
user172818's user avatar
  • 6,545
4 votes

How do I efficiently subset a very large line-based file?

Some related questions appear in other sites, with potentially interesting solutions, which I report here: To sample approximately 1% of the non-empty lines: ...
bli's user avatar
  • 3,130
3 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

For FASTA files, I've implemented a relatively efficient method in pyfaidx v0.4.9.1. This post made me realize that my previous code was quite slow and easy to replace: ...
Matt Shirley's user avatar
3 votes

Random access on a FASTQ file

One possibility is to: reformat the data such that each record is a single line containing the read description, bases, and quality scores pad out each record to a maximum length in each field such ...
kvg's user avatar
  • 31
3 votes

What is the fastest way to get the reverse complement of a DNA sequence in python?

Use a bytearray instead of a string and then employ maketrans to translate You do not need ...
Jack Aidley's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible