57
votes
Accepted
What is the difference between FASTA, FASTQ, and SAM file formats?
Let’s start with what they have in common: All three formats store
sequence data, and
sequence metadata.
Furthermore, all three formats are text-based.
However, beyond that all three formats are ...
- 4,745
24
votes
Accepted
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
For FASTQ:
seqtk fqchk in.fq | head -2
It gives you percentage of "N" bases, not the exact count, though.
For FASTA:
...
- 5,735
18
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
5 hours and no benchmarks posted? I am sorely disappointed.
I'll restrict the comparison to just be fasta files, since fastq will end up being the same. So far, the contenders are:
R with the ...
- 19.2k
18
votes
What is the difference between FASTA, FASTQ, and SAM file formats?
In a nutshell,
FASTA file format is a DNA sequence format for specifying or representing DNA sequences and was first described by Pearson ...
- 309
16
votes
Accepted
Why does this human bam file only have one copy of each chromosome?
The maternal and paternal copies of a chromosome are called haplotypes. Many metazoans (animals) are diploid and have maternal and paternal chromosome contribution during sexual reproduction, not just ...
- 3,071
14
votes
Accepted
Fast way to count number of reads and number of bases in a fastq file?
It's difficult to get this to go massively quicker I think - as with this question working with large gzipped FASTQ files is mostly IO-bound. We could instead focus on making sure we are getting the ...
- 861
12
votes
Accepted
How can I improve the yield of MinION sequencing runs?
I attended a talk by Josh Quick at PoreCampAU 2017, in which he discussed some common barriers to getting both good sequencing yield and long read length. It mostly boils down to being more careful ...
- 11.7k
11
votes
Random access on a FASTQ file
Arbitrary record access in constant time
To get a random record in constant time, it is sufficient to get an arbitrary record in constant time.
I have two solutions here: One with ...
- 2,051
9
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
I think that this should be pretty fast:
FASTA:
grep -v "^>" seqs.fa | tr -cd N | wc -c
FASTQ:
...
- 1,869
9
votes
Accepted
Extract nanopore read ID & start times from fastq file
awk '{if(NR%4==1) print $1, $5}' file.fastq | sed -e "s/ start_time=/, /" -e "s/^@//"
The awk command gets the first of every ...
- 19.2k
9
votes
Delete all 4 lines of a fastq read from a fastq file using read ID
Don't do it: Your FASTQ file is either malformed or your FASTQ record spans more than four lines, which is allowed in FASTQ. For a detailed description of what can go wrong in FASTQ parsing see for ...
- 2,051
8
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
FASTQ
As it was pointed out, fastq can be complicated. But in a simple case when you have four lines per record, one possible solution in bash is:
...
- 2,695
7
votes
Accepted
Random access on a FASTQ file
As wkretzsch suggested this was worthy of an actual answer, I feel the obvious solution is missing here; index the FASTQ.
Index it
As much as I typically hesitate ...
- 762
7
votes
The effects of incomplete bisulfite conversion upon mapping efficiency
Bisulfite conversion efficiency has no effect on the mapping rate in bismark and similar tools. The reason is that the reads are fully bisulfite converted in silico before alignment to minimize ...
- 19.2k
7
votes
What is the difference between FASTA, FASTQ, and SAM file formats?
FASTA (officially) just stores the name of a sequence and the sequence, unofficially people also add comment fields after the name of the sequence. FASTQ was invented to store both sequence and ...
- 744
7
votes
Accepted
5' and 3' bias in Rna-seq data
I expect there was a sequencing problem during the last base, where some of the reagents were running low on the sequencer. This won't pose any real problem, RNAseq aligners like STAR will just soft-...
- 19.2k
7
votes
Accepted
Script to run everything in a loop for extracting tar.gz files into fastq and to bam with alignment?
This should work for you:
...
- 1,806
7
votes
Accepted
Calculating read average length in a Fastq file with bioawk/awk
This can also be done with regular awk.
awk '{if(NR%4==2) {count++; bases += length} } END{print bases/count}' <fastq_file>
The ...
- 2,556
7
votes
Accepted
Multithread fastq processing with kseq.h in C++11?
You might take a look at my FQFeeder library (https://github.com/rob-p/FQFeeder), which pairs the kseq parser with a multi-producer / multi-consumer queue to allow efficient parallel processing of the ...
- 472
6
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
Decompression of gzipped FASTQ is the main issue
If we take real world gzipped FASTQ (which as the OP suggested would be beneficial) rather than trivial FASTA as the starting point then the real ...
- 1,029
6
votes
What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?
Using bioawk
With bioawk (counting "A" in the C. elegans genome, because there seem to be no "N" in this file), on my computer:
...
- 2,960
6
votes
Fast way to count number of reads and number of bases in a fastq file?
The following is more than twice as fast; however, wc counts newline characters as well. We thus need to subtract the line count from the base count (using Bash):
<...
- 4,745
6
votes
Fast way to count number of reads and number of bases in a fastq file?
pigz | awk | wc is the fastest method
First off for benchmarks with FASTQ it's best to use a specific real-world example with a known answer. I've chosen this file:
ftp://ftp.1000genomes.ebi.ac.uk/...
- 1,029
6
votes
Extract nanopore read ID & start times from fastq file
Since the string start_time will only appear on the header line, or else you don't have a valid fastq file, you can simply do:
...
- 8,111
6
votes
Accepted
FASTQC overrepresented sequences after trimming
As @AaronBerlin mentioned, you didn't remove reads that were completely trimmed. Next time use the --minimum-length option and set it to something reasonable, like ...
- 19.2k
6
votes
Accepted
How to convert fastq to fast5
NOTICE:
I have altered my answer slightly from the original as I have turned the original script into a pip installable program (with tests) and have updated the ...
- 573
6
votes
Accepted
How to get the count of each kmer past 255 using khmer
255 is the default maximum size of a Counttable in khmer. You want to do the following:
...
- 19.2k
6
votes
Counting repeated kmers sequences that match at least x % of reads sequence
The following javascript does what you want. You need node.js to run it. It should be easy to translate the code to Python. I have not carefully tested it. Use with caution.
EDIT (response to new ...
- 5,735
6
votes
Accepted
How to align output of grep --color=always? (To QC fasta/fastq files)
Perhaps, grep is not the best tool to use in this case, but it should be in principle possible by using grep & ...
- 2,695
Only top scored, non community-wiki answers of a minimum length are eligible