Hot answers tagged

62 votes
Accepted

What is the difference between FASTA, FASTQ, and SAM file formats?

Let’s start with what they have in common: All three formats store sequence data, and sequence metadata. Furthermore, all three formats are text-based. However, beyond that all three formats are ...
Konrad Rudolph's user avatar
26 votes
Accepted

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

For FASTQ: seqtk fqchk in.fq | head -2 It gives you percentage of "N" bases, not the exact count, though. For FASTA: ...
user172818's user avatar
  • 6,485
20 votes

What is the difference between FASTA, FASTQ, and SAM file formats?

In a nutshell, FASTA file format is a DNA sequence format for specifying or representing DNA sequences and was first described by Pearson ...
eastafri's user avatar
  • 329
18 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

5 hours and no benchmarks posted? I am sorely disappointed. I'll restrict the comparison to just be fasta files, since fastq will end up being the same. So far, the contenders are: R with the ...
Devon Ryan's user avatar
  • 19.6k
16 votes
Accepted

Why does this human bam file only have one copy of each chromosome?

The maternal and paternal copies of a chromosome are called haplotypes. Many metazoans (animals) are diploid and have maternal and paternal chromosome contribution during sexual reproduction, not just ...
conchoecia's user avatar
  • 3,141
14 votes
Accepted

Fast way to count number of reads and number of bases in a fastq file?

It's difficult to get this to go massively quicker I think - as with this question working with large gzipped FASTQ files is mostly IO-bound. We could instead focus on making sure we are getting the ...
sjcockell's user avatar
  • 861
13 votes
Accepted

How can I improve the yield of MinION sequencing runs?

I attended a talk by Josh Quick at PoreCampAU 2017, in which he discussed some common barriers to getting both good sequencing yield and long read length. It mostly boils down to being more careful ...
gringer's user avatar
  • 13.8k
12 votes

Good / recommended way to archive fastq and bam files?

EDIT: I am rewriting the answer in response to updates to the original question. TL;DR: use CRAM Background 1: quality binning and FASTQ compression In the old days, base callers outputted base ...
user172818's user avatar
  • 6,485
11 votes

Random access on a FASTQ file

Arbitrary record access in constant time To get a random record in constant time, it is sufficient to get an arbitrary record in constant time. I have two solutions here: One with ...
winni2k's user avatar
  • 2,236
9 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

I think that this should be pretty fast: FASTA: grep -v "^>" seqs.fa | tr -cd N | wc -c FASTQ: ...
Karel Břinda's user avatar
9 votes
Accepted

Extract nanopore read ID & start times from fastq file

awk '{if(NR%4==1) print $1, $5}' file.fastq | sed -e "s/ start_time=/, /" -e "s/^@//" The awk command gets the first of every ...
Devon Ryan's user avatar
  • 19.6k
9 votes

Delete all 4 lines of a fastq read from a fastq file using read ID

Don't do it: Your FASTQ file is either malformed or your FASTQ record spans more than four lines, which is allowed in FASTQ. For a detailed description of what can go wrong in FASTQ parsing see for ...
winni2k's user avatar
  • 2,236
8 votes

What is the difference between FASTA, FASTQ, and SAM file formats?

FASTA (officially) just stores the name of a sequence and the sequence, unofficially people also add comment fields after the name of the sequence. FASTQ was invented to store both sequence and ...
BaCh's user avatar
  • 754
8 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

FASTQ As it was pointed out, fastq can be complicated. But in a simple case when you have four lines per record, one possible solution in bash is: ...
Iakov Davydov's user avatar
8 votes
Accepted

Calculating read average length in a Fastq file with bioawk/awk

This can also be done with regular awk. awk '{if(NR%4==2) {count++; bases += length} } END{print bases/count}' <fastq_file> The ...
Bioathlete's user avatar
  • 2,574
7 votes
Accepted

Random access on a FASTQ file

As wkretzsch suggested this was worthy of an actual answer, I feel the obvious solution is missing here; index the FASTQ. Index it As much as I typically hesitate ...
Sam Nicholls's user avatar
7 votes

The effects of incomplete bisulfite conversion upon mapping efficiency

Bisulfite conversion efficiency has no effect on the mapping rate in bismark and similar tools. The reason is that the reads are fully bisulfite converted in silico before alignment to minimize ...
Devon Ryan's user avatar
  • 19.6k
7 votes
Accepted

5' and 3' bias in Rna-seq data

I expect there was a sequencing problem during the last base, where some of the reagents were running low on the sequencer. This won't pose any real problem, RNAseq aligners like STAR will just soft-...
Devon Ryan's user avatar
  • 19.6k
7 votes
Accepted

Script to run everything in a loop for extracting tar.gz files into fastq and to bam with alignment?

This should work for you: ...
heathobrien's user avatar
  • 1,816
7 votes
Accepted

Multithread fastq processing with kseq.h in C++11?

You might take a look at my FQFeeder library (https://github.com/rob-p/FQFeeder), which pairs the kseq parser with a multi-producer / multi-consumer queue to allow efficient parallel processing of the ...
nomad's user avatar
  • 472
7 votes

Compressing read data (converting fastq to fastq.gz) on Windows

7zip is file archiver capable of running gzip-style compression, and runs on Windows: https://www.7-zip.org/
TimD1's user avatar
  • 292
6 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

Decompression of gzipped FASTQ is the main issue If we take real world gzipped FASTQ (which as the OP suggested would be beneficial) rather than trivial FASTA as the starting point then the real ...
Matt Bashton's user avatar
  • 1,059
6 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

Using bioawk With bioawk (counting "A" in the C. elegans genome, because there seem to be no "N" in this file), on my computer: ...
bli's user avatar
  • 3,100
6 votes

Fast way to count number of reads and number of bases in a fastq file?

The following is more than twice as fast; however, wc counts newline characters as well. We thus need to subtract the line count from the base count (using Bash): <...
Konrad Rudolph's user avatar
6 votes

Fast way to count number of reads and number of bases in a fastq file?

pigz | awk | wc is the fastest method First off for benchmarks with FASTQ it's best to use a specific real-world example with a known answer. I've chosen this file: ftp://ftp.1000genomes.ebi.ac.uk/...
Matt Bashton's user avatar
  • 1,059
6 votes

Extract nanopore read ID & start times from fastq file

Since the string start_time will only appear on the header line, or else you don't have a valid fastq file, you can simply do: ...
terdon's user avatar
  • 9,662
6 votes
Accepted

FASTQC overrepresented sequences after trimming

As @AaronBerlin mentioned, you didn't remove reads that were completely trimmed. Next time use the --minimum-length option and set it to something reasonable, like ...
Devon Ryan's user avatar
  • 19.6k
6 votes
Accepted

How to convert fastq to fast5

NOTICE: I have altered my answer slightly from the original as I have turned the original script into a pip installable program (with tests) and have updated the ...
Michael Hall's user avatar
6 votes
Accepted

How to get the count of each kmer past 255 using khmer

255 is the default maximum size of a Counttable in khmer. You want to do the following: ...
Devon Ryan's user avatar
  • 19.6k

Only top scored, non community-wiki answers of a minimum length are eligible