Hot answers tagged

57 votes
Accepted

What is the difference between FASTA, FASTQ, and SAM file formats?

Let’s start with what they have in common: All three formats store sequence data, and sequence metadata. Furthermore, all three formats are text-based. However, beyond that all three formats are ...
user avatar
24 votes
Accepted

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

For FASTQ: seqtk fqchk in.fq | head -2 It gives you percentage of "N" bases, not the exact count, though. For FASTA: ...
user avatar
  • 5,735
18 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

5 hours and no benchmarks posted? I am sorely disappointed. I'll restrict the comparison to just be fasta files, since fastq will end up being the same. So far, the contenders are: R with the ...
user avatar
  • 19.2k
18 votes

What is the difference between FASTA, FASTQ, and SAM file formats?

In a nutshell, FASTA file format is a DNA sequence format for specifying or representing DNA sequences and was first described by Pearson ...
user avatar
  • 309
16 votes
Accepted

Why does this human bam file only have one copy of each chromosome?

The maternal and paternal copies of a chromosome are called haplotypes. Many metazoans (animals) are diploid and have maternal and paternal chromosome contribution during sexual reproduction, not just ...
user avatar
  • 3,071
14 votes
Accepted

Fast way to count number of reads and number of bases in a fastq file?

It's difficult to get this to go massively quicker I think - as with this question working with large gzipped FASTQ files is mostly IO-bound. We could instead focus on making sure we are getting the ...
user avatar
  • 861
12 votes
Accepted

How can I improve the yield of MinION sequencing runs?

I attended a talk by Josh Quick at PoreCampAU 2017, in which he discussed some common barriers to getting both good sequencing yield and long read length. It mostly boils down to being more careful ...
user avatar
  • 11.7k
11 votes

Random access on a FASTQ file

Arbitrary record access in constant time To get a random record in constant time, it is sufficient to get an arbitrary record in constant time. I have two solutions here: One with ...
user avatar
  • 2,051
9 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

I think that this should be pretty fast: FASTA: grep -v "^>" seqs.fa | tr -cd N | wc -c FASTQ: ...
user avatar
9 votes
Accepted

Extract nanopore read ID & start times from fastq file

awk '{if(NR%4==1) print $1, $5}' file.fastq | sed -e "s/ start_time=/, /" -e "s/^@//" The awk command gets the first of every ...
user avatar
  • 19.2k
9 votes

Delete all 4 lines of a fastq read from a fastq file using read ID

Don't do it: Your FASTQ file is either malformed or your FASTQ record spans more than four lines, which is allowed in FASTQ. For a detailed description of what can go wrong in FASTQ parsing see for ...
user avatar
  • 2,051
8 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

FASTQ As it was pointed out, fastq can be complicated. But in a simple case when you have four lines per record, one possible solution in bash is: ...
user avatar
7 votes
Accepted

Random access on a FASTQ file

As wkretzsch suggested this was worthy of an actual answer, I feel the obvious solution is missing here; index the FASTQ. Index it As much as I typically hesitate ...
user avatar
7 votes

The effects of incomplete bisulfite conversion upon mapping efficiency

Bisulfite conversion efficiency has no effect on the mapping rate in bismark and similar tools. The reason is that the reads are fully bisulfite converted in silico before alignment to minimize ...
user avatar
  • 19.2k
7 votes

What is the difference between FASTA, FASTQ, and SAM file formats?

FASTA (officially) just stores the name of a sequence and the sequence, unofficially people also add comment fields after the name of the sequence. FASTQ was invented to store both sequence and ...
user avatar
  • 744
7 votes
Accepted

5' and 3' bias in Rna-seq data

I expect there was a sequencing problem during the last base, where some of the reagents were running low on the sequencer. This won't pose any real problem, RNAseq aligners like STAR will just soft-...
user avatar
  • 19.2k
7 votes
Accepted

Script to run everything in a loop for extracting tar.gz files into fastq and to bam with alignment?

This should work for you: ...
user avatar
  • 1,806
7 votes
Accepted

Calculating read average length in a Fastq file with bioawk/awk

This can also be done with regular awk. awk '{if(NR%4==2) {count++; bases += length} } END{print bases/count}' <fastq_file> The ...
user avatar
  • 2,556
7 votes
Accepted

Multithread fastq processing with kseq.h in C++11?

You might take a look at my FQFeeder library (https://github.com/rob-p/FQFeeder), which pairs the kseq parser with a multi-producer / multi-consumer queue to allow efficient parallel processing of the ...
user avatar
  • 472
6 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

Decompression of gzipped FASTQ is the main issue If we take real world gzipped FASTQ (which as the OP suggested would be beneficial) rather than trivial FASTA as the starting point then the real ...
user avatar
  • 1,029
6 votes

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

Using bioawk With bioawk (counting "A" in the C. elegans genome, because there seem to be no "N" in this file), on my computer: ...
user avatar
  • 2,960
6 votes

Fast way to count number of reads and number of bases in a fastq file?

The following is more than twice as fast; however, wc counts newline characters as well. We thus need to subtract the line count from the base count (using Bash): <...
user avatar
6 votes

Fast way to count number of reads and number of bases in a fastq file?

pigz | awk | wc is the fastest method First off for benchmarks with FASTQ it's best to use a specific real-world example with a known answer. I've chosen this file: ftp://ftp.1000genomes.ebi.ac.uk/...
user avatar
  • 1,029
6 votes

Extract nanopore read ID & start times from fastq file

Since the string start_time will only appear on the header line, or else you don't have a valid fastq file, you can simply do: ...
user avatar
  • 8,111
6 votes
Accepted

FASTQC overrepresented sequences after trimming

As @AaronBerlin mentioned, you didn't remove reads that were completely trimmed. Next time use the --minimum-length option and set it to something reasonable, like ...
user avatar
  • 19.2k
6 votes
Accepted

How to convert fastq to fast5

NOTICE: I have altered my answer slightly from the original as I have turned the original script into a pip installable program (with tests) and have updated the ...
user avatar
6 votes
Accepted

How to get the count of each kmer past 255 using khmer

255 is the default maximum size of a Counttable in khmer. You want to do the following: ...
user avatar
  • 19.2k
6 votes

Counting repeated kmers sequences that match at least x % of reads sequence

The following javascript does what you want. You need node.js to run it. It should be easy to translate the code to Python. I have not carefully tested it. Use with caution. EDIT (response to new ...
user avatar
  • 5,735
6 votes
Accepted

How to align output of grep --color=always? (To QC fasta/fastq files)

Perhaps, grep is not the best tool to use in this case, but it should be in principle possible by using grep & ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible