Questions tagged [fastq]

FASTQ is a file format use to store short reads and their quality values

Filter by
Sorted by
Tagged with
42 votes
4 answers
60k views

What is the difference between FASTA, FASTQ, and SAM file formats?

I'd like to learn the differences between 3 common formats such as FASTA, FASTQ and SAM. How they are different? Are there any benefits of using one over another? Based on Wikipedia pages, I can't ...
kenorb's user avatar
  • 1,313
21 votes
10 answers
7k views

What is the fastest way to calculate the number of unknown nucleotides in FASTA / FASTQ files?

I used to work with publicly available genomic references, where basic statistics are usually available and if they are not, you have to compute them only once so there is no reason to worry about ...
Kamil S Jaron's user avatar
20 votes
12 answers
2k views

Random access on a FASTQ file

I would like to select a random record from a large set of n unaligned sequencing reads in log(n) time complexity (big O ...
winni2k's user avatar
  • 2,256
14 votes
5 answers
2k views

Good / recommended way to archive fastq and bam files?

We have a lot of Illumina sequenced exome data. Currently we are using spring for its great lossless compression, but we are looking if there is anything better (and most preferably opensource) which ...
Karthik Nair's user avatar
14 votes
3 answers
2k views

How can I improve the yield of MinION sequencing runs?

This is a frequently-asked question within the nanopore community. Oxford Nanopore currently claims that they are able to generate run yields of 10-15 gigabases (e.g. see here and here), and yet it's ...
gringer's user avatar
  • 13.9k
13 votes
8 answers
15k views

Fast way to count number of reads and number of bases in a fastq file?

I am looking for a tool, preferably written in C or C++, that can quickly and efficiently count the number of reads and the number of bases in a compressed fastq file. I am currently doing this using <...
terdon's user avatar
  • 9,901
13 votes
2 answers
3k views

How to convert fastq to fast5

fast5 is a variant of HDF5 the native format in which raw data from Oxford Nanopore MinION are provided. You can easily extract ...
aechchiki's user avatar
  • 2,676
10 votes
2 answers
2k views

How do you generate read-length vs read-quality plot for long-read sequencing data (e.g., MinION)?

How do you generate read-length vs read-quality plot (heat map with histograms in the margin) for long-read sequencing data from the Oxford Nanopore Technologies (ONT) MinION? The MinKNOW software ...
Mark Ebbert's user avatar
  • 1,354
9 votes
1 answer
710 views

Why does this human bam file only have one copy of each chromosome?

As we know that in human DNA sequence, one copy of chromosome comes from mother's DNA and another copy comes from father's DNA so as to form two copies of each chromosome in human DNA. So, if we ...
Lot_to_learn's user avatar
9 votes
1 answer
399 views

How to get the count of each kmer past 255 using khmer

I have a Fastq file and I want to get the exact count of each possible kmer from this file. On a previous post called How to use khmer to count k-mers? Daniel Standage proposed a custom script based ...
hilta007's user avatar
  • 173
9 votes
1 answer
2k views

FASTQC overrepresented sequences after trimming

I have a set of RNA-seq samples from different experiments (Single and Paired End, depending on the experiment). I ran FASTQC in all the samples and found overrepresented adapter sequences: I removed ...
plat's user avatar
  • 1,032
8 votes
1 answer
486 views

Counting repeated kmers sequences that match at least x % of reads sequence

Working on a fastQ file, I would like to get the occurrences of repeated sequences for all possible kmers of a given length that cover at least 90% of the read's length for the whole data set. ...
hilta007's user avatar
  • 173
7 votes
3 answers
1k views

Extract nanopore read ID & start times from fastq file

I have a fastq file from minION (albacore) that contains information on the read ID and the start time of the read. I want to extract these two bits of information into a single csv file. I've been ...
roblanf's user avatar
  • 962
7 votes
5 answers
5k views

Convert FASTA to FASTQ with dummy quality scores

I have a FASTA file which I would like to convert into FASTQ format as the tool I want to use my data in requires it in FASTQ format. So dummy quality scores are fine. Note: I am not using any ...
Michael Hall's user avatar
7 votes
1 answer
848 views

Multithread fastq processing with kseq.h in C++11?

Background I am using the wonderful kseq.h as my fastq/fasta parser for a C++ project. My actual implementation uses something like this, where I am counting kmers ...
conchoecia's user avatar
  • 3,141
7 votes
4 answers
9k views

Bash scripting FastQC for multiple fastq files in multiple directories

I am completely new to bioinformatics so I'm looking to learn how to do this. I have multiple directories with fastq files: E.g; 10 Directories with each time series, each with Treatment and control ...
user avatar
7 votes
1 answer
647 views

The effects of incomplete bisulfite conversion upon mapping efficiency

This question has also been posted on Biostars I have sequenced numerous multiplexed pools of BS amplicon-seq libraries derived from human samples on a MiSeq over the past few weeks. I have been ...
David Ross's user avatar
7 votes
1 answer
536 views

Split FASTQ and matching BAM into matching chunks

I am running a slow downstream analysis on a large set of nanopore reads (approx 3 million), and would like to split them into smaller chunks, run the analysis in massively parallel, and then ...
Scott Gigante's user avatar
7 votes
2 answers
305 views

Canu assembly not making a single consensus?

I've downloaded reads from this BioProject. Using canu with default parameters (no correction), I've got 4 contigs, none of which really look like the reference plasmid here. The command I used was: ...
thestatnoob's user avatar
6 votes
5 answers
2k views

Delete all 4 lines of a fastq read from a fastq file using read ID

I have the following error when running bowtie2: Error: Read HWI-D00466:116:CC62WANXX:3:1102:7363:63646 1:N:0:GCACACG has more read characters than quality values. I now want to remove all 4 ...
T. Ntsowe's user avatar
6 votes
2 answers
5k views

5' and 3' bias in Rna-seq data

I'm working with rna-seq samples. I see 5' bias and also 3' bias in the per-base sequence content plot. From this link I see that the bias at the start of the sequences appears to be the result of ...
stack_learner's user avatar
6 votes
2 answers
945 views

How can I edit a specific FASTQ read in place, given the read ID?

I am introducing SNVs into specific samples in order to estimate false negative rates for a variant calling pipeline. I know reads can be simulated but I would actually prefer to use the real data so ...
dkainer's user avatar
  • 128
6 votes
2 answers
1k views

How to safely and efficiently convert subset of bam to fastq?

Question How can I extract reads from a bam file (produced by bwa-mem) to fastq given a ...
Kamil S Jaron's user avatar
6 votes
1 answer
2k views

Convert BAM to properly paired FASTQ files

I thought I had figured this one out. But apparently not. I need to convert a BAM file of paired-end alignments to two FASTQ files of paired reads to realign them, with a twist: I only want reads ...
Konrad Rudolph's user avatar
6 votes
1 answer
1k views

How to maximize fastq parsing with FastqGeneralIterator (Bio.SeqIO.QualityIO)

I'm contributing to a python-based project that uses Biopython to analyze fastq files. It currently uses SeqIO.parse, which ...
Mark Ebbert's user avatar
  • 1,354
6 votes
1 answer
75 views

Determine reference for reference-compressed SRA file

Note: this question was also asked on Github. I have 241 SRA files that appear to be reference compressed. I didn't even know this was a thing until I tried to convert them to Fastq files without an ...
Daniel Standage's user avatar
5 votes
1 answer
3k views

Estimating BAM file from compressed fastq file size

Is there a way to estimate the size of a BAM file will have after mapping with BWA? The input file are two mates fastq files, compressed with gzip, each one about 70G.
gc5's user avatar
  • 1,783
4 votes
2 answers
10k views

Calculating read average length in a Fastq file with bioawk/awk

I found here this awk script: ...
user977828's user avatar
4 votes
2 answers
469 views

How to align output of grep --color=always? (To QC fasta/fastq files)

Grepping out short sequences from a fasta or fastq file is a really useful way to look at sequencing data. Using the option --color=always makes this even more ...
conchoecia's user avatar
  • 3,141
4 votes
3 answers
3k views

What do the FASTQ file names mean here?

I would like to run an analysis on Nanopore data. I'm trying to download sample data from https://github.com/nanopore-wgs-consortium/chm13. I would like to run minimap2 on the FASTQ files, resulting ...
SmallChess's user avatar
  • 2,749
4 votes
1 answer
88 views

Telling grep to treat N as [ATCG]

Okay so I'm using grep to try and get a preview of some trimming operations that are not going as expected.. Lets say that my sequence in the FastQ file is: ATNGCNATCG What I want to do is.. ...
RPINerd's user avatar
  • 51
4 votes
2 answers
97 views

Is the quality score of fastq used somewhere besides trimming/fastqc?

In the fastq format every 4k+4-th line contains the positionwise qualityscore (ascii encoded): ...
Paul's user avatar
  • 317
4 votes
2 answers
414 views

Is there a way to quickly verify the presence of some SNPs in Fastq files?

Is there a tool that can scan fastq files without assembling them for a custom list of user defined snps?
Dan K's user avatar
  • 89
4 votes
4 answers
1k views

How to get a file with the number of reads for several fastq.gz files?

I have generated several FASTQ files and I would like to know the amount of reads for each of them. I am planning to run FastQC on the files which I know would give me the number of reads per sample ...
pythonbeginner's user avatar
4 votes
1 answer
321 views

Find pattern that is present twice and allow <=2 mismatches on each

I have a fastq file of 400,000 reads (so speed is important). In the sequences there are barcodes integrated that should be present twice. Given a barcode, I want to find the sequences that have the ...
nafizh's user avatar
  • 69
4 votes
1 answer
657 views

What is a proper way for random subsampling of metagenomic data?

Let's say we have a metagenomic sample that is paired-end FASTQ files including 10,000,000 DNA reads collected using shotgun sequencing. How would one make a random subsample of the mentioned ...
Remy's user avatar
  • 53
3 votes
2 answers
313 views

Frequency of specific viral sequence in .BAM or .fastq

I was wondering if anyone had any experience 'counting' the frequency of a particular viral sequence in an individual's fastq sequence. So basically, counting the number of occurrences of a particular ...
h3ab74's user avatar
  • 836
3 votes
2 answers
763 views

Does this FASTQ data contain single or paired end calls?

I have this fastq data from GEO: ...
0x90's user avatar
  • 1,427
3 votes
1 answer
295 views

Fast processing of fastq data

I am trying to write python script for customized filtering for fastq file (size >3 GB). My proposed script is as follows: ...
Lot_to_learn's user avatar
3 votes
2 answers
252 views

What can be the bias of aligning paired-reads in a single-end mode?

I don't know if I'm in the right place but I have a technical problem to fix. I would have to align paired-end reads from Illumina sequencing to compare a normal genome with a tumor one. When I align ...
cucalorda's user avatar
3 votes
2 answers
326 views

Getting data from fastq by generator

I have a task in a training that I have to read and filter the 'good' reads of big fastq files. I downsampled, got the code working, saving in a python dictionary. But turns out the original files ...
Paulo Sergio Schlogl's user avatar
3 votes
3 answers
854 views

How to demultiplex pair-end fastq reads with barcode 2 in the identifier line?

I have multiplexed pair-end fastq reads with dual barcodes. The issue is that one barcode is present in the header and one is present at the beginning of the read. I need a method to demultiplex this ...
Caleb Benson's user avatar
3 votes
1 answer
291 views

ncbi fastq dump error in loop

I have a very basic objective which I want to give list of id to a ncbi-srafastq tool kit which will prefetch the id which is in .sra file extension and then it will run fastq-dump on the file to ...
PesKchan's user avatar
  • 207
3 votes
1 answer
117 views

Shell script to validate fastq issue

I have two GSM ID as test case where both of them are having data when I checked through sra explorer tool but when I try to fetch through a script only one of them returns output where the other one ...
kcm's user avatar
  • 1,804
3 votes
1 answer
360 views

Multiple file Python script

I am currently doing an analysis requiring a multiple-input script. I have a directory filled with 900+ samples. Each sample is comprised of two-files in a fastq format. This is an example of a sample:...
pvp's user avatar
  • 67
3 votes
1 answer
165 views

Can the canu assembler output a fastq file of the final assembly just like HGAP4?

I have assembled some genome from Sequel PacBio data both with HGAP4 on the SMRT Link interface and using canu on the command line. The HGAP4 assembler outputs a fastq file of the final assembly such ...
Biomagician's user avatar
  • 2,459
3 votes
1 answer
161 views

Multiple SRR ID downloaded for single GSM IDs input issue

So here I'm trying to donwload multiple fastq files based on the GSM ID input the script works fine when then input is only SRR id but it runs into error when it is given GSM ID My small GSM ID input ...
PesKchan's user avatar
  • 207
3 votes
1 answer
141 views

CRISPR genome wide screen analysis using MAGeCK-- Loss of reads/sgRNA using MAGeCK count function?

Sorry if this question seems basic. I am trying to run the MAGeCK pipeline to analyze CRISPR knockout screen data produced by someone in my lab around 5 years ago. I was given the data as bam files ...
Cassie Bishop's user avatar
3 votes
1 answer
173 views

Tool for calibrating base quality scores?

A given sequencing machine assigns a 'certainty' to each base call in the form of a per-base quality score (FASTQ format). I have reads from several machines aligned to the corresponding reference (...
Dan Bolser's user avatar
3 votes
0 answers
33 views

Template files for bioattributes and meta-data [closed]

I am in process of submiting my FASTQ files to the SRA database. I am a little confused about how to fill the biosample attributes and SRA metadata. I would really appreciate if you anyone could share ...
Riya's user avatar
  • 307