There was a nice question on random access on a fastq file, however I have the opposite problem. I want to programatically access non-random fastq entries quickly (using C++).

Say I have several pairs of fastq.gz files that could be hundreds of millions or > 1 billion read pairs total. I know the number of read pairs in each file and I have a list of 10-20 reads that I want to access from each file. I only know the 0-based index of the fastq entry that I want to access, not the read ID. So if I want the 234th and 863rd reads the naive solution is to open the fastq file and scan through 864 records. I need to perform some tasks with these reads, then I move on and access/process another batch of reads.

  • I need to perform the above task tens of thousands of times, so the options are:
      1. Low RAM usage. Many file IO ops. No disk usage: Opening/scanning/closing the fastq.gz files then processing the desired reads tens of thousands of times.
      1. Extreme RAM usage. One round of file reading. No disk usage: Read in each fastq.gz file once, store all of the desired reads in RAM and access and process those reads as needed
      1. Low RAM usage. One round of file reading. Some disk usage: Coming up with an intermediate file format similar to a bam that will allow me to parse and sort the reads with their groups and other metadata later on.

If it makes a difference the fastq.gz files are on spinning disks, not SSDs.

  • 1
    $\begingroup$ Load all lists into RAM and sequentially scan all FASTQ files. $\endgroup$ – user172818 Jan 16 '19 at 23:52

You don't have a RAM problem because you only have about:

10-20 reads that I want to access from each file.

Please go for (2). If you are concerned about RAM usage, you may selectively write some of your data back to disk.

Tens of thousands of 10-20 reads is not a problem for modern computing.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.