Non-random access on a fastq file

Question

There was a nice question on random access on a fastq file, however I have the opposite problem. I want to programatically access non-random fastq entries quickly (using C++).

Say I have several pairs of fastq.gz files that could be hundreds of millions or > 1 billion read pairs total. I know the number of read pairs in each file and I have a list of 10-20 reads that I want to access from each file. I only know the 0-based index of the fastq entry that I want to access, not the read ID. So if I want the 234th and 863rd reads the naive solution is to open the fastq file and scan through 864 records. I need to perform some tasks with these reads, then I move on and access/process another batch of reads.

• I need to perform the above task tens of thousands of times, so the options are:
  • 1. Low RAM usage. Many file IO ops. No disk usage: Opening/scanning/closing the fastq.gz files then processing the desired reads tens of thousands of times.
  • 2. Extreme RAM usage. One round of file reading. No disk usage: Read in each fastq.gz file once, store all of the desired reads in RAM and access and process those reads as needed
  • 3. Low RAM usage. One round of file reading. Some disk usage: Coming up with an intermediate file format similar to a bam that will allow me to parse and sort the reads with their groups and other metadata later on.

If it makes a difference the fastq.gz files are on spinning disks, not SSDs.

Answer 1

You don't have a RAM problem because you only have about:

10-20 reads that I want to access from each file.

Please go for (2). If you are concerned about RAM usage, you may selectively write some of your data back to disk.

Tens of thousands of 10-20 reads is not a problem for modern computing.