I wrote a tool called sample that you can use to do random sampling without reading the entire file into memory.

It can be used where GNU shuf fails for lack of sufficient memory.

It requires two passes through the file to do a random sample, but the second pass is generally fast(er) as it uses mmap routines to do cached reads.

If you do repeated samples, the repeated samples are also mmap-ed (cached) and will run quickly.

You might use it on a FASTQ file like so:

$ sample -k 1234 -l 4 in.fq > out.fq

It parses the input file into records by every four newline characters (such as the format of a FASTQ file), reading line offset positions into memory. So the memory overhead is relatively very low.

It then applies reservoir sampling on those line offsets to write out a random sample (say, 1234 records in this example) to standard output.

I wrote a tool called sample that you can use to do random sampling without reading the entire file into memory.

It can be used where GNU shuf fails for lack of sufficient memory.

It requires two passes through the file to do a random sample, but the second pass is generally fast(er) as it uses mmap routines to do cached reads.

If you do repeated samples, the repeated samples are also mmap-ed (cached) and will run quickly.

You might use it on a FASTQ file like so:

$ sample -k 1234 -l 4 in.fq > out.fq

It parses the input file into records by every four newline characters (such as the format of a FASTQ file), reading line offset positions into memory. So the memory overhead is relatively very low.

It then applies reservoir sampling on those line offsets to write out a random sample (say, 1234 records in this example) to standard output.