I have a task in a training that I have to read and filter the 'good' reads of big fastq files. I downsampled, got the code working, saving in a python dictionary. But turns out the original files are huge and I rewrite the code to give a generator. It did work for the down-sampled sample. But I was wondering if it's a good idea to get out all the data and filtering in a dictionary. Does anybody here a better idea? I am asking because I am learning python on my own and I am not sure if it makes sense.
I got some ideas from a code in Biostar:
import sys import gzip filename = sys.argv def parsing_fastq_files(filename): with gzip.open(filename, "rb") as infile: count_lines = 0 for line in infile: line = line.decode() if count_lines % 4 == 0: ids = line[1:].strip() yield ids if count_lines == 1: reads = line.rstrip() yield reads count_lines += 1 total_reads = parsing_fastq_files(filename) print(next(total_reads)) print(next(total_reads))
I now need to figure out to get the data filtered by using 'if value.endswith('expression'):' but if I use a dict for example, but that's my doubt because of the number of keys and values.