This question was also asked on Stack Overflow
Bioinformatics rationale eggNOG files can be very big and sump all available RAM for regular to medium sized desktops.
I am looking for advice on using a pandas iterator.
I performed a parsing operation using Python pandas, the size of the input files is eggNOG a program used in metagenomic gene function identification but has broader application. The files are large and parsing is resulting in 'RAM bottleneck' phenomenon and will not processing/parse the file.
The obvious solution is to shift to an iterator, which for pandas is the chunksize option
import pandas as pd import numpy as np df = pd.read_csv('myinfile.csv', sep="\t", chunksize=100)
Whats changed with the original code is the
chunksize=100 bit, forcing an iterator.
The next step is just to perform a simple operation, dropping a few columns and moving all '-' characters to np.nan then writing the whole file to disk.
df.drop(['score', 'evalue', 'Description', 'EC', 'PFAMs'],axis=1).replace('-', np.nan) df.to_csv('my.csv',sep='\t',index=False)
How is the step performed under an iterator?