3
$\begingroup$

I have a large tsv file (~15 GB) that is a kmer count matrix and I am trying to calculate jaccard distance of the counts but every time I try to run the following code the process ends before it's done with a killed error message in the STDOUT:

jaccard = pd.DataFrame(squareform(pdist(kmer_matrix, 'jaccard')))
jaccard.to_csv(matrix_path+kmer+"_"+method+"_jacDistance_noFilter.tsv", sep='\t')

This process works really well with smaller tsv files so I know the code works but I wanted to know if anyone knew what would help run the process quicker/completely. I know the problem must be with my RAM/CPU but is there a way to maybe break up the matrix to calculate everything in batches to make it easier to work with?

$\endgroup$

1 Answer 1

3
$\begingroup$

You've overloaded the memory. One route is to deploy "chunking". pandas will do this. This is complicated to implement code wise however.

My suggested solution (because it simpler) would be to do this via pandas2 which uses Apache Arrow via PyArrow. You need to specify which method is being used at the onset of declaring the dataframe. Apache Arrow is for lifting large amounts of data. If this works there's no complicated coding you just import pandas2 as pd and in the dataframe construction describe the precise heavy lift method (I think there's an additional approach besides PyArrow).

$\endgroup$
3
  • 1
    $\begingroup$ Thank you! I was able to use a chunking method to calculate all the distances! Now the issue I'm having is I can't open the file to see the distances. Normally, I'd convert the .tsv file to a .meg file by going into the file and adding the proper headers to visualize the tree with MEGAX. But since I can't open the file I can't normally convert it. Any tips? $\endgroup$
    – rimo
    Commented Jan 10 at 22:01
  • 1
    $\begingroup$ Hi @rimo, great! Can you put this as a separate question please? It should be easily solvable. If you do I'll answer it on Saturday, because I've work commitments. $\endgroup$
    – M__
    Commented Jan 11 at 1:02
  • 1
    $\begingroup$ Thanks @M__!! I appreciate your help! I just posted another question. Take a look when you can! $\endgroup$
    – rimo
    Commented Jan 12 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.