3
$\begingroup$

I have a script that extracts the set of unique kmer pairs from a kmer dump file (& then using them downstream to guess ploidy).

The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.

I don't really get it; the script actually prints the last line

*_families_2.tsv and *_coverages_2.tsv files saved.

, but then it runs further (& forever). I thought it could be related to longer time to free the memory as discussed here, but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).

Does anybody an idea where the problem could be? Or how to figure out?

The reported problem on GitHub.

Improve this question
$\endgroup$
6
  • $\begingroup$ Can you trace the memory usage of the progress to see if it's related to freeing memory? we had a similar issue some time ago, tried a lot of things like disabling garbage collection etc. but never resolved it. Also have you tried running on another system? our issue was linked to a particular system with an old OS/software $\endgroup$
    – Chris_Rands
    Commented Dec 20, 2018 at 15:01
  • 2
    $\begingroup$ Also this might fit better on Stack Overflow (although they might complain about it being no small and reproducible) $\endgroup$
    – Chris_Rands
    Commented Dec 20, 2018 at 15:02
  • $\begingroup$ After the last line is printed the program seems to occupy exactly the same amount of memory over time. $\endgroup$
    – Kamil S Jaron
    Commented Dec 20, 2018 at 15:02
  • $\begingroup$ I was thinking about posting it to Stack Overflow, but since it's kmer related problem and since I prefer to interact with this community I have decided to post it here. $\endgroup$
    – Kamil S Jaron
    Commented Dec 20, 2018 at 15:04
  • $\begingroup$ As the author suggested, this is probably caused by a deadlock in multi-threading. $\endgroup$
    – user172818
    Commented Dec 20, 2018 at 15:33

1 Answer 1

Reset to default
2
$\begingroup$

You've hit the RAM bottleneck. Its very common in de novo assemblies and resolved using template assemblies. I don't know this program but suspect it is performing a pairwise analysis for every piece of data in a genomics data set.

The solution is to either:

  1. find a bigger machine, e.g. a big multi-core machine and make sure the program distributes across every core.

  2. parallelise the calculation (an excellent way to overcome RAM problems), or,

  3. find an approximation to the calculation that is less computationally demanding, templates are a good way to circumvent this.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.