Skip to main content
Bioinformatics

Return to Question

Bumped by Community user
Fixed a typo in the first link
Source Link
user172818
user172818
  • 6.6k
  • 2
  • 14
  • 31

I have a scripta script that extracts the set of unique kmer pairs from a kmer dump file (& then using them downstream to guess ploidy).

The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.

I don't really get it; the script actually prints the last line

*_families_2.tsv and *_coverages_2.tsv files saved.

, but then it runs further (& forever). I thought it could be related to longer time to free the memory as discussed here, but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).

Does anybody an idea where the problem could be? Or how to figure out?

The reported problem on GitHub.

I have a script that extracts the set of unique kmer pairs from a kmer dump file (& then using them downstream to guess ploidy).

The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.

I don't really get it; the script actually prints the last line

*_families_2.tsv and *_coverages_2.tsv files saved.

, but then it runs further (& forever). I thought it could be related to longer time to free the memory as discussed here, but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).

Does anybody an idea where the problem could be? Or how to figure out?

The reported problem on GitHub.

I have a script that extracts the set of unique kmer pairs from a kmer dump file (& then using them downstream to guess ploidy).

The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.

I don't really get it; the script actually prints the last line

*_families_2.tsv and *_coverages_2.tsv files saved.

, but then it runs further (& forever). I thought it could be related to longer time to free the memory as discussed here, but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).

Does anybody an idea where the problem could be? Or how to figure out?

The reported problem on GitHub.

Source Link
Kamil S Jaron
Kamil S Jaron
  • 5.7k
  • 2
  • 28
  • 59

python does not quit when the input file is too big

I have a script that extracts the set of unique kmer pairs from a kmer dump file (& then using them downstream to guess ploidy).

The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.

I don't really get it; the script actually prints the last line

*_families_2.tsv and *_coverages_2.tsv files saved.

, but then it runs further (& forever). I thought it could be related to longer time to free the memory as discussed here, but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).

Does anybody an idea where the problem could be? Or how to figure out?

The reported problem on GitHub.