5
$\begingroup$

We're preparing to analyze thousands of .bam files beginning with re-alignment, sorting, etc.

Complicated question: Has anyone investigated the optimal thread and memory settings to sort hundreds to thousands of .bam files using samtools sort on a cluster?

By 'optimal', I'm looking to sort as many samples as possible in the shortest time frame. I'm mostly interested in optimizing samtools sort because the other steps (e.g., alignment) are much more obvious to fine-tune.

For a single sample, throwing large amounts of memory + threads is trivial, but that isn't scalable for thousands of samples. There is also likely a diminishing return at a certain point, even for a single sample.

Ideally, I'd like to see plots identifying the 'sweet spot' for threads + memory for a single sample (e.g., x == threads, y == memory, z == time), and then plots maximizing throughput for more samples. I would expect the data to follow a -log function, where we could identify the optimal settings at the knee in the curve.

There are obviously many other factors (e.g., cluster size, administrative scheduler settings, etc.), but I suspect findings on a given cluster will generalize to other clusters.

Really appreciate any feedback.

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
8
$\begingroup$

I usually use samtools sort -@4 -m4g. Samtools will be faster with more threads and more RAM, but I feel the gain is not significant. I haven't done a proper evaluation, but I have run samtools sort with different settings from time to time on clusters when I have to consider speed-resource tradeoffs.

When you use samtools, make sure you compile it with libdeflate, which is about twice as fast as zlib on compression. I don't know if the bioconda version built with libdeflate. You may consider the precompiled binary here.

Samtools sort, especially with libdeflate, is fairly fast these days. Performance metrics in the other answer are outdated.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Super helpful info about libdeflate. I was not aware of that. I really want to do a comparison, but it's difficult to make the time. I may do it soon, and will try to include a comparison between zlib and libdeflate. $\endgroup$
    – Mark Ebbert
    Commented Feb 12, 2022 at 16:38
  • $\begingroup$ Side note: if you want to see if the samtools binary you're using is using zlib or libdeflate, you can see which libraries it was compiled against using ldd /path/to/<samtools_binary>. zlib will show as something similar to libz.so.1. $\endgroup$
    – Mark Ebbert
    Commented Feb 12, 2022 at 16:48
8
$\begingroup$

Update 2

Ok, for what I expect will be my final update for this answer, I compared the following:

  1. samtools 1.14 + zlib
  2. samtools 1.15 + libdeflate (this is a different version, but shouldn't have a major effect. They released a new version just before I asked cluster admins to compile against libdeflate and didn't notice until later.)
  3. sambamba v0.8.2

Note: This time I used the following settings:

  1. Only two samples instead of three
  2. Max Nextflow queue size of 30 to avoid too many threads reading from the same two files.
  3. I tested CPU requests from 1 to 7 (step size 1) and then from 9 to 17 (step size 2).
  4. For jobs where I allotted few CPUs and low memory, I provided a 200% buffer to prevent the jobs from failing with OUT OF MEMORY errors. i.e., I gave the SLURM job 2x the amount of memory than I told the tools they could use. sambamba was the worst offender for exceeding memory in these conditions, but I gave them all 200% buffer to keep comparisons equal.

Seems pretty clear to me that sambamba is the fastest and uses requested CPU resources most efficiently (i.e., is able to use all of the CPUs requested up until ~8 or 9 CPUs). For reference, the difference between sambamba and samtools + libdeflate with 9 CPUs is ~10 minutes on average.

We will certainly be using sambamba going forward, probably with ~9 CPUs and ~9GB per thread since I had to provide a 200% buffer in lower CPU/Mem jobs to avoid job failure, anyway.

Again, all code and results are available on my GitHub repository (Samtools sort optimization test).

(Angle 1) Comparison between samtools + zlib, samtools + libdeflate (Lsamtools), and sambamba

Here, I'm showing the general trend that real time decreases with increased CPUs for all three tools, but Mem Per CPU doesn't have a huge effect. sambamba out performs samtools + zlib regardless of settings. samtools + libdeflate out performs samtools + zlib until ~11 CPUs, where they perform the same (plot 3 below explains why).

Interesting finding: samtools + libdeflate performs better than sambamba with a single thread, but sambamba takes over from there as CPUs increase (again, plot #3 explains why).

samtools, Lsamtools, and sambamba

(Angle 2) Comparison between samtools + zlib, samtools + libdeflate (Lsamtools), and sambamba

Here, I'm showing that samtools + zlib and sambamba performance flattens out at ~11 CPUs (again, plot 3 below explans why).

samtools, Lsamtools, and sambamba

Requested CPUs vs. CPU utilization for samtools + zlib, samtools + libdeflate (Lsamtools), and sambamba

Here, I'm showing how well each tool is able to utilize the CPUs allotted to it. It was fascinating to see that samtools + libdeflate flattens out very early in it's CPU utilization (i.e., ~3 CPUS) and struggles to cross the 400% CPU utilization with even >10 CPUs. This is why samtools + libdeflate performance flattens out quickly.

samtools + zlib does a much better job utilizing allotted CPUs, but still lags behind sambamba throughout (and is slower, anyway). CPU usage by requested CPUs

Update 1

Realtime vs. CPUs and mem per thread (samtools sort)

My first plots (below) were showing duration on the z-axis, which is the time from job submission to completion rather than job start to completion. That's why the larger memory requests were much longer in the original plots. Duration is still a useful measure, but I'm now including real time, which is from start to completion.

Judging solely by real time (above), there is a clear gain between 5 & 7 CPUs (~10 minutes, on average when compared to the same mem-per-thread).

More data coming, including samtools compiled against libdeflate and sambamba.

Original post

I made time to semi-formally look at this. I'm glad I did because the results surprised me. I don't answer all of my questions directly, but these results are good enough to get me going in the right direction.

My code and results are in my GitHub repository (Samtools_sort_optimization_test) where you will also find an interactive plot you can download to see the results more clearly (see SAMTOOLS_SORT/samtools_sort_CPU_and_memory_comparison.tar.gz in the repository). It's hard to truly appreciate the 3D plotly plot until you start rotating it.

NOTES:

  1. Even though this is from only three samples, I think my results are accurate. I welcome any feedback in case there's a bug.
  2. In my experience, samtools often exceeds its allotted memory (at least on a SLURM scheduler). i.e., the job gets killed by SLURM. To avoid this, I requested an additional 20% of memory to provide a buffer for samtools. I don't know if this discrepancy is because of SLURM or samtools, though.
  3. These results are using samtools compiled against zlib. We are preparing a version of samtools compiled against libdeflate, per @user172818's suggestion. I will add those results once completed. I may also compare against sambamba, per @acvill's suggestion.
  4. These results are related to my other question about the best sorting/merging method. I will post those results in that question and provide a brief summary here.

tl;dr (too long, didn't read)

Primary conclusions based on the plots below

  1. Optimal CPUs (results as expected): There's a significant gain with some additional CPUs, but it levels off around 7 CPUs (total, not 'additional'; i.e., samtools sort -@ 6). I thought we might still see some nice gains above 7, but this is probably what most would expect (e.g., similar to what @user172818 suggested).
  2. Optimal memory per thread (results NOT as expected): There is very little to gain with more than 1GB of memory (per thread); that surprised me. There are some performance gains up until ~7GB per thread (~2-4 minutes faster). What really surprised me, however, is that performance dramatically decreases as memory per thread goes >7GB. The updated plot (above) shows there is some gain with >1GB mem per thread. When running lots of samples at once (hundreds to 1000), mem per thread seems to play a much larger role (data coming).
  3. Effect from sorting status (results as expected): Largely in an attempt to do this as quickly as possible (laziness?), I used three samples, where one was from an aligned and unsorted .bam file, and the other two were from .bam files that had been aligned and sorted, but I then collated them (samtools collate), rather than re-aligning them. Unsurprisingly, the collated samples (not the average use case) sorted slightly faster than the unsorted sample (average use case).

Initial opinion

  1. "Small" sample set: For a small sample set, I would probably use 7 total CPUs and 7GB per thread.
  2. "Large" sample set: I will probably use 7 CPUs (maybe 5) with only 1GB memory per thread. That will allow the scheduler to keep as many jobs going as possible.

samtools sort: duration vs. CPUs and mem per thread

As mentioned above, an interactive version of this plot is available on my GitHub.

Angle 1 Duration vs. CPUs and mem per thread (samtools sort)

Angle 2 Duration vs. CPUs and mem per thread (samtools sort)

Additional details

Analyses were run on University of Kentucky's MCC Cluster, which uses AMD 7702P processors, using samtools v.1.14

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.