15
$\begingroup$

I am working with over a million (long) reads, and aligning them to a large genome. I am considering running my alignment jobs in parallel, distributing horizontally across hundreds of nodes rather than trying to run a single job with dozens of cores.

I would like to merge the sorted BAM files together for further downstream analysis. What is the most efficient way to do so while maintaining a valid file header and taking advantage of the fact that the input bam files are already sorted?

$\endgroup$
14
$\begingroup$

samtools merge merged.bam *.bam is efficient enough since the input files are sorted. You can get a bit faster with sambamba and/or biobambam, but they're not typically already installed and IO quickly becomes a bottleneck anyway.

$\endgroup$
3
  • $\begingroup$ Any idea of how this compares to bamtools merge? $\endgroup$ – Scott Gigante May 19 '17 at 7:18
  • $\begingroup$ Samtools tends to be a bit slower than other tools since it's the reference implementation, so I'd expect bamtools to be a bit faster. $\endgroup$ – Devon Ryan May 19 '17 at 7:20
  • 1
    $\begingroup$ Bamtools is usually slower because it is not optimized and now unmaintained. Sambamba etc are developed with a specific goal to be faster. $\endgroup$ – user172818 May 19 '17 at 11:29
8
$\begingroup$

Merging sorted files is a linear operation, so any well-implemented tools that do it will do it with approximately the same efficiency. So samtools merge (use the most up-to-date version, as there have been improvements in merge header handling in the 1.3.x and 1.4.x versions), picard MergeSamFiles, etc.

These tools need to hold all the input BAM files open simultaneously, so depending on how many hundred of input files you have you may run into the system limit on open file descriptors. Use ulimit to maximise this first; if there are still too many, you may need to merge the first 500 files, then merge the next 500 into that, etc. Samtools does not do this internally; I'm not sure whether any of the other merge implementations do.

$\endgroup$
2
  • $\begingroup$ Maybe merging stepwise as you mention in your second paragraph could speed up things as well as one doesn't hit IO limitations? $\endgroup$ – sargas May 19 '17 at 7:23
  • 1
    $\begingroup$ No, some of the data is now being written and read more than once, so it must be slower. With crafty use of pipes and process substitution (samtools merge … <(samtools merge …) …) this can be minimised and avoid hitting disk, so the slowdown can be trivial (at the cost of using a few more cores). $\endgroup$ – John Marshall May 19 '17 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.