16
$\begingroup$

I am working with over a million (long) reads, and aligning them to a large genome. I am considering running my alignment jobs in parallel, distributing horizontally across hundreds of nodes rather than trying to run a single job with dozens of cores.

I would like to merge the sorted BAM files together for further downstream analysis. What is the most efficient way to do so while maintaining a valid file header and taking advantage of the fact that the input bam files are already sorted?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
15
$\begingroup$

samtools merge merged.bam *.bam is efficient enough since the input files are sorted. You can get a bit faster with sambamba and/or biobambam, but they're not typically already installed and IO quickly becomes a bottleneck anyway.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Any idea of how this compares to bamtools merge? $\endgroup$
    – Scott Gigante
    Commented May 19, 2017 at 7:18
  • $\begingroup$ Samtools tends to be a bit slower than other tools since it's the reference implementation, so I'd expect bamtools to be a bit faster. $\endgroup$
    – Devon Ryan
    Commented May 19, 2017 at 7:20
  • 1
    $\begingroup$ Bamtools is usually slower because it is not optimized and now unmaintained. Sambamba etc are developed with a specific goal to be faster. $\endgroup$
    – user172818
    Commented May 19, 2017 at 11:29
8
$\begingroup$

Merging sorted files is a linear operation, so any well-implemented tools that do it will do it with approximately the same efficiency. So samtools merge (use the most up-to-date version, as there have been improvements in merge header handling in the 1.3.x and 1.4.x versions), picard MergeSamFiles, etc.

These tools need to hold all the input BAM files open simultaneously, so depending on how many hundred of input files you have you may run into the system limit on open file descriptors. Use ulimit to maximise this first; if there are still too many, you may need to merge the first 500 files, then merge the next 500 into that, etc. Samtools does not do this internally; I'm not sure whether any of the other merge implementations do.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Maybe merging stepwise as you mention in your second paragraph could speed up things as well as one doesn't hit IO limitations? $\endgroup$
    – sargas
    Commented May 19, 2017 at 7:23
  • 1
    $\begingroup$ No, some of the data is now being written and read more than once, so it must be slower. With crafty use of pipes and process substitution (samtools merge … <(samtools merge …) …) this can be minimised and avoid hitting disk, so the slowdown can be trivial (at the cost of using a few more cores). $\endgroup$
    – John Marshall
    Commented May 19, 2017 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.