2
$\begingroup$

Related to my other question (Samtools sort: most efficient memory and thread settings for many samples on a cluster), we need to optimize samtools sort as we prepare to analyze thousands of .bam files, beginning with re-alignment, sorting, etc.

Ultimate question: what is the fastest way to sort a single sample after aligning the sample across many separate cluster jobs? i.e., after splitting .fastqs for a single sample and aligning as distinct jobs. I'm looking for recent and clear time/thread/memory comparisons.

Unless I'm missing something, samtools merge was designed exactly for the purpose of merging pre-sorted bams. An excerpt from the manpage (http://www.htslib.org/doc/samtools-merge.html): "Merge multiple sorted alignment files, producing a single sorted output file that contains all the input records and maintains the existing sort order."

Is it fastest to:

  1. samtools sort each 'mini-bam' and then samtools merge?
  2. samtools cat the 'mini-bams' together and then samtools sort?
  3. Another option I haven't considered?

Note: I have seen the following similar questions on other forums, but they are outdated (~8 years ago) and do not answer this question directly:

  1. Best Way To Merge A Many Thousand Small Bam Files Into One Big Bam File?
  2. Merging BAM files before sorting
$\endgroup$
0

1 Answer 1

1
$\begingroup$

As part of its operations, samtools sort will split the input up into multiple temporary BAM files (which are each individually sorted) if the reads can't fit entirely within memory. Those files are merged for the final output. From the manpage:

The sorted output is written to standard output by default, or to the specified file (out.bam) when -o is used. This command will also create temporary files tmpprefix.%d.bam as needed when the entire alignment data cannot fit into memory (as controlled via the -m option).

Given this, I would expect that a concatenated, unsorted BAM will take more time to process than merging sorted BAM files, because in addition to creating the concatenated BAM file, the splitting would need to be done again (assuming the reads don't fit in memory) prior to the final merge carried out by samtools sort.

I appreciate that this is not a quantified answer; it's just my guess based on what I have observed to happen when doing my own analyses.

$\endgroup$
1
  • $\begingroup$ It's funny. Now that you mention that, it seems so obvious! Great point. I may still quantify it just to see the difference. $\endgroup$ Commented Feb 11, 2022 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.