Skip to main content
added 346 characters in body
Source Link
gringer
  • 15.1k
  • 5
  • 24
  • 83

Related to my other question (Samtools sort: most efficient memory and thread settings for many samples on a cluster), we need to optimize samtools sort as we prepare to analyze thousands of .bam files, beginning with re-alignment, sorting, etc.

Ultimate question: what is the fastest way to sort a single sample after aligning the sample across many separate cluster jobs? i.e., after splitting .fastqs for a single sample and aligning as distinct jobs. I'm looking for recent and clear time/thread/memory comparisons.

Unless I'm missing something, samtools merge was designed exactly for the purpose of merging pre-sorted bams. An excerpt from the manpage (http://www.htslib.org/doc/samtools-merge.html): "Merge multiple sorted alignment files, producing a single sorted output file that contains all the input records and maintains the existing sort order."

Is it fastest to:

  1. samtools sort each 'mini-bam' and then samtools merge?
  2. samtools cat the 'mini-bams' together and then samtools sort?
  3. Another option I haven't considered?

Note: I have seen the following similar questions on other forums, but they are outdated (~8 years ago) and do not answer this question directly:

  1. Best Way To Merge A Many Thousand Small Bam Files Into One Big Bam File?
  2. Merging BAM files before sorting

Related to my other question (Samtools sort: most efficient memory and thread settings for many samples on a cluster), we need to optimize samtools sort as we prepare to analyze thousands of .bam files, beginning with re-alignment, sorting, etc.

Ultimate question: what is the fastest way to sort a single sample after aligning the sample across many separate cluster jobs? i.e., after splitting .fastqs for a single sample and aligning as distinct jobs. I'm looking for recent and clear time/thread/memory comparisons.

Is it fastest to:

  1. samtools sort each 'mini-bam' and then samtools merge?
  2. samtools cat the 'mini-bams' together and then samtools sort?
  3. Another option I haven't considered?

Note: I have seen the following similar questions on other forums, but they are outdated (~8 years ago) and do not answer this question directly:

  1. Best Way To Merge A Many Thousand Small Bam Files Into One Big Bam File?
  2. Merging BAM files before sorting

Related to my other question (Samtools sort: most efficient memory and thread settings for many samples on a cluster), we need to optimize samtools sort as we prepare to analyze thousands of .bam files, beginning with re-alignment, sorting, etc.

Ultimate question: what is the fastest way to sort a single sample after aligning the sample across many separate cluster jobs? i.e., after splitting .fastqs for a single sample and aligning as distinct jobs. I'm looking for recent and clear time/thread/memory comparisons.

Unless I'm missing something, samtools merge was designed exactly for the purpose of merging pre-sorted bams. An excerpt from the manpage (http://www.htslib.org/doc/samtools-merge.html): "Merge multiple sorted alignment files, producing a single sorted output file that contains all the input records and maintains the existing sort order."

Is it fastest to:

  1. samtools sort each 'mini-bam' and then samtools merge?
  2. samtools cat the 'mini-bams' together and then samtools sort?
  3. Another option I haven't considered?

Note: I have seen the following similar questions on other forums, but they are outdated (~8 years ago) and do not answer this question directly:

  1. Best Way To Merge A Many Thousand Small Bam Files Into One Big Bam File?
  2. Merging BAM files before sorting
Source Link
Mark Ebbert
  • 1.4k
  • 12
  • 22

Samtools sort: most efficient approach to sort a single sample after aligning split .fastq files

Related to my other question (Samtools sort: most efficient memory and thread settings for many samples on a cluster), we need to optimize samtools sort as we prepare to analyze thousands of .bam files, beginning with re-alignment, sorting, etc.

Ultimate question: what is the fastest way to sort a single sample after aligning the sample across many separate cluster jobs? i.e., after splitting .fastqs for a single sample and aligning as distinct jobs. I'm looking for recent and clear time/thread/memory comparisons.

Is it fastest to:

  1. samtools sort each 'mini-bam' and then samtools merge?
  2. samtools cat the 'mini-bams' together and then samtools sort?
  3. Another option I haven't considered?

Note: I have seen the following similar questions on other forums, but they are outdated (~8 years ago) and do not answer this question directly:

  1. Best Way To Merge A Many Thousand Small Bam Files Into One Big Bam File?
  2. Merging BAM files before sorting