I am running Picard MarkDuplicates with the following parameters below. On the file described, it takes about 41.6Gb of RAM memory and about 20-25 minutes to compute (only uses 1 core AFAICS).

java -Xmx54890m -jar picard.jar MarkDuplicates \
    I=/home/dnanexus/in/sorted_bam/TST73-86-2IC_S16_L00.bml.GRCh38.karyo.bam \
    O=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.deduplicated.bam \
    M=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.duplication_metrics \

What equivalent free/open source software (samtools?, sambamba?) can I use to obtain the same deduplicated bam file but at a lower cost, considering the instance used here has 8 cores but only one of them is being used, and has about 68Gb of RAM out of which it's using 46Gb odd at peak.

I think previous versions of samtools/sambamba didn't behave the same way as Picard MarkDuplicates with the parameters above, but I would like to know which recent versions would give me the same output.

  • 1
    $\begingroup$ I assume you only want free (in terms of price, at least) software, right? Sentieon have a tool for this that is orders of magnitude faster (and I would guess uses less RAM but I'd have to check) but you need to pay for it. $\endgroup$
    – terdon
    Dec 11 '17 at 15:58
  • $\begingroup$ Correct, open source. $\endgroup$
    – 719016
    Dec 11 '17 at 16:05
  • $\begingroup$ I done no benchmark, but I guess samblaster is bit faster (by my subjective perception of time). $\endgroup$
    – Kamil S Jaron
    Dec 11 '17 at 20:24
  • $\begingroup$ You should specify a smaller "-Xmx". MarkDuplicate doesn't need that much memory, but seeing a large "-Xmx", java will use extra. $\endgroup$
    – user172818
    Dec 12 '17 at 15:21

As you suggested, sambamba is faster at marking duplicates than picard (it's also multithreaded). Recent versions of samtools have a rewritten duplicate marking algorithm, though I doubt it'll be as quick as sambamba. 46GB of RAM seems excessive for marking duplicates unless you're having it store the whole file in memory.

  • $\begingroup$ Does sambamba with default parameters equate to Picard MarkDuplicates? $\endgroup$
    – 719016
    Dec 12 '17 at 21:50
  • $\begingroup$ I'm not sure that any tool exactly equates (there are a lot of edge cases in duplicate marking and picard misses some things), but it's a sufficient replacement. Sambamba is also used for this purpose by a number of international consortia (I think either DEEP or IHEC, if I remember correctly). $\endgroup$
    – Devon Ryan
    Dec 12 '17 at 23:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.