3
$\begingroup$

Can anyone suggest a strategy for speeding up VCF merging?

I have ~44,000 single-sample VCFs that I am trying to merge into a multi-sample VCF with bcftools merge, but the job keeps timing out on the cluster (just timed out after 4 days). I could obviously try running it for a couple of weeks, but I was hoping there is a better strategy someone can suggest?

The exact command I am running is

bcftools merge --no-version -m all -i 'DP:avg,RO:avg,AO:avg' -F + -o merged.vcf --file-list merge.fofn

Happy to try any other tool that will give the same merge results.

$\endgroup$
0

3 Answers 3

4
$\begingroup$

We created this over the pandemic when we needed to merge loads of VCFs which all have the same header and first few columns (ie same variants)

https://github.com/iqbal-lab-org/ivcfmerge

That page gives detailed instructions but i'll paste a couple of examples of how you can use it

== In Python =4.1.1 If the number of input files is small (can be opened all at once)

from contextlib import ExitStack
from ivcfmerge import ivcfmerge

filenames = [...]    # List/iterator of relative/absolute paths to input files
output_path = '...'  # Where to write the merged VCF to

with ExitStack() as stack:
    files = map(lambda fname: stack.enter_context(open(fname)), filenames)
    with open(output_path) as outfile:
        ivcfmerge(files, outfile)

=4.1.2 If the number of input files is big (cannot be opened all at once) from ivcfmerge import ivcfmerge_batch

filenames = [...]    # List/iterator of relative/absolute paths to input files
output_path = '...'  # Where to write the merged VCF to
batch_size = 1000    # How many files to open and merge at once

ivcfmerge_batch(filenames, output_path, batch_size)

Pretty sure we initially did batched bcftools merge, I can't honestly remember why we moved to this. Thankfully not having to do these merges very often

$\endgroup$
1
  • $\begingroup$ Thanks Zam. Unfortunately I fail the first two assumptions listed in the README. $\endgroup$ Commented Apr 18, 2023 at 22:50
3
$\begingroup$

I ended up using an iterative approach (inspired by this post), which took 1.5 hours - so a massive speed up (and that was just the sequential version).

Note: fd is analogous to find, but way nicer to use.

First, merge subsets of n VCFs

# size of the subsets
n=200
# create file of filenames for each subset
fd -e vcf . path/to/vcfs/ | split -l $n --additional-suffix '.fofn' - subset_vcfs

-e vcf tells fd to find all files with a .vcf extension, you can obviously change this to .bcf/.vcf.gz etc. if necessary. split splits the list of all VCFs files into subsets of size -l (200) and creates the file of filenames with the prefix subset_vcfs. It will add a random suffix to this - e.g. subset_vcfsav. Plus the additional suffix .fofn. See the split manual for all options.

Now iterate through these subsets and merge the VCFs listed in them.

Subset merge option 1 - sequential

This just merges the VCFs one at a time

for sub in *.fofn;
do
    echo "Merging subset ${sub}..."
    out_vcf="merge.${sub}.bcf"
    bcftools merge --file-list "$sub" -o "$out_vcf"
    bcftools index "$out_vcf"
done

Subset merge option 2 - parallel

You can easily parallelise the previous step using fd on the subset files. See this post for more details. Or basic bash for loop parallelisation.

Create a script (merge_subset.sh) of what we want to run on each subset

#!/usr/bin/env bash
sub="$1"
echo "Merging subset ${sub}..."
out_vcf="merge.${sub}.bcf"
bcftools merge --file-list "$sub" -o "$out_vcf"
bcftools index "$out_vcf"

Now we run this script with each subset file of filenames using fd

# number of processes to use. remove this option to use all available
jobs=4
fd -j $jobs -e fofn -x bash merge_subset.sh '{}'

Final merge

Now we merge all of these VCFs into our final VCF

ls merge.*.bcf > merged.fofn
bcftools merge -o merged.bcf --file-list merged.fofn
$\endgroup$
1
$\begingroup$

Perform a tree based merge of the files.

  • Sort the files into groups that can easily be merged in a measurable amount of time.
  • Merge each group into a separate Variant Call Format (VCF) file
    • Has the benefit that files can also be grouped by type / source / run
    • Further benefit that small merges can be checked to ensure you like the output before proceeding to the next stage.
    • Further benefit that there are also subgroup files available to be distributed or shared separately.
    • Downside, extra storage space until you have a final product.
  • Merge subfiles upward in stages until you arrive at a complete 44,000 entry file
    • If you're happy with the earlier test results, process can be automated with relatively simple iterative shell scripts.
    • Possibly faster, as the bulk merge needs to load every single file into memory, and you may be hitting a capacity bottleneck.
    • Worked in Fluid Dynamics for years, and often ran into a similar issue. Could run many small cases representing a large case faster than running the large case, because large case would overwhelm memory and transfer capacity.
    • In some clusters you may also be causing nodes to silently crash, which then stalls the entire process without a visible notification.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.