Strategy for merging many VCFs

Question

Can anyone suggest a strategy for speeding up VCF merging?

I have ~44,000 single-sample VCFs that I am trying to merge into a multi-sample VCF with bcftools merge, but the job keeps timing out on the cluster (just timed out after 4 days). I could obviously try running it for a couple of weeks, but I was hoping there is a better strategy someone can suggest?

The exact command I am running is

bcftools merge --no-version -m all -i 'DP:avg,RO:avg,AO:avg' -F + -o merged.vcf --file-list merge.fofn

Happy to try any other tool that will give the same merge results.

Answer 1

We created this over the pandemic when we needed to merge loads of VCFs which all have the same header and first few columns (ie same variants)

https://github.com/iqbal-lab-org/ivcfmerge

That page gives detailed instructions but i'll paste a couple of examples of how you can use it

== In Python =4.1.1 If the number of input files is small (can be opened all at once)

from contextlib import ExitStack
from ivcfmerge import ivcfmerge

filenames = [...]    # List/iterator of relative/absolute paths to input files
output_path = '...'  # Where to write the merged VCF to

with ExitStack() as stack:
    files = map(lambda fname: stack.enter_context(open(fname)), filenames)
    with open(output_path) as outfile:
        ivcfmerge(files, outfile)

=4.1.2 If the number of input files is big (cannot be opened all at once) from ivcfmerge import ivcfmerge_batch

filenames = [...]    # List/iterator of relative/absolute paths to input files
output_path = '...'  # Where to write the merged VCF to
batch_size = 1000    # How many files to open and merge at once

ivcfmerge_batch(filenames, output_path, batch_size)

Pretty sure we initially did batched bcftools merge, I can't honestly remember why we moved to this. Thankfully not having to do these merges very often

Answer 2

I ended up using an iterative approach (inspired by this post), which took 1.5 hours - so a massive speed up (and that was just the sequential version).

Note: fd is analogous to find, but way nicer to use.

First, merge subsets of n VCFs

# size of the subsets
n=200
# create file of filenames for each subset
fd -e vcf . path/to/vcfs/ | split -l $n --additional-suffix '.fofn' - subset_vcfs

-e vcf tells fd to find all files with a .vcf extension, you can obviously change this to .bcf/.vcf.gz etc. if necessary. split splits the list of all VCFs files into subsets of size -l (200) and creates the file of filenames with the prefix subset_vcfs. It will add a random suffix to this - e.g. subset_vcfsav. Plus the additional suffix .fofn. See the split manual for all options.

Now iterate through these subsets and merge the VCFs listed in them.

Subset merge option 1 - sequential

This just merges the VCFs one at a time

for sub in *.fofn;
do
    echo "Merging subset ${sub}..."
    out_vcf="merge.${sub}.bcf"
    bcftools merge --file-list "$sub" -o "$out_vcf"
    bcftools index "$out_vcf"
done

Subset merge option 2 - parallel

You can easily parallelise the previous step using fd on the subset files. See this post for more details. Or basic bash for loop parallelisation.

Create a script (merge_subset.sh) of what we want to run on each subset

#!/usr/bin/env bash
sub="$1"
echo "Merging subset ${sub}..."
out_vcf="merge.${sub}.bcf"
bcftools merge --file-list "$sub" -o "$out_vcf"
bcftools index "$out_vcf"

Now we run this script with each subset file of filenames using fd

# number of processes to use. remove this option to use all available
jobs=4
fd -j $jobs -e fofn -x bash merge_subset.sh '{}'

Final merge

Now we merge all of these VCFs into our final VCF

ls merge.*.bcf > merged.fofn
bcftools merge -o merged.bcf --file-list merged.fofn

Answer 3

Perform a tree based merge of the files.

• Sort the files into groups that can easily be merged in a measurable amount of time.
• Merge each group into a separate Variant Call Format (VCF) file
  • Has the benefit that files can also be grouped by type / source / run
  • Further benefit that small merges can be checked to ensure you like the output before proceeding to the next stage.
  • Further benefit that there are also subgroup files available to be distributed or shared separately.
  • Downside, extra storage space until you have a final product.
• Merge subfiles upward in stages until you arrive at a complete 44,000 entry file
  • If you're happy with the earlier test results, process can be automated with relatively simple iterative shell scripts.
  • Possibly faster, as the bulk merge needs to load every single file into memory, and you may be hitting a capacity bottleneck.
  • Worked in Fluid Dynamics for years, and often ran into a similar issue. Could run many small cases representing a large case faster than running the large case, because large case would overwhelm memory and transfer capacity.
  • In some clusters you may also be causing nodes to silently crash, which then stalls the entire process without a visible notification.