2
$\begingroup$

I am running bcftools merge to create a single .vcf.gz file out of a collection of chromosomal .vcf.gz files from the 1000g project.

The merge process has been running for a couple of days, and I want to estimate how much longer it will take. The file is a growing .vcf.gz that is being bgzip -c compressed from a Linux pipe:

bcftools merge [...] | some processing and filtering | bgzip -c > final.file.vcf.gz

How can I know at which point is the file currently, to predict how much longer it will take to complete the merge?

Is there a way to do something like:

tail --with-some-magic final.file.vcf.gz | bgzip -d -c

So that I can read at which chromosome is currently on?

Is there a way to guesstimate the final size of the merge file and compare from the sum of sizes of the original files? E.g. the sum of compressed sizes originally (before filtering and munging) is 20G:

find $PWD -name "ALL.chr*.vcf.gz" | xargs ls -l | awk '{print $5}' | csvtk summary -H -f 1:sum | numfmt --to-si

This gives 20G, but the merge file is at 27G and still running.

Any ideas?

$\endgroup$

2 Answers 2

5
$\begingroup$

27GB is not huge. You can zcat the file and see the last few lines. It will take ~10 minutes. It is theoretically possible to peek a bgzf file, but no such tools are available.

By the way,

I am running bcftools merge to create a single .vcf.gz file out of a collection of chromosomal .vcf.gz files from the 1000g project.

Probably you should use bcftools concat, not bcftools merge. merge combines VCFs from different samples. concat concatenates VCFs for individual chromosomes; each input VCF contains the same set of samples.

$\endgroup$
1
  • $\begingroup$ Ah! I didn't know about bcftools concat, I will definitely try that. The bcftools merge has been running for several days and no idea what's taking that long. $\endgroup$
    – 719016
    Jun 28, 2019 at 17:11
0
$\begingroup$

You could find which Chromosome the merged file is currently writing on by:

bcftools query -f'%CHROM\n' final.file.vcf.gz | tail

Note that also you might want to try multithreading the command that you are using to write the file, as bcftools is capable of using several threads for the compression of output files. E.g.:

bcftools merge -Ob --threads 32 file1.vcf.gz file2.vcf.gz > mergedFile.vcf.gz 
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.