extracting last few lines of a growing bgzip compressed .vcf.gz file

Question

I am running bcftools merge to create a single .vcf.gz file out of a collection of chromosomal .vcf.gz files from the 1000g project.

The merge process has been running for a couple of days, and I want to estimate how much longer it will take. The file is a growing .vcf.gz that is being bgzip -c compressed from a Linux pipe:

bcftools merge [...] | some processing and filtering | bgzip -c > final.file.vcf.gz

How can I know at which point is the file currently, to predict how much longer it will take to complete the merge?

Is there a way to do something like:

tail --with-some-magic final.file.vcf.gz | bgzip -d -c

So that I can read at which chromosome is currently on?

Is there a way to guesstimate the final size of the merge file and compare from the sum of sizes of the original files? E.g. the sum of compressed sizes originally (before filtering and munging) is 20G:

find $PWD -name "ALL.chr*.vcf.gz" | xargs ls -l | awk '{print $5}' | csvtk summary -H -f 1:sum | numfmt --to-si

This gives 20G, but the merge file is at 27G and still running.

Any ideas?

Answer 1

27GB is not huge. You can zcat the file and see the last few lines. It will take ~10 minutes. It is theoretically possible to peek a bgzf file, but no such tools are available.

By the way,

I am running bcftools merge to create a single .vcf.gz file out of a collection of chromosomal .vcf.gz files from the 1000g project.

Probably you should use bcftools concat, not bcftools merge. merge combines VCFs from different samples. concat concatenates VCFs for individual chromosomes; each input VCF contains the same set of samples.

Answer 2

You could find which Chromosome the merged file is currently writing on by:

bcftools query -f'%CHROM\n' final.file.vcf.gz | tail

Note that also you might want to try multithreading the command that you are using to write the file, as bcftools is capable of using several threads for the compression of output files. E.g.:

bcftools merge -Ob --threads 32 file1.vcf.gz file2.vcf.gz > mergedFile.vcf.gz