# How to concatenate "by chromosome"-VCFs?

I have a several VCFs which are VCF which only contain information by chromosome. That is, there's a chromosome 1 VCF (with only chr1), a chromosome 2 VCF (with only chr2), etc.

I checked to make sure that these VCFs were valid via VCFtools, i.e.

$vcf-validator chr1.vcf  which works---these are valid VCFs I was given. Now, I would like to combine these VCFs into one VCF. I naïvely tried the following cat operation: $ cat chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf > total_chroms.vcf


This doesn't work properly though

$vcf-validator total_chroms.vcf The header tag 'contig' not present for CHROM=chr1. (Not required but highly recommended.) Could not parse the line, wrong number of columns: [##fileformat=VCFv4.2\n] at /path/vcftools-0.1.14/perl/Vcf.pm line 172, <__ANONIO__> line 191016. Vcf::throw('Vcf4_2=HASH(0x1ae7208)', 'Could not parse the line, wrong number of columns: [##filefor...') called at /path/vcftools-0.1.14/perl/Vcf.pm line 335 Vcf::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at /path/vcftools-0.1.14/perl/Vcf.pm line 3457 Vcf4_1::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at /path/vcftools-0.1.14/perl/Vcf.pm line 2574 VcfReader::run_validation('Vcf4_2=HASH(0x1ae7208)') called at /path/vcftools-0.1.14//bin/vcf-validator line 60 main::do_validation('HASH(0x16ada68)') called at /path/vcftools-0.1.14//bin/vcf-validator line 14$


What tools are available to merge these VCF's together into a total VCF?

I would recommend bcftools concat. You can't just cat them together because each file has a header section. The bcftools command will handle all that for you. Each vcf file must be sorted prior to calling concat

bcftools concat -o total_chroms.vcf chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf

• It's a bit odd. vcf-validator chr1.vcf shows no problems. I believe the output of bcftools concat -o chroms12.vcf chr1.vcf chr2.vcf shows an error: FIXME: sequence name chr1 in chr1.vcf Jan 15 '18 at 3:27
• @ShanZhengYang the errors seem to indicate that you have chr1 in the CHROM field for some rows, but no corresponding line in the header. Can you check if that is true in the chr1.vcf file? Maybe the validator somehow is missing this error?
– juod
Jan 15 '18 at 15:09
• @juod The VCF header looks correct to me. #CHROM POS ID REF ALT QUAL FILTER INFO Jan 15 '18 at 19:06
• @juod "but no corresponding line in the header" Maybe I don't understand this. Besides the code pasted above, there is no other "chrom" information in the header Jan 15 '18 at 23:41
• According to the VCF spec samtools.github.io/hts-specs/VCFv4.1.pdf there should be header lines for each chromosome this is from the first example in the linked documents - ##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x> Jan 16 '18 at 0:58

The best tool for the job is probably bcftools as suggested by Bioathlete, but you can also do this manually. You just need to collect all header lines from all vcf files, remove any duplicates, and then print all the headers + the actual data into the new one:

grep '^##' chr*vcf | sort | uniq > all.vcf
grep -m1 '^#CHR' chr1.vcf >> all.vcf  ## Get the chr header line
grep -v '^#' chr*vcf >> all.vcf

• I appear to get a malformed VCF header with the above commands Jan 16 '18 at 1:57
• @ShanZhengYang malformed how? Could you edit your question and show us an example of your files? Jan 16 '18 at 8:26
• So, I see the above grep commands give the same behavior with all of my standard VCFs. I'll get something like this in all.vcf: chr10.vcf:##contig=<ID=chr10,length=133797422> chr10.vcf:##fileDate=20121011 chr10.vcf:##fileformat=VCFv4.2 chr10.vcf:##reference=path/reference.fa chr10.vcf:##source=foobar chr11.vcf:##contig=<ID=chr10,length=133797422> chr11.vcf:##fileDate=20121011 chr11.vcf:##fileformat=VCFv4.2 chr11.vcf:##reference=path/reference.fa chr11.vcf:##source=foobar .... Jan 16 '18 at 18:32

• The error I get here is Exception in thread "main" picard.PicardException: In order to index the resulting VCF input VCFs must contain ##contig lines.. It is true, my "by chromosome"-VCFs do not contain a ##contig line in the header...but I'm not sure that would matter Jan 16 '18 at 1:53
• Edit above: Apparently this issue has been discussed before: github.com/samtools/bcftools/issues/326 Question: how could I add these contig lines if I don't know this information?I know that these VCFs are from hg38. Could I simply add the line ##contig=<ID=chr1,length=248956422>, etc. to the VCFs in question? I would just take the chromosome lengths from hg38 Jan 16 '18 at 2:06