5
$\begingroup$

I have a several VCFs which are VCF which only contain information by chromosome. That is, there's a chromosome 1 VCF (with only chr1), a chromosome 2 VCF (with only chr2), etc.

I checked to make sure that these VCFs were valid via VCFtools, i.e.

$ vcf-validator chr1.vcf

which works---these are valid VCFs I was given.

Now, I would like to combine these VCFs into one VCF.

I naïvely tried the following cat operation:

$ cat chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf > total_chroms.vcf

This doesn't work properly though

$ vcf-validator total_chroms.vcf
The header tag 'contig' not present for CHROM=chr1. (Not required but highly recommended.)
Could not parse the line, wrong number of columns: [##fileformat=VCFv4.2\n]
 at /path/vcftools-0.1.14/perl/Vcf.pm line 172, <__ANONIO__> line 191016.
	    Vcf::throw('Vcf4_2=HASH(0x1ae7208)', 'Could not parse the line, wrong number of columns: [##filefor...') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 335
	    Vcf::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 3457
	    Vcf4_1::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 2574
	    VcfReader::run_validation('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 60
	    main::do_validation('HASH(0x16ada68)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 14
$

What tools are available to merge these VCF's together into a total VCF?

$\endgroup$
9
$\begingroup$

I would recommend bcftools concat. You can't just cat them together because each file has a header section. The bcftools command will handle all that for you. Each vcf file must be sorted prior to calling concat

bcftools concat -o total_chroms.vcf chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf

| improve this answer | |
$\endgroup$
  • $\begingroup$ It's a bit odd. vcf-validator chr1.vcf shows no problems. I believe the output of bcftools concat -o chroms12.vcf chr1.vcf chr2.vcf shows an error: FIXME: sequence name chr1 in chr1.vcf $\endgroup$ – ShanZhengYang Jan 15 '18 at 3:27
  • $\begingroup$ @ShanZhengYang the errors seem to indicate that you have chr1 in the CHROM field for some rows, but no corresponding line in the header. Can you check if that is true in the chr1.vcf file? Maybe the validator somehow is missing this error? $\endgroup$ – juod Jan 15 '18 at 15:09
  • $\begingroup$ @juod The VCF header looks correct to me. #CHROM POS ID REF ALT QUAL FILTER INFO $\endgroup$ – ShanZhengYang Jan 15 '18 at 19:06
  • $\begingroup$ @juod "but no corresponding line in the header" Maybe I don't understand this. Besides the code pasted above, there is no other "chrom" information in the header $\endgroup$ – ShanZhengYang Jan 15 '18 at 23:41
  • 1
    $\begingroup$ According to the VCF spec samtools.github.io/hts-specs/VCFv4.1.pdf there should be header lines for each chromosome this is from the first example in the linked documents - ##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x> $\endgroup$ – Bioathlete Jan 16 '18 at 0:58
2
$\begingroup$

The best tool for the job is probably bcftools as suggested by Bioathlete, but you can also do this manually. You just need to collect all header lines from all vcf files, remove any duplicates, and then print all the headers + the actual data into the new one:

grep '^##' chr*vcf | sort | uniq > all.vcf
grep -m1 '^#CHR' chr1.vcf >> all.vcf  ## Get the chr header line 
grep -v '^#' chr*vcf >> all.vcf
| improve this answer | |
$\endgroup$
  • $\begingroup$ I appear to get a malformed VCF header with the above commands $\endgroup$ – ShanZhengYang Jan 16 '18 at 1:57
  • $\begingroup$ @ShanZhengYang malformed how? Could you edit your question and show us an example of your files? $\endgroup$ – terdon Jan 16 '18 at 8:26
  • $\begingroup$ So, I see the above grep commands give the same behavior with all of my standard VCFs. I'll get something like this in all.vcf: chr10.vcf:##contig=<ID=chr10,length=133797422> chr10.vcf:##fileDate=20121011 chr10.vcf:##fileformat=VCFv4.2 chr10.vcf:##reference=path/reference.fa chr10.vcf:##source=foobar chr11.vcf:##contig=<ID=chr10,length=133797422> chr11.vcf:##fileDate=20121011 chr11.vcf:##fileformat=VCFv4.2 chr11.vcf:##reference=path/reference.fa chr11.vcf:##source=foobar .... $\endgroup$ – ShanZhengYang Jan 16 '18 at 18:32
1
$\begingroup$

use picard GatherVcf: http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs

Gathers multiple VCF files from a scatter operation into a single VCF file. Input files must be supplied in genomic order and must not have events at overlapping positions.

| improve this answer | |
$\endgroup$
  • $\begingroup$ The error I get here is Exception in thread "main" picard.PicardException: In order to index the resulting VCF input VCFs must contain ##contig lines.. It is true, my "by chromosome"-VCFs do not contain a ##contig line in the header...but I'm not sure that would matter $\endgroup$ – ShanZhengYang Jan 16 '18 at 1:53
  • $\begingroup$ Edit above: Apparently this issue has been discussed before: github.com/samtools/bcftools/issues/326 Question: how could I add these contig lines if I don't know this information?I know that these VCFs are from hg38. Could I simply add the line ##contig=<ID=chr1,length=248956422>, etc. to the VCFs in question? I would just take the chromosome lengths from hg38 $\endgroup$ – ShanZhengYang Jan 16 '18 at 2:06
  • $\begingroup$ "but I'm not sure that would matter" it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary broadinstitute.github.io/picard/command-line-overview.html $\endgroup$ – Pierre Jan 16 '18 at 16:38
  • $\begingroup$ "it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary". Ah, I understand now. Thanks for the help! $\endgroup$ – ShanZhengYang Jan 16 '18 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.