Selecting 65000 SNPs where AF is close to 0.5 in all or most populations

Question

I am evaluating the tool somalier (https://github.com/brentp/somalier) and I need to create a list of about 65,000 SNPs where the allele frequency (AF) is as close to 0.5 as possible across the most representative set of populations possible with today's data.

My alignments are in GRCh38, so I need a vcf input file for somalier also in GRCh38, not the one that is available in the somalier website, which is in GRCh37.

I am starting as input with the vcfs from the 1000genomes project in GRCh38 positions, and I would like to come up with a filtering strategy that: (1) Sorts all SNPs by the AF being close to 0.5 in as many populations as possible. (2) Takes the first 65,000 SNPs from that list and produces a vcf file from them.

My simplistic strategy so far has been to take the global AF value if it's between 0.3-0.7 and print it if it's a SNP with this perl one-liner:

gunzip -c my.vcf.gz | grep -v '#' | perl -lne 'if ($_ =~ /\;AF=0\.[34567]/) { if ($_ =~ /\w+\t\d+\t\S+\t\w\t\w\t/) {print $_} }'

Any recommendations?

Answer 1

Just use bcftools view for filtering:

$ bcftools view -i 'AF>0.3 && AF<0.7' input.vcf.gz > output.vcf

To truncate this list to 65,000 SNPs count the header lines, sum them to the number of SNPs you like to have and use head -n.

$ bcftools view -h input.vcf.gz|wc -l
255

$ bcftools view -i 'AF>0.3 && AF<0.7' input.vcf.gz | head -n 65255 > output.vcf

Answer 2

Using bcftools is the right way, but you could also greatly simplify the command you're using. You can do the same thing with a single zgrep call:

zgrep -P '^[^#]+\t\d+\t\S+\t\w\t\w\t.*;AF=0\.[3-7]' my.vcf.gz

And to limit to the first 65000 results:

zgrep -m 65000 -P '^[^#]+\t\d+\t\S+\t\w\t\w\t.*;AF=0\.[3-7]' my.vcf.gz

Finally, to keep the header of the original vcf unchanged, you could do:

zgrep -m 65000 -P '^(#|[^#]+\t\d+\t\S+\t\w\t\w\t.*;AF=0\.[3-7])' my.vcf.gz

I also suggest you use hg19 instead of hg38 data since the best available source for frequencies, the gnomAD database, is based on hg19 and the tool you want to test also provides a site file for hg19. But that may not be possible for you.

If your VCF file has multiple AF entries for different sub-populations and you want to use those, you'll probably need to script something. We can't really help with that unless you show us some example input lines and start writing the basic script yourself though.