With tabix one can index a VCF file for rapid variant retrieval based on genomic position. I'm wondering if there are any tools that will index a VCF file to allow rapid retrieval using rsIDs and/or other metadata? I'm aware of awk/grep/vcftools one-liners for this purpose, but I'd like to avoid scanning a huge VCF each time I need to retrieve the coordinates of a new batch of rsIDs.
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Is re-sorting the VCF files feasible? It's possible to use Tabix on generic tab-delimited files, so sorting and indexing by the RSID column might work. $\endgroup$– afaulconbridgeCommented Jul 2, 2020 at 15:37
-
$\begingroup$ I attempted this, but it failed on most large VCFs I tried. $\endgroup$– Daniel StandageCommented Jul 21, 2020 at 11:46
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
I am not aware of any widespread index formats for query-by-QNAME for BAM or query-by-ID for VCF. I thought there were historical samtools{-devel,-help} threads to the effect of “in principle one could index a named-sorted BAM file, but there's insufficient demand so no-one's ever implemented it”, but I can't find any just now.
Moreover note that VCF files are by definition sorted by genomic position, so such an index would just be a hash table of rsID to file offset or genomic position.
If you are using dbSNP, probably you have a dbSNP database or API that you can use to query by rsID and get genomic position back. So you can retrieve your VCF contents by rsID in two stages: do that dbSNP query, and use its results to find the variants in your VCF file by position.
$\begingroup$
$\endgroup$
3
high-perf-bio.
I present my high-perf-bio project, which, for the most part, solves the problem of fast extraction of anything values from VCF, BED and unformatted data.
Preparation.
1/ Run the automatic downloading of the toolkit and installation of the dependencies.
wget https://github.com/PlatonB/high-perf-bio/archive/refs/heads/master.zip && unzip -o master.zip && rm master.zip && cd high-perf-bio-master && bash __install_3rd_party.sh
2/ Reboot.
3/ cd path/to/high-perf-bio-master
Create DB with rsIDs.
python create.py -S path/to/gzipped_tables_dir -i rsIDs_field_path
Search rsIDs set in DB.
python annotate.py -S path/to/annotated_gzipped_tables_dir -D db_name -T path/to/trg_dir #-c rsIDs_column_num -f rsIDs_field_path
P.S.1. Detailed help for each high-perf-bio component: python tool_name.py -h
P.S.2. For VCF and BED you can specify the minimum of arguments.
P.S.3. Readme in English is in the early stages of writing. If you have any questions, write to Issues.
-
$\begingroup$ This is not too different from Daniel's project - he is more credible on the site plus has written on the performance of his tool. Adding that information would help us compare your program to his at a glance. $\endgroup$– Ram RSCommented Feb 24, 2022 at 15:14
-
1$\begingroup$ @RamRS "plus has written on the performance of his tool." The performance of such tools directly depends on the performance of the underlying DBMS. I don't think the speed difference between MongoDB and SQLite is significant for indexed data. "This is not too different from Daniel's project" My toolkit has more functionality. It contains 10 high-level command utilities for solving typical bioinformatics data retrieval tasks. Searching rsIDs is not the only feature of high-perf-bio. $\endgroup$– PlatonBCommented Mar 9, 2022 at 19:57
-
$\begingroup$ Sure, your tool might be a jack of all trades but for the task at hand, is your tool faster - that is, for the same dataset, does MongoDB give you a significant advantage? In any case, my suggestion is based on the fact that you're not as well knows as Daniel, so providing at least as much info as he does will only help your case. $\endgroup$– Ram RSCommented Mar 10, 2022 at 18:46
$\begingroup$
$\endgroup$
Take a look at grabix. Using grabix you could implement a binary search on a VCF sorted by rsID and compressed using bgzip.
$\begingroup$
$\endgroup$
Moreover note that VCF files are by definition sorted by genomic position, so such an index would just be a hash table of rsID to file offset or genomic position.
I just published rsidx, a new package for indexing VCF files by rsID. As John suggests, I simply created a mapping from rsID to genomic coordinates. These are stored in an sqlite3 database.
It currently takes...several...hours to index build 151 of dbSNP on GRCh38. I'm looking into optimizing sqlite3 parameters to accelerate the indexing, as well as an alternative approach based on a two step process (populate an in-memory sqlite3 db, then dump DB to disk). But in the meantime, the search capability works great.