1
$\begingroup$

With tabix one can index a VCF file for rapid variant retrieval based on genomic position. I'm wondering if there are any tools that will index a VCF file to allow rapid retrieval using rsIDs and/or other metadata? I'm aware of awk/grep/vcftools one-liners for this purpose, but I'd like to avoid scanning a huge VCF each time I need to retrieve the coordinates of a new batch of rsIDs.

$\endgroup$
2
  • $\begingroup$ Is re-sorting the VCF files feasible? It's possible to use Tabix on generic tab-delimited files, so sorting and indexing by the RSID column might work. $\endgroup$ Jul 2, 2020 at 15:37
  • $\begingroup$ I attempted this, but it failed on most large VCFs I tried. $\endgroup$ Jul 21, 2020 at 11:46

4 Answers 4

2
$\begingroup$

I am not aware of any widespread index formats for query-by-QNAME for BAM or query-by-ID for VCF. I thought there were historical samtools{-devel,-help} threads to the effect of “in principle one could index a named-sorted BAM file, but there's insufficient demand so no-one's ever implemented it”, but I can't find any just now.

Moreover note that VCF files are by definition sorted by genomic position, so such an index would just be a hash table of rsID to file offset or genomic position.

If you are using dbSNP, probably you have a dbSNP database or API that you can use to query by rsID and get genomic position back. So you can retrieve your VCF contents by rsID in two stages: do that dbSNP query, and use its results to find the variants in your VCF file by position.

$\endgroup$
2
$\begingroup$

high-perf-bio.

I present my high-perf-bio project, which completely solves the problem of fast extraction of anything values from VCF, BED and unformatted data.

Preparation.

1/ Install MongoDB according to the official instruction for your distro. This is a little painful, but is done only once:).

2/ Install PyMongo library.

pip install pymongo

3/ Download and unpack current repository slice.

4/ cd path/to/high-perf-bio-master

Create DB with rsIDs.

python create.py -S path/to/gzipped_tables_dir -i rsIDs_field_path

Search rsIDs set in DB.

python annotate.py -S path/to/annotated_gzipped_tables_dir -D db_name -T path/to/trg_dir #-c rsIDs_column_num -f rsIDs_field_path

P.S.1. Detailed help for each component: python tool_name.py -h

P.S.2. For VCF and BED you can specify the minimum of arguments.

P.S.3. Readme in English is in the early stages of writing. If you have any questions, write to Issues.

$\endgroup$
3
  • $\begingroup$ This is not too different from Daniel's project - he is more credible on the site plus has written on the performance of his tool. Adding that information would help us compare your program to his at a glance. $\endgroup$
    – Ram RS
    Feb 24 at 15:14
  • $\begingroup$ @RamRS "plus has written on the performance of his tool." The performance of such tools directly depends on the performance of the underlying DBMS. I don't think the speed difference between MongoDB and SQLite is significant for indexed data. "This is not too different from Daniel's project" My toolkit has more functionality. It contains 10 high-level command utilities for solving typical bioinformatics data retrieval tasks. Searching rsIDs is not the only feature of high-perf-bio. $\endgroup$
    – PlatonB
    Mar 9 at 19:57
  • $\begingroup$ Sure, your tool might be a jack of all trades but for the task at hand, is your tool faster - that is, for the same dataset, does MongoDB give you a significant advantage? In any case, my suggestion is based on the fact that you're not as well knows as Daniel, so providing at least as much info as he does will only help your case. $\endgroup$
    – Ram RS
    Mar 10 at 18:46
1
$\begingroup$

Take a look at grabix. Using grabix you could implement a binary search on a VCF sorted by rsID and compressed using bgzip.

$\endgroup$
0
$\begingroup$

John Marshall said:

Moreover note that VCF files are by definition sorted by genomic position, so such an index would just be a hash table of rsID to file offset or genomic position.

I just published rsidx, a new package for indexing VCF files by rsID. As John suggests, I simply created a mapping from rsID to genomic coordinates. These are stored in an sqlite3 database.

It currently takes...several...hours to index build 151 of dbSNP on GRCh38. I'm looking into optimizing sqlite3 parameters to accelerate the indexing, as well as an alternative approach based on a two step process (populate an in-memory sqlite3 db, then dump DB to disk). But in the meantime, the search capability works great.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.