0
$\begingroup$

I need to store a large number of variants with their annotations. I need to import dbSNP, as well as Clinvar, and ExAC etc. Also I have sample variant data from patients. I need to be able to search on genomic regions as well as annotations. And it needs to be scalable. What would be a preferred data storage technology for such a case? What is the industry using? It needs to be accessible by APIs/REST. We are using ElasticSearch, but that is proving to be expensive in terms of servers needed.

Something similar to this has been discussed in The state, limitations and comparisons of large variant stores, but that post is a little old, I am sure there has been progress since then.

$\endgroup$
1
$\begingroup$

The question you linked is about storing genotypes. Your question is about keeping site information, which is different and much easier.

As to solutions, you can parse the information you want and store them in a standard SQL database. You can use UCSC binning for region queries. Modern SQL databases all provide R-tree indices, which can be adapted for region query and is probably better these days.

$\endgroup$
2
  • $\begingroup$ A standard SQL database will be slow for such large amounts of data. I will be filtering on annotations as well as regions. $\endgroup$ May 30 '19 at 17:51
  • 1
    $\begingroup$ @user2689782 Have you tried? SQLite for example easily handles tens of millions of variants at my hand. UCSC/EnsEMBL keep the entire dbSNP in MySQL as well. The most efficient way to deal with annotations highly depends on the type of annotations and queries. It is more about schema design. $\endgroup$
    – user172818
    May 30 '19 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.