0
$\begingroup$

I need to store a large number of variants with their annotations. I need to import dbSNP, as well as Clinvar, and ExAC etc. Also I have sample variant data from patients. I need to be able to search on genomic regions as well as annotations. And it needs to be scalable. What would be a preferred data storage technology for such a case? What is the industry using? It needs to be accessible by APIs/REST. We are using ElasticSearch, but that is proving to be expensive in terms of servers needed.

Something similar to this has been discussed in The state, limitations and comparisons of large variant stores, but that post is a little old, I am sure there has been progress since then.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The question you linked is about storing genotypes. Your question is about keeping site information, which is different and much easier.

As to solutions, you can parse the information you want and store them in a standard SQL database. You can use UCSC binning for region queries. Modern SQL databases all provide R-tree indices, which can be adapted for region query and is probably better these days.

Improve this answer
$\endgroup$
2
  • $\begingroup$ A standard SQL database will be slow for such large amounts of data. I will be filtering on annotations as well as regions. $\endgroup$
    – user2689782
    Commented May 30, 2019 at 17:51
  • 1
    $\begingroup$ @user2689782 Have you tried? SQLite for example easily handles tens of millions of variants at my hand. UCSC/EnsEMBL keep the entire dbSNP in MySQL as well. The most efficient way to deal with annotations highly depends on the type of annotations and queries. It is more about schema design. $\endgroup$
    – user172818
    Commented May 30, 2019 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.