Background: We're increasingly needing some way of storing lots of variant data associated with lots of subjects: think clinical trials and hospital patients, looking for disease-causing or relevant genes. A thousand subjects is where we'd start, there's talk of millions on the horizon. With various genomic medicine initiatives, this is likely a wider need.
The problem: While there's plenty of platforms out there, it's a rapidly evolving field. It's difficult to get a feel for how (and if) they perform and how they line up against each other:
- What's scalable and can handle a lot of data? What sort of limits?
- What's robust and not a teetering pile of hacked-together components?
- What has a large community behind it and is actually used widely?
- What makes for easy access and search from another service? (Commandline, REST or software APIs)
- What sort of variants they handle?
- What sort of parameters can be used in searching?
Solutions I've seen so far:
- BigQ: used with i2b2, but its wider use is unclear
- OpenCGA: looks the most developed, but I've heard complaints about the size of data it spits out
- Using BigQuery over a Google Genomics db: doesn't seem to be a general solution
- Gemini: recommended but is it really scalable and accessible from other services?
- SciDb: a commercial general db
- Whatever platform DIVAS & RVD run on: which may not be freely available
- Several graphical / graph genome solutions: We (and most other people) are probably not dealing with graph genome data at the moment, but is this a possible solution?
- Roll your own: Frequently recommended but I'm sceptical this is a plausible solution for a large dataset.
Anyone with experience give a review or high-level guide to this platform space?