2
$\begingroup$

I am working on developing a tool to classify covid variants. So I am trying to download SARS-CoV-2 sequence data along with lineage annotation from Covid-19 Data Portal (I could not find a better source that provides both sequence and lineage annotated data.)

Since it is a very large data (given 4,050,433 samples), my idea is to first download the metadata (.tsv file) with accession ids and respective lineages from the portal, filter them to keep only the variants I am interested in, and pass the accession_ids as a text file that I would like to download to CDP File downloader as suggested in the portal.

However, downloading metadata .tsv file from the portal is taking forever (download speed is in bytes sometimes). I have tried downloading directly as a file, using wget and also using curl. I am not sure if there is a way to download it quicker since it is losing connection in between and restarting the download. Just as a side note, the sequence data is downloading pretty fast at 4+Mbps.

I hope my question is clear. Any suggestions/tips on how can I make this happen is very much appreciated. Thanks in advance.

$\endgroup$
0

1 Answer 1

0
$\begingroup$

30MB isn't a lot of data at all, given the issue is the EBI servers and not your institutes provision - you would have noticed this before on other projects if it was an institute issue. It therefore suggests someone else is hogging the EBI capacity and there is no provision to bypass this.

The best solution is contact them directly

[email protected]

... and state the download size and download speed.

There could be a bot sucking up the database (that is what it sounds like), which would be easy to write given FTP is permitted. They will have the ability to block an IP and NCBI have done this before albeit a long time ago.

Finally filling the user experience form would help EBI which is here. However the issue you described suggests escalation. NCBI are very responsive and I would be surprised if EBI did not respond quickly.

Good luck with the project, sounds interesting.

Generic response below if its just traditional over-demand for the service.


I suspect that the beautiful database EBI has provided (and it is amazing) has not anticipated the level of demand for the database size. They are far from being alone.

Modern web architectures, such as Spotify, dedicate enormous effort to matching demand with service provision, notably using Kubernetes. When the server is overloaded they will simply duplicate their core database onto additional machines, which they will be buying in from AWS/GCP. It can even be done dynamically without intervention. NCBI are certainly heading in that direction.

If the EBI wrote this under Kubernetes (it could be a monolith however), the limiting factor is the budget in computer resource allocation (again you can increase Kubernetes provision dynamically). However, if its a monolith this will not be easy and again ultimately someone has pay, i.e. EBI.

Regardless, the fact you can use FTP means EBI are hosting their own service, GCP doesn't permit this and AWS is almost certainly similar. This would mean they have to physically provided additional in-house machines. I visited EBI some time ago and at that time the then director went to some lengths describing how focused EBI were on matching their in-house storage capacity to exponential rise in sequence data. Sudden popularity of a webserver is not trivial issue if it hasn't been anticipated.


Alternatives Using the NCBI service COVID service is the alternative if there is an urgent need to get the project moving. You may also wish to use GISAID here.

If you want to use the NCBI service, I provided the detailed instructions here: How to programatically download SARS-CoV-2 fasta from NCBI/Genbank via API?

It is however important to understand the possible underlying causes from EBI. I do recall the then EBI director stating their biggest issue was a user simply switching to Genbank, which from their point of view isn't cool given the amount effort going into service provision.

$\endgroup$
4
  • $\begingroup$ Thank you for the detailed explanation and alternatives. I have found an alternative to EBI from the answer you have suggested. It was quick to download from NCBI. I will report/inform EBI about the data speed issues and update their response here. $\endgroup$
    – Rajesh M
    Mar 10, 2022 at 22:46
  • $\begingroup$ You are welcome. It is a pity for EBI because I think they try hard, however it is also a learning curve for them $\endgroup$
    – M__
    Mar 11, 2022 at 0:00
  • $\begingroup$ I have gotten a response from EBI saying, "They are aware of these downloading issues with few files and they have allocated my ticket to a colleague and help me resolve it asap" $\endgroup$
    – Rajesh M
    Mar 11, 2022 at 12:32
  • $\begingroup$ That's good, fingers crossed. They will be sensitive about you switching to Genbank. $\endgroup$
    – M__
    Mar 11, 2022 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.