3
$\begingroup$

My whole (professional) life I am downloading sequences from SRA and ENA databases. Long time ago, I used prefetch from sra-tools, but later I switched to simply using wget from ftp servers of the two respective servers (used to work for both SRA and ENA). But it seems that SRA ftp server does not have the same data in there anymore and I kind of worry about reproducibility of my old pipelines - so what happened to sra ftp server?

The paths that used to work were formatted as follows

ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra

but it seems sra-instant directory is not accessible anymore. Did the files move? Or were the access been just restricted?

The alternative workaround is to dl the files using prefetch instead of wgetting. Prefetch is part of sra-tools and available on conda and it's not such a bad solution. I guess I am mostly curious what happened to the ftp server and if anyone knows what is coming next, is ENA going to restrict the access too some time soon?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
3
$\begingroup$

SRA moved all of the data to the cloud per https://ncbiinsights.ncbi.nlm.nih.gov/2020/02/24/sra-cloud/

You can still download data for free using sratoolkit as described in the official documentation here: https://www.ncbi.nlm.nih.gov/sra/docs/sradownload/

You can download the data using sratoolkit without having to pay cloud providers. There are some user-pays data on the cloud but for FASTQ files you don't have to pay.

For documentation on how to download data from the cloud, see https://www.ncbi.nlm.nih.gov/sra/docs/sra-aws-download/

Improve this answer
$\endgroup$
3
  • $\begingroup$ Cool and thanks $\endgroup$
    – M__ ♦
    Commented Feb 8, 2022 at 17:40
  • $\begingroup$ This is certainly useful and relevant, but does not truly answers the question. The article does not say anything about discontinuing the ftp servers, rather offering a new way to access the data. $\endgroup$
    – Kamil S Jaron
    Commented Feb 8, 2022 at 18:31
  • 1
    $\begingroup$ I don't think downloading from the FTP was never an officially advertised method. The entire transition from NCBI FTP to cloud is not a concern for the users of sratoolkit, the preferred method to download data. $\endgroup$
    – vkkodali
    Commented Feb 8, 2022 at 18:37
2
$\begingroup$

NCBI appeared at least in part to shift onto Google Cloud buckets, one previous OP raised a question about access via GCP (I think the question was closed). Fundamentally, traditional FTP is not sustainable against data growtch. GCP is accessed via gsutil. It would be good if you could contact NCBI and post their official response here. They will response when their database provision is in a state of flux.

sratoolkit appears to be a front-end to GCPs gsutil. Just to point out, it would be very unusual for NCBI to charge the user for e.g. direct gsutil download, because usually the provider will charge e.g. NCBI on a per download basis. This will likely work out cheaper overall for NCBI.

Just to formalise my bit, if SRA are hosting on a GCP bucket, Google will not permit ftp access which is at least part the reason behind your question.

They do have their own ftp equivalent called gsutil which will permit direct download if you know the bucket name. gsutil is part of the ClouD SDK package and can be downloaded via conda/anaconda conda install gcloud-sdk. This will give you command line access if you know the bucket names. There is a past post on this BTW.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ You don't really need to use gsutil. For example, if you run srapath SRR13397412 you will see the path as https://storage.googleapis.com/nih-sequence-read-archive/run/SRR13397412/SRR13397412 which can be downloaded using wget. $\endgroup$
    – vkkodali
    Commented Feb 8, 2022 at 19:50
  • $\begingroup$ update: conda install -c conda-forge google-cloud-sdk $\endgroup$
    – envs_h_gang_5
    Commented Jun 16, 2022 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.