2
$\begingroup$

It appears that AWS and GCP host SRA data and it's beneficial to grab the data from that source when running on GCP for example.

Given an SRR accession like SRR1929796 https://trace.ncbi.nlm.nih.gov/Traces/sra/?run=SRR1929796

Is there a metadata API / source that can tell me which google cloud bucket to retrieve the data from like they show on the website?

gs://sra-pub-crun-4/SRR1929796/SRR1929796.1

It also sounds like fastq-dump and fasterq-dump can be configured with Google Cloud credentials to download that for me.

I'm just wondering is there a way to determine the google cloud storage bucket path it's in gs://sra-pub-crun-4?

$\endgroup$

1 Answer 1

0
$\begingroup$

It sounds like it to me too. I have never directly accessed NCBIs stuff via Google buckets and I don't know what permissions they are setting for direct access.

So ... I briefly checked it out

gsutil ls gs://sra-pub-crun-4

and 'yes' its possible but ...

BadRequestException: 400 Bucket is a requester pays bucket but no user project provided.

'Requester pay' luckily I wasn't logged in :-)

What the message implies given access to your (my) GCP account and they'll bill you (me). I don't think it will be much, particularly for a 'list' request, however I ain't gonna find out for a data set I don't need. I do doubt I'm going to get billed for 'list' request, however I've never paid 'list' request I prefer to keep it that way. In contrast, NCBI's front-end is free and thats what they want, because they could of course set the bucket so they pay.

So in answer the question, if you wanna pay go ahead ... dial up a GCP account link it to pay.


Here's my first answer

Please note I'm not really certain what information the OP seeks here, e.g. gs:// locations type stuff. If the question is where's the SRS database on every gs:// path ... I don't know.

I think the OP wants an API ... Google Cloud certainly use APIs by the bucket load (no pun intended but I thought it was funny). I don't use them because they are excessively verbose and I prefer SDK and gcloud tools, of which gsutil is one.

It also sounds like fastq-dump and fasterq-dump can be configured with Google Cloud credentials to download that for me.

If I have understood correctly and that is all you want then sure

gsutil cp gs://sra-pub-crun-4/SRR1929796/SRR1929796.1 .

will do it, if permitted (or in hindsight prepared to pay) and there will be an API version of this.


Background

Installation via Google's Cloud SDK, via https://cloud.google.com/storage/docs/gsutil_install#install

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.