3
$\begingroup$

I want to download the original BAM files that the authors had uploaded to SRA. Normally, I would just use sam-dump, but the files are having issues that seem related to this issue. Since according to the entry, AWS S3 also hosts the original BAM files, I thought I could download these directly.

Example SRA Entry

enter image description here

NCBI documentation implies that I can't download this directly, but I can freely copy to other AWS locations within the region. To this end, I created my own S3 bucket (mm-mneuron) and am now trying to copy from the SRA bucket to mine. Here's what I try:

import boto3
import botocore

s3 = boto3.resource('s3')

bam_file = {
  'Bucket': 'sra-pub-src-6',
  'Key': 'SRR5253957/RPI25_0.bam'
}

my_bucket = s3.Bucket('mm-mneuron')

my_bucket.copy(bam_file, 'RPI25_0.bam')

This fails with:

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

That is, it sounds like I can't access the SRA bucket. I've tested downloading and uploading to my bucket, so I know I have write permissions. Not sure what else to try here.

How can I access the SRA data on S3?

Improve this question
$\endgroup$
3
  • $\begingroup$ It seems like is not a public bucket. You should download inside an (SRA AMI)[ncbi.nlm.nih.gov/sra/docs/sra-aws-download/] from sra-pub-src-6 to your instance, and then, in you local terminal download from SRA AMI in EC2 to your computer. $\endgroup$
    – zorbax
    Apr 17 '20 at 23:57
  • $\begingroup$ @zorbax yes that's what I did end up doing (see answer), but there is something specifically wrong with the deposits I'm trying to download. In general, are SRA buckets only accessible through SRA tools? i.e., one has to run an EC2 instance? It seems like direct transfers across S3 buckets would an obvious interface to provide. $\endgroup$
    – merv
    Apr 18 '20 at 4:21
  • $\begingroup$ You only have access to the public data sets with s3, and the key prj_phs710EA_test.ngc in EC2 only bring acces to 1000 Genomes data. biostars.org/p/393172 $\endgroup$
    – zorbax
    Apr 23 '20 at 13:35
1
$\begingroup$

A member of the SRA submission staff pointed out that using

prefetch --type all SRR5253957

will download the original files. In this case, it means running the above within an EC2 instance colocated with the S3 bucket (so, us-east-1) and having installed and configured SRA Toolkit to work from AWS (as per this documentation).

Unfortunately, the particular files I am concerned with are not currently accessible, but this should generically work in most situations.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.