0
$\begingroup$

What's the best/easiest way for accessing .bam/.cram files from S3?

I have .bam/.bai .cram/.crai and .bed + .gff files that I want to make available easily to others so they can browse in IGV and other solutions from their computers. There seem to be different solutions after cursory Googling, what would you recommend?

Here is what I've found so far:

IGV:

Samtools: samtools directly supports AWS

The full URL format is:

    s3[+SCHEME]://[ID[:SECRET[:TOKEN]]@]BUCKET/PATH
$\endgroup$
3
$\begingroup$

You could set up a public read-only bucket in an S3 account and put your files there. URLs would look like:

https://${username}.s3.amazonaws.com/public/${filename}

For samtools, the request for a BAM file's index should be built off that URL. For IGV, I think you can specify separate, custom URLs for the data and the index resources.

Public buckets are available to all, without need for credentials. So that might be an issue to think about, with respect to billing for web requests and data transfer.

I'm still not 100% certain that samtools/tabix support https requests reliably (I have a prior question on here about that), but that might be one approach.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Doo you have your heart set on AWS, or are you open to using Google Cloud Platform (GCP)?

If you're open to using GCP, there are some advantages. For example GATK can read and write directly to and from Google storage buckets, and IGV can use your login credentials to access data in Google storage as described here:

https://googlegenomics.readthedocs.io/en/latest/use_cases/browse_genomic_data/igv.html

Note that if you make the data public, by default you're going to get billed if people download it. Not sure how that's handled on AWS, but on GCP you can set the bucket (or just the file) to be "requester-pays", ie the person who downloads is the person who pays for it:

https://cloud.google.com/storage/docs/requester-pays

In general there are several ways to interact with Google buckets.

Disclosure: I work at the Broad Institute, which uses a lot of GCP infrastructure; I do not get any compensation from Google Cloud, but I recently wrote a book about Genomics in the Cloud that uses GCP.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.