2
$\begingroup$

What's the best/easiest way for accessing .bam/.cram files from S3?

I have .bam/.bai .cram/.crai and .bed + .gff files that I want to make available easily to others so they can browse in IGV and other solutions from their computers. There seem to be different solutions after cursory Googling, what would you recommend?

Here is what I've found so far:

IGV:

Samtools: samtools directly supports AWS

The full URL format is:

    s3[+SCHEME]://[ID[:SECRET[:TOKEN]]@]BUCKET/PATH
$\endgroup$
1

3 Answers 3

3
$\begingroup$

You could set up a public read-only bucket in an S3 account and put your files there. URLs would look like:

https://${username}.s3.amazonaws.com/public/${filename}

For samtools, the request for a BAM file's index should be built off that URL. For IGV, I think you can specify separate, custom URLs for the data and the index resources.

Public buckets are available to all, without need for credentials. So that might be an issue to think about, with respect to billing for web requests and data transfer.

I'm still not 100% certain that samtools/tabix support https requests reliably (I have a prior question on here about that), but that might be one approach.

$\endgroup$
1
$\begingroup$

Doo you have your heart set on AWS, or are you open to using Google Cloud Platform (GCP)?

If you're open to using GCP, there are some advantages. For example GATK can read and write directly to and from Google storage buckets, and IGV can use your login credentials to access data in Google storage as described here:

https://googlegenomics.readthedocs.io/en/latest/use_cases/browse_genomic_data/igv.html

Note that if you make the data public, by default you're going to get billed if people download it. Not sure how that's handled on AWS, but on GCP you can set the bucket (or just the file) to be "requester-pays", ie the person who downloads is the person who pays for it:

https://cloud.google.com/storage/docs/requester-pays

In general there are several ways to interact with Google buckets.

Disclosure: I work at the Broad Institute, which uses a lot of GCP infrastructure; I do not get any compensation from Google Cloud, but I recently wrote a book about Genomics in the Cloud that uses GCP.

$\endgroup$
1
$\begingroup$

I have recently open sourced a Java NIO SPI for S3 that allows direct reads from S3 using Java applications that use the NIO model. This allows applications like GATK and some parts of Picard to understand s3:// URIs.

It is used as a drop in so as long as the jar is on your classpath or in your JRE ext folder then Java will delegate nio operations to this lib when it "sees" an s3:// URI. It will use the credentials and configuration of your current profile so no additional config is require and no recompilation of an application is needed.

The github repo is https://github.com/awslabs/aws-java-nio-spi-for-s3

Disclosure: I am the author of the library, employed by the AWS Health AI Genomics Team, and have financial interests in Amazon

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.