A reliable fetcher of short read using SRA/ENA accession

Question

I am trying to build a workflow that gets data automatically from databases of sequencing reads (SRA - sequence read archive, ENA - european nucleotide archive). Till now I was pulling everything from SRA and my script have worked, but now I encountered a first entry that is only in ENA (ERR2135453), not in SRA. And I found that my download script has its limitations.

My bash script looks like this (requires aspera ascp and fastq-dump) and I wanted to make changes that are commented now

#!/bin/bash

ACCESION=$1
OPENSSH=/my/part/to/key/asperaweb_id_dsa.openssh

# if [ ${ACCESION:0:3} == SRA ]; then
    [email protected]:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/
# else
#    URL=ena_url
# fi    

ascp -QT -i $OPENSSH $URL

fastq-dump data/"$ACCESION".sra --outdir data/ --split-files --gzip
rm data/"$ACCESION".sra

The problem is in getting ENA url. Unlike SRA url that I can easily get from the accession ([email protected]:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/), the ftp server of ENA is bit more confusing. The url corresponding to ERR2135453 accession is ftp://ftp.sra.ebi.ac.uk//vol1/err/ERR213/003/ERR2135453. Where 003 there comes from? Also not all reads are in err subdirectory.

Is there an easy way how to download reads from ENA knowing just the accession number?

--- edit ---

I found on the relevant ENA site about downloading data using aspera that ftp urls should be possible to derive only from accession also in ENA.

Submitted read data files are organised by submission accession number under vol1/ directory in ftp.sra.ebi.ac.uk:

ftp://ftp.sra.ebi.ac.uk/vol1//

where contains the first 6 letters and numbers of the SRA Submission accession.

For example, the files submitted in the SRA Submission ERA007448 are available at: ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/.

The reason why there is this strange 003 will probably be that it's quite recent submission and it has not been properly categorized yet (other accessions I tried were on "expected" urls).

However, the problem still persist. Is there a way how to fetch data of new submissions using accessions only?

Answer 1

Here are two python functions I use for getting stuff from ENA based on an accession:

import os
import itertools
from urllib.request import urlopen

def fetch_ENA_files(accession):
    """Get the names of the files matching the ENA accession"""

    # Get the paths to the files
    url = "http://www.ebi.ac.uk/ena/data/warehouse/filereport?accession=%s&result=read_run&fields=fastq_ftp" \
          % accession

    try:
        paths = urlopen(url).readlines()[1:]
    except:
        E.debug("couldn't access %s" % url)
        raise

    paths = list(itertools.chain.from_iterable(
        [p.strip().split(";") for p in paths]))
    filenames = [os.path.basename(x) for x in paths]
    dl_paths = [os.path.join(*x.split("/")[1:]) for x in paths]
    return filenames, dl_paths

def fetch_ENA(dl_path, outdir, protocol="ascp"):
    """Fetch fastq from ENA given accession"""

    if protocol == "ascp":
        statement = """ascp -QT -P33001 -l 100M -i $ASCP_KEY_PATH
                       [email protected]:/%(dl_path)s %(outdir)s """ % locals()
    elif protocol == "http":
        fn = os.path.basename(dl_path)
        outfile = os.path.join(outdir, fn)
        statement = "wget -O %(outfile)s ftp://ftp.sra.ebi.ac.uk:/%(dl_path)s" % locals()

    return statement

fetch_ENA_files queries the ENA database for the names of all the files associated with a accession and returns a tuple of lists. The first entry in the tuple contains the filenames (which you might need later in your application), the second contains a list of paths on the ENA servers where those files can be found.

fetch_ENA(dl_path, outdir, protocol) takes a download path and downloads it to outdir using with ftp or ascp.

You might use it like:

filenames, paths = fetch_ENA_files("ERR2135453")
for dlp in paths:
    bash_statement = fetch_ENA(dlp, "downloads", "ascp")
    os.system(bash_statement)

(we don't we actaully use our modules to send it to the cluster)

Answer 2

Apparently there is no 100% reliable solution, but in the most cases, you should be able to use the enaBrowserTools python scripts (enaDataGet) to quite reliably download data.

The run ERR2135453 is a special case because the sample submission (ERS1939590) for it has been cancelled. The solution for these reads is to download the data manually from the ftp location.

This answer is based on kind assistance of Suran from ebi helpdesk.