Downloading multiple SRA files from several SRA accession IDs does not work

Question

I am trying to download multiple SRA files located in several SRA accessions. Some of my accession numbers are as follows:

SRR1002678
SRR1158898
SRR1158907
SRR1158950
SRR1181152
SRR1181155
SRR1181196
SRR1181204
SRR1181205
SRR1181208
SRR1269497
SRR2100227
SRR2100419
SRR2100830
SRR2101344
SRR671725
SRR671726
SRR671740
SRR671759
SRR671760
SRR671761
SRR671779
SRR671782
SRR671815
SRR671821
SRR671842
SRR671862
SRR671863
SRR671864
SRR671865
.............

I tried the following command:

for i in {SRR1002678..SRR1184123}; do wget ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR1*/$i; done

but this did not work.

I tried prefetch as well:

sort -u SRR.txt | parallel -j 4 "prefetch {}"

where SRR.txt is the list of my SRA IDs. However, with this command only a few files are downloaded as *.sra files, while with several other SRA IDs I get the following error:

2018-11-15T06:17:20 prefetch.2.8.2: 1) Downloading 'SRR1183962'...
2018-11-15T06:17:20 prefetch.2.8.2:  Downloading via https...
2018-11-15T06:20:21 prefetch.2.8.2: 1) 'SRR1183962' was downloaded successfully
2018-11-15T06:20:27 prefetch.2.8.2: 'SRR1183962' has 0 unresolved dependencies
2018-11-15T06:20:38 prefetch.2.8.2: 'SRR1183962' has no remote vdbcache

2018-11-15T06:18:59 prefetch.2.8.2: 1) Downloading 'SRR1184034'...
2018-11-15T06:18:59 prefetch.2.8.2:  Downloading via https...
2018-11-15T06:20:40 prefetch.2.8.2: 1) failed to download SRR1184034
2018-11-15T06:20:40 prefetch.2.8.2 sys: libs/kns/unix/syssock.c:606:KSocketTimedRead: timeout exhausted while reading file within network system module - mbedtls_ssl_read returned -76 ( NET - Reading information from the socket failed )
2018-11-15T06:20:40 prefetch.2.8.2 int: libs/kns/unix/syssock.c:606:KSocketTimedRead: timeout exhausted while reading file within network system module -

I do not have a range of SRA files to download but quite random SRA files to download. What am I doing wrong? Any help is highly appreciated.

Answer 1

First of all, the brace expansion you tried to use isn't valid. While bash can expand {1..3} to 1 2 3, it has no way of knowing how to expand {SRR1002678..SRR1184123}:

$ echo {1..3}
1 2 3
$ echo {SRR1002678..SRR1184123}
{SRR1002678..SRR1184123}

Perhaps you meant SRR{1002678..1184123}? But there's no need for this, you already have the accessions in a file and can read that. I see that the FTP server you are accessing has organized the data like this:

sra/sra-instant/reads/ByRun/sra/SRR/SRRNNN/SRRNNNXXXX/SRRNNNXXXX.sra

Where NNN are the 1st three digits of the accession. So, for example, the file for accession SRR1002678 is at:

ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR118/SRR1181152/SRR1181152.sra

With this in mind, you can write a little script that extracts the URL from each accession:

while read i; do 
    n=$(echo "$i" | sed -E 's/(SRR...).*/\1/'); 
    wget ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/$n/$i/$i.sra
done < SRR.txt

The only trick is the sed command which simply looks for SRR and the three next characters and removes everything else on the line (it "captures" the found pattern and then replaces the line with what was captured (\1)).

Answer 2

For me it works best to combine cat, while and official download tool fasterq-dump.

cat your_accession_list.txt | while read i
do  
fasterq-dump ${i} <options>
done

As commented above, your_accession_list.txt should have Linux endings. Link to fasterq-dump is here.