I'm trying to download three WGS datasets from the SRA that are each between 60 and 100GB in size. So far I've tried:
- Fetching the .sra files directly from NCBI's ftp site
- Fetching the .sra files directly using the aspera command line (
- Using the SRA toolkit's
It's excruciatingly slow. I've had three
fastqdump processes running in parallel now for approximately 18 hours. They're running on a large AWS instance in the US east (Virginia) region, which I figure is about as close to NCBI as I can get. In 18 hours they've downloaded a total of 33GB of data. By my calculation that's ~500kb/s. They do appear to still be running - the fastq files continue to grow and their timestamps continue to update.
At this rate it's going to take me days or weeks just to download the datasets. Surely the SRA must be capable of moving data at higher rates that this? I've also looked, and unfortunately the datasets I'm interested have not been mirrored out to ENA or the Japanese archive, so it looks like I'm stuck working with the SRA.
Is there a better way to fetch this data that wouldn't take multiple days?