# Trying to download a large number of files from ENA programmatically

PPS: I then tried decreasing to 6 threads, and again the same issue. One file will lag. I don't have this issue if I directly download from Firefox, and I get a significant speed up, but I'm trying to automate the process so downloading from Firefox isn't an option.

For reference, here's the code:

import queue
import time

start = time.perf_counter()

def __init__(self, name):
self.name = name
def run(self):
process_queue()

def process_queue():
while True:
try:
url = my_queue.get(block=False)
filename = url.split('/')[-1]
with closing(request.urlopen(url)) as r:
with open(filename, 'wb') as f:
shutil.copyfileobj(r, f)
except queue.Empty:
return

# setting up variables
urls = [
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/000/SRR8240860/SRR8240860_1.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/000/SRR8240860/SRR8240860_2.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/001/SRR8240861/SRR8240861_1.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/001/SRR8240861/SRR8240861_2.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/002/SRR8240862/SRR8240862_1.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/002/SRR8240862/SRR8240862_2.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/003/SRR8240863/SRR8240863_1.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/003/SRR8240863/SRR8240863_2.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/004/SRR8240864/SRR8240864_1.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/004/SRR8240864/SRR8240864_2.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/005/SRR8240865/SRR8240865_1.fastq.gz',
'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR824/005/SRR8240865/SRR8240865_2.fastq.gz',
]

# filling the queue
my_queue = queue.Queue()

for url in urls:
my_queue.put(url)

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

• Their web server may be throttling simultaneous connections. Generally, multiple incoming requests from one client is not good. Jul 12 '20 at 21:25
• @burger I think you're right. I added a time.sleep(1) after thread.start() and it seems like it's better now. Though I'll have to test some more Jul 12 '20 at 21:31
• @burger hmm so I performed some tests and it still seems like I have an issue. This is despite the fact that I have time.sleep(60) Firefox is still a lot faster at downloading the files than ftp Jul 13 '20 at 0:18