I am downloading a large number of files from the ENA in Python using Multithreading. As an example, to be more concrete, I tried downloading 12 files simultaneously using 12 threads. I noticed that 10 files download relatively quickly, while 2 lag behind. Even when the 10 files are done downloading, the speed at which the other 2 files download is very slow. If I then try downloading these 2 files again, the speed will be a lot better. So what exactly is happening and how can I fix this?

PS: I tried decreasing the thread count to 8, so 8 files were downloading simultaneously, and I had 1 file that was lagging the rest again.

PPS: I then tried decreasing to 6 threads, and again the same issue. One file will lag. I don't have this issue if I directly download from Firefox, and I get a significant speed up, but I'm trying to automate the process so downloading from Firefox isn't an option.

For reference, here's the code:

import queue
import threading
import time

start = time.perf_counter()

class MyThread(threading.Thread):
    def __init__(self, name):
        self.name = name
    def run(self):
        print('Starting thread %s.' % self.name)
        print('Exiting thread %s.' % self.name)
def process_queue():
    while True:
            url = my_queue.get(block=False)
            filename = url.split('/')[-1]
            with closing(request.urlopen(url)) as r:
                with open(filename, 'wb') as f:
                    shutil.copyfileobj(r, f)
        except queue.Empty:
# setting up variables
urls = [

# filling the queue
my_queue = queue.Queue()

for url in urls:

# initializing and starting num_threads threads
num_threads = 8
threads = []

for i in range(num_threads):
    thread = MyThread(i)

for thread in threads:

for thread in threads:
finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')
  • $\begingroup$ Their web server may be throttling simultaneous connections. Generally, multiple incoming requests from one client is not good. $\endgroup$
    – burger
    Jul 12 '20 at 21:25
  • $\begingroup$ @burger I think you're right. I added a time.sleep(1) after thread.start() and it seems like it's better now. Though I'll have to test some more $\endgroup$ Jul 12 '20 at 21:31
  • $\begingroup$ @burger hmm so I performed some tests and it still seems like I have an issue. This is despite the fact that I have time.sleep(60) Firefox is still a lot faster at downloading the files than ftp $\endgroup$ Jul 13 '20 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.