Too many requests error when querying NCBI

Question

I keep getting a "too many requests error" when querying the NCBI SRA database, even though I'm running less than 10 requests per second, and I have an API key, which supposedly should allow me to run 10 per second.

Here is my (Python) code:

import subprocess
import concurrent.futures
import time

bioprojects = [
    "PRJNA644722",
    "PRJNA644892"
]

all_metadata_generators = []

def fetch_metadata(bioproject):
    return subprocess.run(f"esearch -db sra -query '{bioproject}[bioproject]' | efetch -format runinfo", shell=True, capture_output=True, text=True)

with concurrent.futures.ThreadPoolExecutor() as executor:
    for i in range(0, len(bioprojects), 10):
        all_metadata_generators.append(executor.map(fetch_metadata, bioprojects[i:i+10]))
        time.sleep(1)


for metadata_generator in all_metadata_generators:
    for metadata in metadata_generator:
        print(metadata)

Answer 1

The execution of your EDirect commands may be limited unless you are using an API Key, which you can create from your MyNCBI account. Your API Key can be used in one of several different ways.

In an argument to an EDirect command:

esearch -db nuccore -query 'some query' -api_key 12345

In a UNIX environment:

You can either run this just before you are running some edirect commands or add the following line to your .bashrc file.

export NCBI_API_KEY=12345

In the environment, from a Perl script:

$ENV{NCBI_API_KEY} = "12345"

Answer 2

I had the same problem. I was making a lot of requests and with the thread it blocked very quickly. But I solved the problem with multiprocessing. Pool and imap_unordered. I can even put ThreadPoolExecutor inside the function. I'm making an application in Gradio (hence the gr.Process()).

Then there are a few subtleties, if you see correctly, there's an infinite loop in process_gene if there's a 429 error or something else. If it's a 429 error it's just that the server doesn't want to because too many requests have been made, if it's another error you should stop and restart the gene or at least analyse the error better. If, for example, it's a 500 error, it's just that the server is down for maintenance, so you might as well try again.

Then I make packets from my gene list, because when in doubt, I prefer to send in packets of 50 to avoid overloading the various processor cores. And to finish everything off, I use pool.close() and pool.join() to make sure it finishes all the processes and "empties" the cores to send the next group of genes.

Now, some information about how I tested and what I have concluded. I tried it on a Windows on Windows 10, 8GB RAM, almost 10 year old i5 processor (4 cores) and I tried it on a Mac Mini M2 8GB RAM, it has 8 cores. As you can see in my code (multiprocessing.Pool(processes=multiprocessing.cpu_count())), I used all the cores. With Windows, I manage to extract 90 gene_names per minute (not with the script below, I have another script where each process_gene() makes between 4 and 5 requests in a row + web-scraping, the script below is just an example). With Windows, sometimes I still get error 429 but as there's a loop, I get through it. In theory, the Mac should be 2x faster because it has 8 cores. But as the API limits to 10 requests per second, I'm stuck at 90 gene_names per minute.

I strongly advise you to use the API key, but you're already doing that.

I hope this helps :)

import random
import time
import requests

def process_gene(gene_id):
    while True:
        url = f"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=gene&id={gene_id}&retmode=json&rettype=xml&api_key={YOUR_API_KEY}"
        response = requests.get(url)

        if response.status_code == 200:
            response_data = response.json()
            try:
                gene_info = response_data['result'][str(gene_id)]
                gene_name = gene_info['name']

                print(f"Response 200: Info for {gene_id} {gene_name} retrieved.")
                return gene_name, f"Response 200: Info for {gene_id} {gene_name} retrieved.", gene_id
            
            except Exception as e:
                print(f"Response 200: Info for {gene_id} not found.")
                return "Error 200", f"Info for {gene_id} not found.", gene_id

        elif response.status_code == 429:
            print(f"Error 429: API rate limit exceeded during get {gene_id} info, try again: {response.text}")
            time.sleep(random.uniform(0.25, 0.5))
        else:
            print(f"Error {response.status_code}: {response.text}")
            time.sleep(random.uniform(0.25, 0.5))
    

def your_function(list_gene, progress=gr.Progress()):
    result = []
    result_error = []
    gene_ids = list_gene.strip().split("\n")
    total_genes = len(gene_ids)
    progress(0.0, desc="🔄 Extracting... ⚠️ PLEASE WAIT UNTIL COMPLETION")

    gene_groups = [gene_ids[i:i + 50] for i in range(0, len(gene_ids), 50)]
    i = 0
    j = 0

    for group in gene_groups:
        j += 1
        with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
            results = pool.imap_unordered(process_gene,[gene_id for gene_id in group])

            for result_output, message, gene_id in results:
                if result_output == "Error 200":
                    result_error.append(message)
                else:
                    result.append(result_output)

                i += 1
                progress(i / total_genes, desc=f"🔄 Extracting {gene_id}... {j}/{len(gene_groups)} - {i}/{total_genes}")

            pool.close()
            pool.join()
            
    return result, result_error

Answer 3

As @vkkodali pointed out, the problem turned out to be that you shouldn't use multithreading.