Too many requests error when querying NCBI

Question

I keep getting a "too many requests error" when querying the NCBI SRA database, even though I'm running less than 10 requests per second, and I have an API key, which supposedly should allow me to run 10 per second.

Here is my (Python) code:

import subprocess
import concurrent.futures
import time

bioprojects = [
    "PRJNA644722",
    "PRJNA644892"
]

all_metadata_generators = []

def fetch_metadata(bioproject):
    return subprocess.run(f"esearch -db sra -query '{bioproject}[bioproject]' | efetch -format runinfo", shell=True, capture_output=True, text=True)

with concurrent.futures.ThreadPoolExecutor() as executor:
    for i in range(0, len(bioprojects), 10):
        all_metadata_generators.append(executor.map(fetch_metadata, bioprojects[i:i+10]))
        time.sleep(1)


for metadata_generator in all_metadata_generators:
    for metadata in metadata_generator:
        print(metadata)

Answer 1

The execution of your EDirect commands may be limited unless you are using an API Key, which you can create from your MyNCBI account. Your API Key can be used in one of several different ways.

In an argument to an EDirect command:

esearch -db nuccore -query 'some query' -api_key 12345

In a UNIX environment:

You can either run this just before you are running some edirect commands or add the following line to your .bashrc file.

export NCBI_API_KEY=12345

In the environment, from a Perl script:

$ENV{NCBI_API_KEY} = "12345"

Answer 2

As @vkkodali pointed out, the problem turned out to be that you shouldn't use multithreading.