1
$\begingroup$

I keep getting a "too many requests error" when querying the NCBI SRA database, even though I'm running less than 10 requests per second, and I have an API key, which supposedly should allow me to run 10 per second.

Here is my (Python) code:

import subprocess
import concurrent.futures
import time

bioprojects = [
    "PRJNA644722",
    "PRJNA644892"
]

all_metadata_generators = []

def fetch_metadata(bioproject):
    return subprocess.run(f"esearch -db sra -query '{bioproject}[bioproject]' | efetch -format runinfo", shell=True, capture_output=True, text=True)

with concurrent.futures.ThreadPoolExecutor() as executor:
    for i in range(0, len(bioprojects), 10):
        all_metadata_generators.append(executor.map(fetch_metadata, bioprojects[i:i+10]))
        time.sleep(1)


for metadata_generator in all_metadata_generators:
    for metadata in metadata_generator:
        print(metadata)

$\endgroup$
6
  • $\begingroup$ Ahh... this was often rumoured and there whispers on the wind of departmental bans - just rumours. You bang on with the sleep, I would just increase it until the signal detection police are happy. If you use a try: exception loop you could even try and attempt to attenuate the sleep time to just about the detection limit. Goodluck $\endgroup$
    – M__
    Commented Jul 9, 2020 at 21:52
  • $\begingroup$ @Michael thing is here sleep is technically not even doing anything since I only have 2 bioproject ids. So I can't even do 2 requests in more than one second without problems $\endgroup$ Commented Jul 9, 2020 at 22:00
  • 1
    $\begingroup$ where do you have the API key? In your .bashrc file? I would run the same command from the command line to check if you still get the 429 error. If you don't then you may have to specify the API key in the python command again. $\endgroup$
    – vkkodali
    Commented Jul 9, 2020 at 23:02
  • $\begingroup$ @vkkodali how do you specify it on the command line? I couldn't find it in the e-utilities documentation $\endgroup$ Commented Jul 9, 2020 at 23:11
  • $\begingroup$ @vkkodali ah I see now that actually the API key was not set. I set it from Biopython (and I'm not using Biopython in my code) so obviously it wasn't going to be used above. Silly me! $\endgroup$ Commented Jul 9, 2020 at 23:14

2 Answers 2

2
$\begingroup$

The execution of your EDirect commands may be limited unless you are using an API Key, which you can create from your MyNCBI account. Your API Key can be used in one of several different ways.

In an argument to an EDirect command:

esearch -db nuccore -query 'some query' -api_key 12345

In a UNIX environment:

You can either run this just before you are running some edirect commands or add the following line to your .bashrc file.

export NCBI_API_KEY=12345

In the environment, from a Perl script:

$ENV{NCBI_API_KEY} = "12345"
$\endgroup$
4
  • $\begingroup$ did as you said, yet I still get "too many requests" for quite a few $\endgroup$ Commented Jul 10, 2020 at 0:48
  • $\begingroup$ Even went as far as setting the sleep time to 10, and the step size for the for loop to 1, and still get too many requests $\endgroup$ Commented Jul 10, 2020 at 0:55
  • 1
    $\begingroup$ I typically don't use Python for this sort of thing. I just run the command in a bash shell. I suspect the reason for the too many requests error is because you are using multithreading. Does your code simultaneously run 10 instances of the fetch_metadata function? May be try without the multithreading to see if it works? $\endgroup$
    – vkkodali
    Commented Jul 10, 2020 at 2:13
  • $\begingroup$ yes that worked. Interesting, thanks! $\endgroup$ Commented Jul 10, 2020 at 2:19
0
$\begingroup$

As @vkkodali pointed out, the problem turned out to be that you shouldn't use multithreading.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.