Remote blastn - what is breaking my (bash) loop?

Question

I am trying to search the NCBI non-redundant database for sequences similar to a few other (~40) sequences that I already have.

So I've tried running blastn remotely, and looping through multiple queries. I've tried it both as a job (via slurm) and from the command line.

In addition, for every query, I am trying to run 3 searches with slightly different parameters (in WebBlast, I noticed that I recover more sequences of interest this way).

I am trying to dump the output data in a single file (stuff like duplicates can be sorted out later).

This is my code:

for i in /directory/*.fa;do #this is the directory containing the queries
j=${i#"/directory/"}
k=${j%".fa"} #getting a short version of the name for the log file
echo "$k" >> "log.log"
blastn -query "$i" -db "nt" -remote -entrez_query 700:1000[slen] -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
blastn -query "$i" -db "nt" -remote -entrez_query 1000:1400[slen] -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
blastn -query "$i" -db "nt" -remote -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
done

However, it...doesn't seem to work? Either only the first search for the first query is completed, and then the code does nothing else, or nothing happens. (in a rare case, the first two searches worked) There seems to be no clear distinction between 'it works a tiny bit' and 'it doesn't work at all', which leads me to think it might be something outside the code, e.g. connection issues.

Has anyone faced this before, and would you know how to fix it?

Update I tried to run a single search as well. As well as adding the waiting period. Right now it doesn't work (same issue, no error message, just sort of appearing to run without results). In the link that Scot provided below, from the NCBI, it was mentioned 'We will move searches of users who submit more than 100 searches in a 24 hour period to a slower queue, or, in extreme cases, will block the requests.' I haven't run >100 searches yesterday (was planning to before I knew this, but that's a different issue).

I think I'll limit the searches as recommended and try to run it from a different computer later on.

Answer 1

I gave it a whirl using a random bit of SARS-CoV-2

blastn -query out.fa -db "nt" -remote  -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"

blastn -query out.fa -db "nt" -remote -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results2.tsv"

Both gave a 500 line output - the limit of the blast specified.

One possibility is to place a sleep 10 line between each blastn command (3 additional lines).

The rationale is specifications NCBI set:

Do not contact the server more often than once every 10 seconds. Do not poll for any single RID more often than once a minute. Use the URL parameter email and tool, so that the NCBI can contact you if there is a problem. Run scripts weekends or between 9 pm and 5 am Eastern time on weekdays if more than 50 searches will be submitted.

---

The suggestions haven't worked, so I'd recommend trying sleep 60 between each blastncommand on the basis this might be an issue with RID. If that fails the fall back is to run the job overnight US time via cron tab, which I'll post the code if it is required. Thus,

Run scripts weekends or between 9 pm and 5 am Eastern time on weekdays if more than 50 searches will be submitted.

The final thing to add is that it is a fairly long job so connectivity might be an issue. If this was the case then it might also be resolved by running the script overnight in the OPs timezone, i.e. via cron

---

sleep 60 works but is very slow.

• Agreed one option is to download the database and perform the Blast locally.
• Alternative use mmseqs2
• You could perform the Blast on the Cloud, like GCP (much faster).
• Personally I would look at performing over night runs when the network was not in use.

Answer 2

This is likely a throttling issue - see here for some details, specifically the one about contacting the NCBI server more than once every 10 seconds.

You may be able to submit a multiFASTA of all your queries to get around this issue, rather than individual FASTA sequences, or alternatively, add in a sleep 10 command between your BLAST commands and at the end of each for loop iteration to throttle the requests.