I am trying to search the NCBI non-redundant database for sequences similar to a few other (~40) sequences that I already have.
So I've tried running blastn remotely, and looping through multiple queries. I've tried it both as a job (via slurm) and from the command line.
In addition, for every query, I am trying to run 3 searches with slightly different parameters (in WebBlast, I noticed that I recover more sequences of interest this way).
I am trying to dump the output data in a single file (stuff like duplicates can be sorted out later).
This is my code:
for i in /directory/*.fa;do #this is the directory containing the queries
j=${i#"/directory/"}
k=${j%".fa"} #getting a short version of the name for the log file
echo "$k" >> "log.log"
blastn -query "$i" -db "nt" -remote -entrez_query 700:1000[slen] -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
blastn -query "$i" -db "nt" -remote -entrez_query 1000:1400[slen] -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
blastn -query "$i" -db "nt" -remote -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
done
However, it...doesn't seem to work? Either only the first search for the first query is completed, and then the code does nothing else, or nothing happens. (in a rare case, the first two searches worked) There seems to be no clear distinction between 'it works a tiny bit' and 'it doesn't work at all', which leads me to think it might be something outside the code, e.g. connection issues.
Has anyone faced this before, and would you know how to fix it?
Update I tried to run a single search as well. As well as adding the waiting period. Right now it doesn't work (same issue, no error message, just sort of appearing to run without results). In the link that Scot provided below, from the NCBI, it was mentioned 'We will move searches of users who submit more than 100 searches in a 24 hour period to a slower queue, or, in extreme cases, will block the requests.' I haven't run >100 searches yesterday (was planning to before I knew this, but that's a different issue).
I think I'll limit the searches as recommended and try to run it from a different computer later on.
sleep 60
? The other idea is to set up acron
job and run it over night, I wouldn't try and manually overcome the challenge. $\endgroup$nt
database, and running the alignments locally.nt
is not a small database, so this may be best done using slurm on your cluster, but if you can limit the alignments to a taxonomic segment ofnt
, you may be able to do this locally using just that taxonomic slice ofnt
. $\endgroup$