Remote NCBI's Blast Perl API: maximising hits returned via command line

Question

Question and Background I'm using NCBI's Blast Perl API to interrogate Genbank for example blastp. I don't get the throughput using their point and click web stuff.

I tried Biopython's Bio.Blast import NCBIWWW but its under access restrictions from the host, notably attempting to use it during the day (works sporadically). The API however gets through and runs as fast as a web search ... that's good.

Here's an example of the command I'm running:

CMD=Put&PROGRAM=blastp&DATABASE=nr&HITLIST_SIZE=2500&ALIGNMENTS=2500&QUERY=%3Emytest%0AMSILANKDTR%0A

The above query gives default Blast output. What I'm trying to do is increase the number of hit recovered to the maximum (5000). The options for this are HITLIST_SIZE, ALIGNMENTS, DESCRIPTIONS. In the above example I'm attempting a hit_size of 2500 ... but the same output is returned as for default. More generally I can't get any of the "number of hits/descriptions/alignments returned" options to work, the default is returned each time.

Has anybody solved this?

Typical options for the API are given below

Typical options

Parameter   Definition  Type    CMD Allowed values
QUERY       Search query    String  Put *   Accession, GI, or FASTA.
DATABASE    BLAST database  String  Put *   Database from appendix 2 or one uploaded to blastdb_custom (see appendix 4)
PROGRAM     BLAST program   String  Put *   One of blastn, blastp, blastx, tblastn, tblastx. To enable megablast, use PROGRAM=blastn&MEGABLAST=on.
FILTER      Low complexity filtering    String  Put F to disable. T or L to enable. Prepend “m” for mask at lookup (e.g., mL)
FORMAT_TYPE Report type String  Put, Get    HTML, Text, XML, XML2, JSON2, or Tabular. HTML is the default.
EXPECT      Expect value    Double  Put Number greater than zero.
HITLIST_SIZE    Number of databases sequences to keep   Integer Put,Get Integer greater than zero.
DESCRIPTIONS    Number of descriptions to print (applies to HTML and Text)  Integer Put,Get Integer greater than zero.
ALIGNMENTS      Number of alignments to print (applies to HTML and Text)    Integer Put,Get Integer greater than zero.

---

Just to feedback on the answer and comment

• Using -remote by locally installing blast as recommended by @Scott. This is a valid answer so definitely upvoted and recommended its further upvoted. I had tried this but not mentioned it in the question (really sorry), but it times out due to traffic issues. The API overcomes traffic issues. Again, @Scott's is a valid solution (considering I'd not explained I'd tried it).

• More details ...

The Perl API is here https://blast.ncbi.nlm.nih.gov/docs/web_blast.pl

Line 95 below ...

$args = "CMD=Put&PROGRAM=$program&DATABASE=$database&QUERY=" . 
$encoded_query;

This is changed to,

$args = "CMD=Put&PROGRAM=$program&DATABASE=$database&HITLIST_SIZE=$maxhits&QUERY=" . $encoded_query;

Also need to add an additional line at line 73

$maxhits = shift;

From the command line,

perl ./web-blast.pl blastp nr 2500 ./example.fa

(Note, The variable order is switched in the program).

example.fa - e.g.

>rleguminosarum
WGFDGSSTQQAEGRSSDCVLKPVAIYPDPARTNGALVMCEVMMPDGVTPHASNARATILDDEDAWFGFEQ
EYFFYQNGRPLGFPEQGYPAPQGPYYTGVGYSNVGDVAREIVEEHLDLCLAAGINHEGINAEVAKGQWEF
QIFGKGSKKAADQIWMARYLLQRLTEKYGIDI

I'm proficient in Perl5, i.e. there's no error in changes to the code in the API, albeit my expertise is object oriented and high performance Python.

---

Limit on the request ... I'm testing that now.

First results eeek @terdon might be right ... :-(

Answer 1

While I haven't used your exact method above, using the BLAST command line tools return the desired number of target sequences

blastp  -query query.fa \
        -task blastp-short \
        -db nr \
        -max_target_seqs 5000 \
        -outfmt 6 \
        -out query.out \
        -remote

query.fa

>mytest
MSILANKDTR

The resulting query.out file does contain 5000 unique hits.

Answer 2

Right here's the response from NCBI,

Please send your future blast related questions to [email protected] directly. The gb-admin alias is for submitters with questions related to their sequence deposition/submission.

For your issue, you should look at the following two parameters: DESCRIPTIONS Number of descriptions to print (applies to HTML and Text) Integer Put,Get Integer greater than zero. ALIGNMENTS Number of alignments to print (applies to HTML and Text) Integer Put,Get Integer greater than zero.

Do please note that asking for this level of hits saved/returned could be problematic for the search. They could make the search exceeds the cpu time allocation and lead to error.

I tried this ... it was the same as "HITLIST_SIZE".

I figured the answer ...

Why its happening? The API uses a default parameter for the *maximum number of hits. It's imposing a limit on the number of hits returned, but this wasn't stated in the documentation of the parameters.

Whats the limit? It's >70 but <= 100. I suspect the limit is 100 hits. Thus anything above this its set at 100.

Summary The reason why the API is working and other approaches simply time out is NCBI is imposing an upper limit on the number of hits for the API and the Blast servers are not permitting requests above this threshold at peak time. Thus because other approaches are not imposing the restrictions on the NCBI server, the API gets through, whilst the other methods (blastp directly and Biopython) are timing out.

NCBI are recommending the API is because it is automating a 100 hit cut-off and the servers are deprioritising limits beyond this

I have filed a note of advice to NCBI because it should be clear in the documentation there's an upper limit on the HITLIST_SIZE parameter. It took me at least a full half day to figure this out (which isn't cool for limited documentation).

---

Proof of concept (untested) Thus, using @Scott's solution

blastp  -query query.fa \
        -task blastp-short \
        -db nr \
        -max_target_seqs 100 \
        -outfmt 6 \
        -out query.out \
        -remote

This will work with the -max_target_seqs 100 and thats all the API is doing.