My solution is simply.
- perform a web-based Blastp using the following parameters:
- most dissimilar
- 5000 hits (maximum limit)
- Restrict to Archaea (which is what you samples appear to be)*
If you hit the limit of 5000 results, I would repeat the search for each class or phylum within Archaea. I can't remember if it was re-classified as a kingdom or was just another phylum. Concatenate all hits into a single fasta file.
Reg-ex your extremophiles database (personally I'd use pandas), so that the species name of each bacterium is recovered no other information is present. Or this is an easily accessible value within a hash/database (this is a better solution).
Use Biopython to loop through your blast fasta file and filter using
re search against the species from your database. If you get a hit save that sequence.
Ideally, the point 2 and 3 would be a pandas merge operation, but looping is easier.
*, If there are extremophiles which are not Archaea (I don't know much about them), then take each general taxonomic group and repeat the operation for the search criteria "restrict the blastp to ... specific group".
Issues The issues with command line blastp, not via the web, is that access is greatly restricted during US working hours. If you are searching more than 100 hits and carrying out the search during the working day US time - they will not let it through. 100 hits or less is fine however. For >100 hits (e.g. 5000) it's got to be done well outside US working hours/ weekends. Web-based blastp doesn't carry this restriction of access during US working hours. Basically, the API route is deprioritised for the web-based approach. I assume this is to stop bots - albeit of course a bot can automate through NCBI's webpage.
Note The original solution by @terdon might easily work as a remote blastp because each query is restricted to <100 - so it will always get through during working hours. What I am saying is if you're using Blast API it needs a carefully defined strategy, else shift to the web for blanket approaches.
Entrez operations from a commandline do work ok during US working day time - albeit it's got to be coded to deal with "random" server refusals otherwise it will break the code (sporadic queries are refused). This is just my recent experience nothing more.
Just to be clear - this is potentially a medium sized project. Once the strategy has been decided then specific code that supports this approach can be given in separate questions.