How to obtain NCBI Nucleotide (Entrez) IDs for all E coli sequences?

Question

I would like to gather a list of ALL E coli NCBI (Entrez/Nucleotide) IDs so that I can download fna and faa files, through a function like this:

# python code #
from Bio import Entrez
from glob import glob

def dl_genome(id, folder='genomes'):
    files=glob('%s/*.fna'%folder)
    out_file = '%s/%s.fna'%(folder, id)

    if out_file in files:
        print (out_file, 'already downloaded')
        return
    else:
        print ('downloading %s from NCBI'%id)
        
    Entrez.email = ""
    handle = Entrez.efetch(db="nucleotide", id=id, rettype="fna", retmode="text")
    with open(out_file, 'w') as fout:
        fout.write(handle.read())

for strain in targetStrainIDs:
    dl_genome(strain, folder='genomes')

My problem is that I don't have a good way to get the E coli strain names and their corresponding NCBI IDs for Entrez. Is there a way to get these NCBI IDs without manually searching for them?

Context

I have been trying to develop an E coli pan-genome for one of my projects. I initially just downloaded the relevant genomes (fna, faa, & features.tab files) from the PATRIC database. However, I've come to realize that the reference sequence on PATRIC has issues. Specifically, the reference sequence on PATRIC's fna/faa files for E coli K12 MG1655 notably do NOT list genes like b2031 or b2033, despite both being present in this strain.

The NCBI reference genome DOES include these genes (with proper annotations), so I'd like to just download what I need from NCBI instead. However, I'd like a systematic way to download all the fna/faa files since my pan-genome is eventually going to be built from hundreds of of these fna/faa files.

Any help would be appreciated.

Answer 1

You should check out NCBI Datasets for this sort of thing. Since this is bacteria, you won't be able to use the web interface but there's an API and a command-line tool that you can use.

For example, you can use the datasets command line tool to download the E. coli RefSeq genome assemblies as follows:

## see how many genomes are in scope
$ datasets summary genome taxon 562 --limit none --refseq
{"total_count":22287}

## download data excluding gff3, rna and genome seq files
## as an example but drop --exclude-seq to download genome seq as well
$ datasets download genome taxon 562 --refseq --exclude-gff3 --exclude-rna --exclude-seq --filename ecoli_genomes.zip

Answer 2

Do you mean the NCBI taxonomy IDs? If so, I've personally always preferred the way UniProt displays NCBI taxonomy, and it's easier to download IDs in mass.

For example, if you find E.coli's entry: https://www.uniprot.org/taxonomy/562

You can then click on "all lower taxonomy nodes", which will show all E.coli strains. If you click 'Preview first 10' before downloading, you'll see that the first column is the NCBI taxonomy ID of each strain of E.coli.

For example, E.coli strain 0.1197 is reported to have the ID 1005443 from this preview page. Indeed, if we search this ID on NCBI's taxonomy page, we do get the correct entry: https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?id=1005443

If this is what you want, click 'download all' to get a nice tab-separated file of all the strains of E.coli and their NCBI taxonomy IDs. This file could then be read and processed before you send API requests to Entrez.

Hope this helps!