How to download multiple proteomes at once?

Question

I'm looking for a way to download multiple proteomes at once from one clade, as fasta. Probably from the NCBI, because it looks more user-friendly than Ensembl, but Ensembl is okay too. In the best case scenario, I would like to constrain the download to a specific clade (eg. Actinobacteria, Chloroflexi etc.), maybe with a specific RefSeq Category (eg. reference/representative genome) or even depending on the genome assembly level. Command line would probably be the best option, but if there's a direct download option it would be nice to know it as well. Does anything like this exist? Help would be much appreciated!

Answer 1

You could try download-refseq-genomes. It will fetch all genomes from the NCBI FTP server that are in a specific subtree of the phylogeny.

For example, downloading the amino acid sequences from all proteins in each genome assembly for all species belonging to Chloroflexi (NCBI taxon ID = 200795):

download-refseq-genomes.pl -t faa 200795

This will download all assemblies with status=Complete. Use option -a to download all assemblies with any status.

Answer 2

You can use NCBI resources to download these data directly. For the entire proteome, you can download the sequences of all proteins annotated on the genome assembly in FASTA format. In the NCBI FTP directories, these files have the names with the suffix _protein.faa.gz. More details about the files in the FTP directories are described in the README.

Command-line method using EDirect

This method uses Entrez Direct to first query the Assembly database for all available genomes, parses the output to extract the ftp path and downloads data using wget command.

esearch -db assembly -query 'txid200795[organism] AND lastest_refseq[filter]' \
  | esummary \
  | xtract -pattern DocumentSummary -element FtpPath_RefSeq \
  | while read -r url ; do 
    path=$(echo $url | perl -pe 's/(GC[FA]_\d+.*)/\1\/\1_protein.faa.gz/g') ; 
    wget -q --show-progress "$path" -P protein_data ; 
  done

Here, I am restricting the results to latest RefSeq data by using the latest_refseq[filter] but you can use any other Entrez filters to further narrow down your results. Along the same lines, you can modify the code minimally as shown below to download multiple file types:

esearch -db assembly -query 'txid200795[organism] AND latest_refseq[filter]' \
  | esummary \
  | xtract -pattern DocumentSummary -element FtpPath_RefSeq \
  | while read -r url ; do 
    base_path=$(echo $url | perl -pe 's/(GCF_\d+.*)/\1\/\1_/g') ; 
    for suf in genomic.fna.gz protein.faa.gz ; do
      wget -q --show-progress "$base_path$suf" -P ncbi_data ; 
    done ;
  done

GUI method

You can use the same query as shown above in the NCBI Assembly portal and then use the 'Download Assemblies' button to pick and choose the type of files you would like to download. By default, if you do not select any assemblies in the results, this interface downloads files for all results.