repophlan script to download bacterial genome

Question

I am trying to donwload the microbial genome using the repophlan_get_microbes.py (https://github.com/SegataLab/repophlan) but now it is running more than 10 days and still not finished on slurm HPC.

Could anyone please suggest what is the issue with the script?

#!/bin/bash
#SBATCH --job-name=download.GENOME    # Job name
#SBATCH --ntasks=60
#SBATCH --nodes=3                  # Number of nodes           
#SBATCH --time=2000:00:00              # Time limit hrs:min:sec
#SBATCH --partition=k2-himem
#SBATCH --mem=800G





module add apps/python3/3.7.4/gcc-4.8.5
module add libs/biopython/1.73/gcc-4.8.5+python-2.7.8+numpy-1.16.0


python repophlan_get_microbes.py --taxonomy taxonomy_reduced.txt --out_dir microbes --nproc 20 --out_summary repophlan_microbes.txt | tee repophlan_microbes.log

$ head repophlan_microbes.log
2020-09-17 20:44:59,964 ./repophlan_get_microbes.py INFO     Reading the taxonomy from taxonomy_reduced.txt...
2020-09-17 20:45:04,921 ./repophlan_get_microbes.py INFO     Done.
2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING  GCF_000001215.4 [Drosophila melanogaster ] excluded from download because of uninteresting phyla!
2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING  GCF_000001405.39 [Homo sapiens ] excluded from download because of uninteresting phyla!
2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING  GCF_000001635.26 [Mus musculus ] excluded from download because of uninteresting phyla!
2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING  GCF_000001735.4 [Arabidopsis thaliana ecotype=Columbia] excluded from download because of uninteresting phyla!
2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING  GCF_000001895.5 [Rattus norvegicus strain=mixed] excluded from download because of uninteresting phyla!
2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING  GCF_000001905.1 [Loxodonta africana ] excluded from download because of uninteresting phyla!
2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING  GCF_000002035.6 [Danio rerio ] excluded from download because of uninteresting phyla!
2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING  GCF_000002075.1 [Aplysia californica ] excluded from download because of uninteresting phyla!


$ tail repophlan_microbes.log


2020-09-21 20:42:26,871 repophlan_get_microbes.py INFO     Parsing of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/635/075/GCA_001635075.1_ASM163507v1/GCA_001635075.1_ASM163507v1_genomic.gbff.gz
2020-09-21 20:42:27,198 repophlan_get_microbes.py INFO     Parsing of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/010/319/175/GCA_010319175.1_PDT000567389.1/GCA_010319175.1_PDT000567389.1_genomic.gbff.gz
2020-09-21 20:42:27,482 repophlan_get_microbes.py INFO     Parsing of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/003/518/025/GCA_003518025.1_ASM351802v1/GCA_003518025.1_ASM351802v1_protein.faa.gz
2020-09-21 20:42:27,608 repophlan_get_microbes.py INFO     Parsing of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/788/535/GCA_001788535.1_ASM178853v1/GCA_001788535.1_ASM178853v1_protein.faa.gz
2020-09-21 20:42:27,628 repophlan_get_microbes.py WARNING  No remote file found (some ffn and faa are know to be missing remotely). Aborting the download of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/010/997/445/GCA_010997445.1_ASM1099744v1/GCA_010997445.1_ASM1099744v1_protein.faa.gz. <urlopen error ftp error: [Errno ftp error] 550 GCA_010997445.1_ASM1099744v1_protein.faa.gz: No such file or directory>
2020-09-21 20:42:27,628 repophlan_get_microbes.py ERROR    Error in downloading of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/010/997/445/GCA_010997445.1_ASM1099744v1/GCA_010997445.1_ASM1099744v1_protein.faa.gz <urlopen error ftp error: [Errno ftp error] 550 GCA_010997445.1_ASM1099744v1_protein.faa.gz: No such file or directory>
2020-09-21 20:42:27,628 repophlan_get_microbes.py ERROR    Error in downloading of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/010/997/445/GCA_010997445.1_ASM1099744v1/GCA_010997445.1_ASM1099744v1_protein.faa.gz: empty fna file!
2020-09-21 20:42:28,239 repophlan_get_microbes.py WARNING  No remote file found (some ffn and faa are know to be missing remotely). Aborting the download of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz. <urlopen error ftp error: [Errno ftp error] 550 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz: No such file or directory>
2020-09-21 20:42:28,239 repophlan_get_microbes.py ERROR    Error in downloading of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz <urlopen error ftp error: [Errno ftp error] 550 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz: No such file or directory>
2020-09-21 20:42:28,239 repophlan_get_microbes.py ERROR    Error in downloading of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz: empty fna file!

Many thanks

Answer 1

UPDATE 2: new log posted

take-home message: I think it is working, it's just that a few errors happen sometimes when downloading things from big databases.

It looks like there are a few warnings or errors associated with getting specific files. I think this is actually ok. For example for one of the genomes, I logged into the ftp manually to check if the file was there:

$ ftp ftp://ftp.ncbi.nlm.nih.gov
Trying 130.14.250.13...
Connected to ftp.wip.ncbi.nlm.nih.gov.
220-
 This warning banner provides privacy and security notices consistent with 
 applicable federal laws, directives, and other federal guidance for accessing 
 this Government system, which includes all devices/storage media attached to 
 this system. This system is provided for Government-authorized use only. 
 Unauthorized or improper use of this system is prohibited and may result in 
 disciplinary action and/or civil and criminal penalties. At any time, and for 
 any lawful Government purpose, the government may monitor, record, and audit 
 your system usage and/or intercept, search and seize any communication or data 
 transiting or stored on this system. Therefore, you have no reasonable 
 expectation of privacy. Any communication or data transiting or stored on this 
 system may be disclosed or used for any lawful Government purpose.
220 FTP Server ready.
331 Anonymous login ok, send your complete email address as your password
230 Anonymous access granted, restrictions apply
Remote system type is UNIX.
Using binary mode to transfer files.
200 Type set to I
ftp> get genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz
local: genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz remote: genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz
ftp: Can't access `genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz': No such file or directory

ftp> ls genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/

229 Entering Extended Passive Mode (|||50025|)
150 Opening ASCII mode data connection for file list
-r--r--r--   1 ftp      anonymous     2725 Feb 23  2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_assembly_report.txt
-r--r--r--   1 ftp      anonymous     3456 Feb 23  2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_assembly_stats.txt
dr-xr-xr-x   3 ftp      anonymous     4096 Feb 23  2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_assembly_structure
-r--r--r--   1 ftp      anonymous      149 Apr 21 19:29 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_feature_count.txt.gz
-r--r--r--   1 ftp      anonymous   949241 Feb 23  2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_genomic.fna.gz
-r--r--r--   1 ftp      anonymous  1353865 Feb 23  2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_genomic.gbff.gz
-r--r--r--   1 ftp      anonymous      150 Feb 23  2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_genomic_gaps.txt.gz
-r--r--r--   1 ftp      anonymous      840 Feb 23  2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_wgsmaster.gbff.gz
-r--r--r--   1 ftp      anonymous    44529 Sep  2 20:26 README.txt
-r--r--r--   1 ftp      anonymous      410 Feb 23  2020 annotation_hashes.txt
-r--r--r--   1 ftp      anonymous       14 Sep 21 07:11 assembly_status.txt
-r--r--r--   1 ftp      anonymous     1399 May 20 23:00 md5checksums.txt
226 Transfer complete

So in this particular case, that genome exists but is just missing a faa.gz file. So the script appears to be working as expected, it's just that some of the NCBI submissions don't have everything they are supposed to have.

(As a tangent, that Rumen genome came from a paper I'm an author on I'm pretty sure!! So I guess you could blame me for the missing file if you wanted, though I was in no way involved with annotations or submission...)

UPDATE: now the log is posted

It does not look like a connectivity issue based on the new information provided in the first and last lines of the log file. There it looks like there are a lot of processes finishing quickly towards the beginning and the end of the process (e.g. 48 hours apart).

Based on this new information, I am wondering if you have simply managed to get a taxonomy file that is much larger (e.g. >10X larger to explain the time difference) than anticipated when the README for the script was written, and thus it simply takes a long time.

It would still probably be best to go through the log file rather carefully and see if there is a big hang in the process, errors, warnings, etc.

---

Here is what they say in the repo:

*** Download the microbial (bacteria+archaea) genomes (fna,ffn,faa,frn) with 
    corresponding taxonomy
$ ./repophlan_get_microbes.py --taxonomy taxonomy_reduced.txt --out_dir /
  microbes --nproc 20 --out_summary repophlan_microbes.txt /
  | tee repophlan_microbes.log
[ ~6 hours ]. Downloads and save all available sequences for microbes. It uses
multiple parallel connections to speed up the retrieval: common exceptions 
including issues with the NCBI ftp which temporarily rejects connections for
ftp load problems are handled gracefully with a 'delay and retry' policy. 
IMPORTANT: specifying more than 20 processors to run in parallel causes serious
problems in terms of exceeding the allowed number of connections by NCBI causing
long delays.

Since your script is taking much much longer than 6 hours, it probably didn't work.

Some thoughts based only on the info in the repo:

• "IMPORTANT: specifying more than 20 processors to run in parallel causes serious problems in terms of exceeding the allowed number of connections by NCBI causing long delays." this sounds like your problem. I think you are asking for 60 nodes ("tasks"? "nodes" is 3 which should be ok). Drop tasks to 20 or less and see what happens, maybe.
• "...temporarily rejects connections for ftp load problems are handled gracefully with a 'delay and retry' policy." It is possible that there is a bad request in there that it keeps trying over and over again unsuccessfully, it could keep going forever if such a problem exists and you don't stop it.

As pointed out in the comments, you should check the log and post that here.

As also pointed out in the comments, you might consider posting an issue at the repo as they are more familiar than we are.

Other things to check:

• how much disk space do your nodes have access to? is it enough for the download?
• are you satisfying all dependencies of the script? In other words do you need to load more modules?