I am trying to donwload the microbial genome using the repophlan_get_microbes.py (https://github.com/SegataLab/repophlan) but now it is running more than 10 days and still not finished on slurm HPC.

Could anyone please suggest what is the issue with the script?

#!/bin/bash
#SBATCH --nodes=3                  # Number of nodes
#SBATCH --time=2000:00:00              # Time limit hrs:min:sec
#SBATCH --partition=k2-himem
#SBATCH --mem=800G

python repophlan_get_microbes.py --taxonomy taxonomy_reduced.txt --out_dir microbes --nproc 20 --out_summary repophlan_microbes.txt | tee repophlan_microbes.log

$head repophlan_microbes.log 2020-09-17 20:44:59,964 ./repophlan_get_microbes.py INFO Reading the taxonomy from taxonomy_reduced.txt... 2020-09-17 20:45:04,921 ./repophlan_get_microbes.py INFO Done. 2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING GCF_000001215.4 [Drosophila melanogaster ] excluded from download because of uninteresting phyla! 2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING GCF_000001405.39 [Homo sapiens ] excluded from download because of uninteresting phyla! 2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING GCF_000001635.26 [Mus musculus ] excluded from download because of uninteresting phyla! 2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING GCF_000001735.4 [Arabidopsis thaliana ecotype=Columbia] excluded from download because of uninteresting phyla! 2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING GCF_000001895.5 [Rattus norvegicus strain=mixed] excluded from download because of uninteresting phyla! 2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING GCF_000001905.1 [Loxodonta africana ] excluded from download because of uninteresting phyla! 2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING GCF_000002035.6 [Danio rerio ] excluded from download because of uninteresting phyla! 2020-09-17 20:45:10,716 ./repophlan_get_microbes.py WARNING GCF_000002075.1 [Aplysia californica ] excluded from download because of uninteresting phyla!$ tail repophlan_microbes.log

2020-09-21 20:42:26,871 repophlan_get_microbes.py INFO     Parsing of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/635/075/GCA_001635075.1_ASM163507v1/GCA_001635075.1_ASM163507v1_genomic.gbff.gz
2020-09-21 20:42:27,198 repophlan_get_microbes.py INFO     Parsing of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/010/319/175/GCA_010319175.1_PDT000567389.1/GCA_010319175.1_PDT000567389.1_genomic.gbff.gz
2020-09-21 20:42:27,482 repophlan_get_microbes.py INFO     Parsing of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/003/518/025/GCA_003518025.1_ASM351802v1/GCA_003518025.1_ASM351802v1_protein.faa.gz
2020-09-21 20:42:27,608 repophlan_get_microbes.py INFO     Parsing of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/788/535/GCA_001788535.1_ASM178853v1/GCA_001788535.1_ASM178853v1_protein.faa.gz
2020-09-21 20:42:27,628 repophlan_get_microbes.py WARNING  No remote file found (some ffn and faa are know to be missing remotely). Aborting the download of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/010/997/445/GCA_010997445.1_ASM1099744v1/GCA_010997445.1_ASM1099744v1_protein.faa.gz. <urlopen error ftp error: [Errno ftp error] 550 GCA_010997445.1_ASM1099744v1_protein.faa.gz: No such file or directory>
2020-09-21 20:42:27,628 repophlan_get_microbes.py ERROR    Error in downloading of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/010/997/445/GCA_010997445.1_ASM1099744v1/GCA_010997445.1_ASM1099744v1_protein.faa.gz <urlopen error ftp error: [Errno ftp error] 550 GCA_010997445.1_ASM1099744v1_protein.faa.gz: No such file or directory>
2020-09-21 20:42:28,239 repophlan_get_microbes.py WARNING  No remote file found (some ffn and faa are know to be missing remotely). Aborting the download of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz. <urlopen error ftp error: [Errno ftp error] 550 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz: No such file or directory>
2020-09-21 20:42:28,239 repophlan_get_microbes.py ERROR    Error in downloading of ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz <urlopen error ftp error: [Errno ftp error] 550 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz: No such file or directory>


Many thanks

• You should post this as an issue on their GitHub repository. You might get a quicker response that way. Sep 17, 2020 at 23:25
• What is the output of repophlan_microbes.log? Sep 18, 2020 at 17:16
• thanks, it is now added in question Sep 21, 2020 at 12:36

UPDATE 2: new log posted

take-home message: I think it is working, it's just that a few errors happen sometimes when downloading things from big databases.

It looks like there are a few warnings or errors associated with getting specific files. I think this is actually ok. For example for one of the genomes, I logged into the ftp manually to check if the file was there:

$ftp ftp://ftp.ncbi.nlm.nih.gov Trying 130.14.250.13... Connected to ftp.wip.ncbi.nlm.nih.gov. 220- This warning banner provides privacy and security notices consistent with applicable federal laws, directives, and other federal guidance for accessing this Government system, which includes all devices/storage media attached to this system. This system is provided for Government-authorized use only. Unauthorized or improper use of this system is prohibited and may result in disciplinary action and/or civil and criminal penalties. At any time, and for any lawful Government purpose, the government may monitor, record, and audit your system usage and/or intercept, search and seize any communication or data transiting or stored on this system. Therefore, you have no reasonable expectation of privacy. Any communication or data transiting or stored on this system may be disclosed or used for any lawful Government purpose. 220 FTP Server ready. 331 Anonymous login ok, send your complete email address as your password 230 Anonymous access granted, restrictions apply Remote system type is UNIX. Using binary mode to transfer files. 200 Type set to I ftp> get genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz local: genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz remote: genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz ftp: Can't access genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/GCA_902778895.1_Rumen_uncultured_genome_RUG12414_protein.faa.gz': No such file or directory ftp> ls genomes/all/GCA/902/778/895/GCA_902778895.1_Rumen_uncultured_genome_RUG12414/ 229 Entering Extended Passive Mode (|||50025|) 150 Opening ASCII mode data connection for file list -r--r--r-- 1 ftp anonymous 2725 Feb 23 2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_assembly_report.txt -r--r--r-- 1 ftp anonymous 3456 Feb 23 2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_assembly_stats.txt dr-xr-xr-x 3 ftp anonymous 4096 Feb 23 2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_assembly_structure -r--r--r-- 1 ftp anonymous 149 Apr 21 19:29 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_feature_count.txt.gz -r--r--r-- 1 ftp anonymous 949241 Feb 23 2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_genomic.fna.gz -r--r--r-- 1 ftp anonymous 1353865 Feb 23 2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_genomic.gbff.gz -r--r--r-- 1 ftp anonymous 150 Feb 23 2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_genomic_gaps.txt.gz -r--r--r-- 1 ftp anonymous 840 Feb 23 2020 GCA_902778895.1_Rumen_uncultured_genome_RUG12414_wgsmaster.gbff.gz -r--r--r-- 1 ftp anonymous 44529 Sep 2 20:26 README.txt -r--r--r-- 1 ftp anonymous 410 Feb 23 2020 annotation_hashes.txt -r--r--r-- 1 ftp anonymous 14 Sep 21 07:11 assembly_status.txt -r--r--r-- 1 ftp anonymous 1399 May 20 23:00 md5checksums.txt 226 Transfer complete  So in this particular case, that genome exists but is just missing a faa.gz file. So the script appears to be working as expected, it's just that some of the NCBI submissions don't have everything they are supposed to have. (As a tangent, that Rumen genome came from a paper I'm an author on I'm pretty sure!! So I guess you could blame me for the missing file if you wanted, though I was in no way involved with annotations or submission...) UPDATE: now the log is posted It does not look like a connectivity issue based on the new information provided in the first and last lines of the log file. There it looks like there are a lot of processes finishing quickly towards the beginning and the end of the process (e.g. 48 hours apart). Based on this new information, I am wondering if you have simply managed to get a taxonomy file that is much larger (e.g. >10X larger to explain the time difference) than anticipated when the README for the script was written, and thus it simply takes a long time. It would still probably be best to go through the log file rather carefully and see if there is a big hang in the process, errors, warnings, etc. Here is what they say in the repo: *** Download the microbial (bacteria+archaea) genomes (fna,ffn,faa,frn) with corresponding taxonomy$ ./repophlan_get_microbes.py --taxonomy taxonomy_reduced.txt --out_dir /
microbes --nproc 20 --out_summary repophlan_microbes.txt /
| tee repophlan_microbes.log
[ ~6 hours ]. Downloads and save all available sequences for microbes. It uses
multiple parallel connections to speed up the retrieval: common exceptions
including issues with the NCBI ftp which temporarily rejects connections for
ftp load problems are handled gracefully with a 'delay and retry' policy.
IMPORTANT: specifying more than 20 processors to run in parallel causes serious
problems in terms of exceeding the allowed number of connections by NCBI causing
long delays.

`

Since your script is taking much much longer than 6 hours, it probably didn't work.

Some thoughts based only on the info in the repo:

• "IMPORTANT: specifying more than 20 processors to run in parallel causes serious problems in terms of exceeding the allowed number of connections by NCBI causing long delays." this sounds like your problem. I think you are asking for 60 nodes ("tasks"? "nodes" is 3 which should be ok). Drop tasks to 20 or less and see what happens, maybe.
• "...temporarily rejects connections for ftp load problems are handled gracefully with a 'delay and retry' policy." It is possible that there is a bad request in there that it keeps trying over and over again unsuccessfully, it could keep going forever if such a problem exists and you don't stop it.

As pointed out in the comments, you should check the log and post that here.

As also pointed out in the comments, you might consider posting an issue at the repo as they are more familiar than we are.

Other things to check: