5
$\begingroup$

I have asked this question in biostars, but I am trying here as well as people working with "omics" data might be able to help. I think my issue relates understanding how large data storage on online server works. I am trying to download large data from the Human Microbiome Project. These are available on online server and can be downloaded via aspera (ascp).

I can't figure out the right path to get the data. On their website, they give an example of command to retreive an example dataset:

 ascp -l 100M -QT user@aspera.hmpdacc.org:/ReferenceGenomes/test_genome.tar /path/to/local/directory

I executed this, using my own username, from this I am asked my password, and I can authenticate. But then I get the following error:

(Error: Server aborted session: No such file or directory)

Then, I downloaded the manifest file via the HMP portal. In this manifest, they provide the path to which these files are supposed to be available. I tried to execute again the command with one of those path, provided by the manifest file:

ascp -l 100M -QT user@aspera.hmpdacc.org:/files/5a950f27980b5d93e4c16da124000378  /path/to/local/directory

I again get the same error of "No such file or directory".

The full url provided by the manifest

I tried putting different chunks of this address into to ascp command but nothing works. I think there is something I don't understand about how to access the server via aspera and what is expected to go in this path. When I copy paste this url in google, this directly download the file I want. So I'm close, but I need to do it via command line as I need to download hundreds of these files and they are huge (6-10GB each).

How can I download the data of the human microbiome project using ascp?

$\endgroup$
  • $\begingroup$ Is the manifest file a list of links like you provided? I mean to ask is the manifest file hundreds of lines with a URLs like http://downloads.hmpdacc.org/data2/hhs/genome/microbiome/wgs/analysis/hmwgsqc/v2/SRS013951.tar.bz2per line? It would helpful to see the format of the manifest file. The first 5 lines should suffice. $\endgroup$ – Kohl Kinning Jul 4 '18 at 19:15
-1
$\begingroup$

I can't test for ascp because they do not longer support my current setup but you can follow the FTP instructions. With FileZilla:

url: public-ftp.ihmpdcc.org

| improve this answer | |
$\endgroup$
  • $\begingroup$ It looks like @CAsimonet has already gone through the instructions on the website and is have fine-tuned issues with getting a lot of specific files. $\endgroup$ – Kohl Kinning Jul 4 '18 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.