Hi I am downloading the data from a website which requires authentication of email_id and password before the download. They then provide the download link which is used to download dataset. This download link is different every time

### Download Cosmic Files

##First you need to login with your login details and the access key ID may vary with each login ID

echo "[email protected]:abc123@" | base64
#The above lines gives a authorization key cHIxNEBzYW5

# get the download link using the authorization key from step 1
#curl -H "Authorization: Basic cHIxNEBzYW5" https://cancer.sanger.ac.uk/cosmic/file_download/GRCh38/cosmic/v97/CosmicSample.tsv.gz

#The above line gives the download link where AWS access key changes everytime like in this case it was https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&Expires=1679591113&Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D

# Download file with the link from step 2
#curl -o CosmicSample.tsv "https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&Expires=1679591113&Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D"

  1. I want to automate the script first by passing used_id and password as an string
  2. I want to replace the AWS key in the step 3 with a string that is generated from step 2

Thank you


1 Answer 1


The cURL command to obtain the download link returns a JSON containing the URL. This can be parsed on the command line using jq and used to construct a command to download the file. For example, with a list of URLs in a file (one per line) called urls.txt, you could try the following:

while IFS= read -r url ; do
    >&2 echo "Downloading: ${url}"

    curl -sSH @cosmic_header "${url}" |
    jq --arg base "$(basename "${url}")" -r '"curl -o \($base) \(.url | @sh)"' |
done < urls.txt

Note that the above stores the HTTP header in a file called cosmic_header, which can be created using:

echo "Authorization: Basic $(echo 'username:password' | base64)" > cosmic_header

This avoids exposing the authorization key on the command line when you go to download your list of files, which might be preferable if working in a shared environment.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.