Passing strings for password, email authentication and download link

Question

Hi I am downloading the data from a website which requires authentication of email_id and password before the download. They then provide the download link which is used to download dataset. This download link is different every time

### Download Cosmic Files

##First you need to login with your login details and the access key ID may vary with each login ID

##STEP-1
echo "pr@gmail.com:abc123@" | base64
#The above lines gives a authorization key cHIxNEBzYW5

##STEP-2
# get the download link using the authorization key from step 1
#curl -H "Authorization: Basic cHIxNEBzYW5" https://cancer.sanger.ac.uk/cosmic/file_download/GRCh38/cosmic/v97/CosmicSample.tsv.gz

#The above line gives the download link where AWS access key changes everytime like in this case it was https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&Expires=1679591113&Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D

##STEP-3
# Download file with the link from step 2
#curl -o CosmicSample.tsv "https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&Expires=1679591113&Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D"

1. I want to automate the script first by passing used_id and password as an string
2. I want to replace the AWS key in the step 3 with a string that is generated from step 2

Thank you

Answer 1

The cURL command to obtain the download link returns a JSON containing the URL. This can be parsed on the command line using jq and used to construct a command to download the file. For example, with a list of URLs in a file (one per line) called urls.txt, you could try the following:

while IFS= read -r url ; do
    >&2 echo "Downloading: ${url}"

    curl -sSH @cosmic_header "${url}" |
    jq --arg base "$(basename "${url}")" -r '"curl -o \($base) \(.url | @sh)"' |
    sh
done < urls.txt

Note that the above stores the HTTP header in a file called cosmic_header, which can be created using:

echo "Authorization: Basic $(echo 'username:password' | base64)" > cosmic_header

This avoids exposing the authorization key on the command line when you go to download your list of files, which might be preferable if working in a shared environment.