Downloading URL using AWK for fixed fields

Question

I have a text file as follows

a   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz
b   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz
c   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz

The file is tab delimited and has 8 columns with 8th column being the url link I need to download the files one by one using wget and changing the name of the downloaded file with -O option in wget. The argument I want to give to the -O option is combination col1 and col2 separated by _ so that for first line I get the downloaded file as a_k.fastq.gz and so on. For example, the first line of the file above should give:

wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz

Following the post from here I realized looping through the file is slow and sometimes difficult. How can I download these files using wget with awk or cut as the tool for fixed fields

Answer 1

awk '{print "wget " $9 " -O " $1 "_" $2 ".fastq.gz"}' tmp 

will give you

wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
wget https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz -O b_k.fastq.gz
wget https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz -O c_k.fastq.gz

If you add a | bash to the command it will then execute all the wget commands.

Answer 2

A variation of Bioathlete’s answer that doesn't suffer from escaping issues:

awk 'BEGIN{ FS="\t" } { printf("%s\t%s_%s.fastq.gz\n", $8, $1, $2) }' input |
while IFS=$'\t' read -r url filename; do
    wget -O "$filename" -- "$url"
done

Or entirely with shell parsing:

while IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 url; do
    wget -O "${c1}_${c2}.fastq.gz" -- "$url"
done < input

Or entirely without shell parsing but with GNU xargs:

awk 'BEGIN{ FS="\t" } { printf("%s_%s.fastq.gz\n%s\n", $1, $2, $8) }' input |
xargs -r -d '\n' -n 2 -- wget -O

… you could even run multiple wget instances in parallel from GNU xargs with the -P option, though you should set the -q option on wget to disable progress bars overwriting each other on the same terminal. Then again it would be better to use a real command-line download manager like Aria or Axel for that task.