I have a text file as follows

a   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz


The file is tab delimited and has 8 columns with 8th column being the url link I need to download the files one by one using wget and changing the name of the downloaded file with -O option in wget. The argument I want to give to the -O option is combination col1 and col2 separated by _ so that for first line I get the downloaded file as a_k.fastq.gz and so on. For example, the first line of the file above should give:

wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz


Following the post from here I realized looping through the file is slow and sometimes difficult. How can I download these files using wget with awk or cut as the tool for fixed fields

• Can you post up an example wget command? Dec 13 '17 at 19:20
awk '{print "wget " $9 " -O "$1 "_" $2 ".fastq.gz"}' tmp  will give you wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz wget https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz -O b_k.fastq.gz wget https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz -O c_k.fastq.gz  If you add a | bash to the command it will then execute all the wget commands. • This works great. Can you tell what |bash is doing ?? Dec 13 '17 at 19:37 • @user3138373 the output of the awk command is a set of shell commands, essentially a mini script. By piping it to bash with | bash you're telling it to execute each input line as a command. So it will run the wget calls and download the sequences. Dec 13 '17 at 19:43 A variation of Bioathlete’s answer that doesn't suffer from escaping issues: awk 'BEGIN{ FS="\t" } { printf("%s\t%s_%s.fastq.gz\n",$8, $1,$2) }' input |
while IFS=$'\t' read -r url filename; do wget -O "$filename" -- "$url" done  Or entirely with shell parsing: while IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 url; do
wget -O "${c1}_${c2}.fastq.gz" -- "$url" done < input  Or entirely without shell parsing but with GNU xargs: awk 'BEGIN{ FS="\t" } { printf("%s_%s.fastq.gz\n%s\n",$1, $2,$8) }' input |

… you could even run multiple wget instances in parallel from GNU xargs with the -P option, though you should set the -q option on wget to disable progress bars overwriting each other on the same terminal. Then again it would be better to use a real command-line download manager like Aria or Axel for that task.