How to debug a bash script

Question

Please somebody debug my code

In sub-folders there are 500000 vcf files from which I only need 500 vcf files listed in temp.txt which randomly distributed in sub-folders 10 to 60. The code reads each of 500000 vcf files one by one, if a file was in temp.txt, the code runs

dx run swiss-army-knife -icmd="tabix -h $vcf -R $REGIONS_FILE > $output_file" -iin="$filename_root" -iin="$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$OUTPUT_FOLDER" --detach --priority high --yes
    done < "$VCF_LIST_FILE"

on that if not, the code goes to next file such a process takes years. I want this code only locates files listed in temp.txt and run the main code on them by iteration.

#!/bin/bash

# Set the working directory to the desired location within DNAnexus

# Number of concurrent processes
N=4

# Iterate from 10 to 60
for folder_number in {10..60}; do
  (
    # Define the input files and parameters for this iteration
    folder_path="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/$folder_number"
    VCF_LIST_FILE="temp.txt"
    REGIONS_FILE='file.bed'
    OUTPUT_FOLDER='/out'  # Replace with your desired output folder

    # Set the working directory to the folder for this iteration
    dx cd "$folder_path"

    # Read each VCF file from the list
    while read -r vcf; do
      filename_root=$(basename "$vcf")
      output_file="${filename_root}_extracted.vcf.gz"

      # Use dx run swiss-army-knife to execute tabix on the VCF file
      dx run swiss-army-knife -icmd="tabix -h $vcf -R $REGIONS_FILE > $output_file" -iin="$filename_root" -iin="$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$OUTPUT_FOLDER" --detach --priority high --yes
    done < "$VCF_LIST_FILE"
  ) &

  # Limit the number of concurrent processes
  ((i=i%N)); ((i++==0)) && wait
done

# Wait for all background jobs to finish
wait

# Finished message
echo "++ Finished extracting genomic coordinates from VCF files"

This is how temp.txt looks. The first two digits of vcf file names defines which sub-folder they are located. For instance, the below vcf files distributed in sub-folders 32, 33 and 34

head temp_new.txt
head: temp_new.txt: No such file or directory
izadia@LV19Y7325V Downloads % head new_temp.txt
11/1195100_24053_0_0.dragen.hard-filtered.vcf.gz
12/1204363_24053_0_0.dragen.hard-filtered.vcf.gz
12/1246629_24053_0_0.dragen.hard-filtered.vcf.gz
13/1311312_24053_0_0.dragen.hard-filtered.vcf.gz
14/1427973_24053_0_0.dragen.hard-filtered.vcf.gz
14/1435043_24053_0_0.dragen.hard-filtered.vcf.gz
15/1558824_24053_0_0.dragen.hard-filtered.vcf.gz
16/1683316_24053_0_0.dragen.hard-filtered.vcf.gz
16/1692707_24053_0_0.dragen.hard-filtered.vcf.gz
17/1749825_24053_0_0.dragen.hard-filtered.vcf.gz
izadia@LV19Y7325V Downloads % tail new_temp.txt
52/5230856_24053_0_0.dragen.hard-filtered.vcf.gz
52/5257892_24053_0_0.dragen.hard-filtered.vcf.gz
54/5453459_24053_0_0.dragen.hard-filtered.vcf.gz
55/5514349_24053_0_0.dragen.hard-filtered.vcf.gz
55/5526010_24053_0_0.dragen.hard-filtered.vcf.gz
57/5718723_24053_0_0.dragen.hard-filtered.vcf.gz
58/5868836_24053_0_0.dragen.hard-filtered.vcf.gz
58/5875882_24053_0_0.dragen.hard-filtered.vcf.gz
58/5889611_24053_0_0.dragen.hard-filtered.vcf.gz
59/5917345_24053_0_0.dragen.hard-filtered.vcf.gz
izadia@LV19Y7325V Downloads % 

After running the script I get this error

dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1195100_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1204363_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1246629_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1311312_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1427973_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1435043_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1558824_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1683316_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1692707_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1749825_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID

because the codes starts searching from sub-folder 10 for each of files in new_temp.txt one by one

Answer 1

The first two digits of vcf file names defines which sub-folder they are located.

You could use this information to edit temp.txt so it contains the full path:

grep -v ".tbi$" temp.txt | sed 's/^\(.\)\(.\)/\1\2\/\1\2/' > new_temp.txt

I'm assuming here that you don't actually want to run dx run swiss-army-knife on the vcf indexes.

This will turn the example of temp.txt you gave into this:

32/3295094_24053_0_0.dragen.hard-filtered.vcf.gz
33/3309510_24053_0_0.dragen.hard-filtered.vcf.gz
34/3468518_24053_0_0.dragen.hard-filtered.vcf.gz
34/3471960_24053_0_0.dragen.hard-filtered.vcf.gz

You can then simply run a single for loop over that, instead of the nested one you are using now.

eg.

while read vcf
  do
    dx run swiss-army-knife [...]
  done <  new_temp.txt