I have a bash script that I would like to parellelize to run on multiple nodes. My goal is to run my python sample_script.py script on pairwise comparisons of samples to see if their variants are a match or not (output is written by echo for the function so we can compare results). I unfortunately can’t post the contents of the Python script, but it is basically comparing germline variants between samples to determine if a sample is a match or not. I’m trying to figure out how to make the script run faster or run multiple pair wise comparisons at once then concatenate results. The samples I'm comparing are listed in the array. Running the script on multiple samples takes too long, so I'd like to qsub the process or run the script on multiple nodes, then concatenate the results. Can someone advise on the best way to tackle this? I don't have much experience with jobs using qsub. Should I consider an SGE array job and if so how would I go about this?
Example code below using bash within jupyter notebook:
%%bash
array=( A037375001 A037646101 A037710601 )
for i in "${array[@]}"
do
for j in "${array[@]}"
do
python3 sample_script.py --run-sample-id-A "$i" --run-sample-id-B "$j"
echo run-sample-id-A "$i" run-sample-id-B "$j"
done
done > "/outputdir/out.csv"
Thank you in advance!