Running a Python Script from a bash script on multiple input files across different directory paths

Question

I am in the process of writing up a bash script that runs three bioinformatics tools and then executes a python script. This python script compiles some of the data from the informatics tools output into a single merged output file. Now I need to run this on a large number of samples at a time so I am trying to set up the script to read through the directory path to where all the output from the tools is stored.

These are the directory paths for the python script input files:

EFS/Analyses/BGC.test/Sample_Name/antismash2/sample.LargeContigs.gbk
EFS/Analyses/BGC.test/Sample_Name/Bagel4/00.OverviewGeneTables.json
EFS/Analyses/BGC.test/Sample_Name/Deepbgc/Deepbgc.bgc.tsv

Now there are multiple Sample_Name directories containing identical structures in the EFS/Analyses/BGC.test path. I would like to run the script on all of them

To do this, I have written a for loop within the bash script as follows:

sample_paths=$(find $EFS/Analyses/BGC.test -mindepth 1 -maxdepth 1 -type d \ )
while read sample_paths; 
do
   sample=$(basename $sample_paths)
   GenBank_Input="$sample/antismash2/$sample.LargeContigs.gbk";
   TsvFile_Input="$sample/Deepbgc/Deepbgc.bgc.tsv";
   Json_Input="$sample/Bagel4/00.OverviewGeneTables.json";
done <<< “$sample_paths”

export GenBank_Input
export TsvFile_Input
export Json_Input

The goal here being to define the sample path for every sample and then for every sample, looping through to the specific directories of each input file. I then exported the variables and used os.environ from the os package to use them in the python script as such:

    #parse gbk input file
    for record in SeqIO.parse(os.environ["GenBank_Input"], 'genbank'):
    #parse tsv input file
    file_handle = open(os.environ["TsvFile_Input"])
    #parse json input file
    json_data = json.load(open(os.environ["Json_Input"]))

Now the error I get when I run the script is a FileNotFoundError: [Errno 2] No such file or directory: '“”/antismash2/“”.LargeContigs.gbk' so the error seems pretty clear cut. The issue seems to be with the path that I defined within the while loop variables but as of right now, I have been unable to resolve. I am trying to familiarize myself more with bash scripting as I am still fairly new to it. Any guidance on how I might adjust the code to resolve this and get the script working is much appreciated, thanks!

Answer 1

Your loop is more complicated than you need and it won't work properly anyway since you're using the same variable (sample_paths) to iterate and save. Try this instead:

for dir in EFS/Analyses/BGC.test/*/; do ... done

Next, all you really need is the sample names. The rest can be inferred:

sample_names=""
for dir in EFS/Analyses/BGC.test/*/; do
    sample_names+="$(basename "$dir") "
done
## remove the extra space added in the end
sample_names=$(sed 's/  *$//' <<< "$sample_names")
export sample_paths

Then, in python, you would do:

#!/usr/bin/env python3

import os

base_dir="EFS/Analyses/BGC.test"
for sample_name in os.environ["sample_names"].split(' '):
    gbk_file = """%s/%s/antismash2/%s.LargeContigs.gbk""" % (
        base_dir, sample_name, sample_name
    )
    tsv_input = """%s/%s/Deepbgc/Deepbgc.bgc.tsv""" % (base_dir, sample_name)
    json_input = """%s/%s//Bagel4/00.OverviewGeneTables.json""" % (base_dir, sample_name)
    print("============== Sample Name:",sample_name, "=============")
    print("GenBank file:", gbk_file)
    print("Tsv file:", tsv_input)
    print("Tsv file:", json_input)

I created the following test directories:

$ tree
.
└── EFS
    └── Analyses
        └── BGC.test
            ├── Sample_1
            │   ├── antismash2
            │   │   └── sample.LargeContigs.gbk
            │   ├── Bagel4
            │   │   └── 00.OverviewGeneTables.json
            │   └── Deepbgc
            │       └── Deepbgc.bgc.tsv
            ├── Sample_2
            │   ├── antismash2
            │   │   └── sample.LargeContigs.gbk
            │   ├── Bagel4
            │   │   └── 00.OverviewGeneTables.json
            │   └── Deepbgc
            │       └── Deepbgc.bgc.tsv
            └── Sample_3
                ├── antismash2
                │   └── sample.LargeContigs.gbk
                ├── Bagel4
                │   └── 00.OverviewGeneTables.json
                └── Deepbgc
                    └── Deepbgc.bgc.tsv

And running the two snippets of code above produces:

$ foo.py
============== Sample Name: Sample_1 =============
GenBank file: EFS/Analyses/BGC.test/Sample_1/antismash2/Sample_1.LargeContigs.gbk
Tsv file: EFS/Analyses/BGC.test/Sample_1/Deepbgc/Deepbgc.bgc.tsv
Tsv file: EFS/Analyses/BGC.test/Sample_1//Bagel4/00.OverviewGeneTables.json
============== Sample Name: Sample_2 =============
GenBank file: EFS/Analyses/BGC.test/Sample_2/antismash2/Sample_2.LargeContigs.gbk
Tsv file: EFS/Analyses/BGC.test/Sample_2/Deepbgc/Deepbgc.bgc.tsv
Tsv file: EFS/Analyses/BGC.test/Sample_2//Bagel4/00.OverviewGeneTables.json
============== Sample Name: Sample_3 =============
GenBank file: EFS/Analyses/BGC.test/Sample_3/antismash2/Sample_3.LargeContigs.gbk
Tsv file: EFS/Analyses/BGC.test/Sample_3/Deepbgc/Deepbgc.bgc.tsv
Tsv file: EFS/Analyses/BGC.test/Sample_3//Bagel4/00.OverviewGeneTables.json

All this said, I don't really understand why you are using the shell at all. You can implement all of this directly in python:

#!/usr/bin/env python3

import os

base_dir="EFS/Analyses/BGC.test"
sample_names = os.listdir('EFS/Analyses/BGC.test/')

for sample_name in sample_names:
    gbk_file = """%s/%s/antismash2/%s.LargeContigs.gbk""" % (
        base_dir, sample_name, sample_name
    )
    tsv_input = """%s/%s/Deepbgc/Deepbgc.bgc.tsv""" % (base_dir, sample_name)
    json_input = """%s/%s//Bagel4/00.OverviewGeneTables.json""" % (base_dir, sample_name)
    
    # do something with these files here