hgsql not found when running a script despite being accessible on command-line

Question

I am working on macOS. I am following the steps described here to lift over an annotation from the reference genome of C. elegans to another assembly of C. elegans.

I have run this code:

time (doSameSpeciesLiftOver.pl -verbose=2 -buildDir=`pwd` \
      -ooc=`pwd`/${target}.ooc -fileServer=localhost -localTmp="/dev/shm" \
  -bigClusterHub=localhost -dbHost=localhost -workhorse=localhost \
  -target2Bit=`pwd`/${target}.2bit -targetSizes=`pwd`/${target}.chrom.sizes \
  -query2Bit=../${query}/${query}.2bit \
  -querySizes=../${query}/${query}.chrom.sizes ${target} ${query}) > do.log 2>&1

However, the do.log file contains:

zsh:1: command not found: hgsql
HgStepManager: executing from step 'align' through step 'cleanup'.
HgStepManager: executing step 'align' Sun Nov 11 15:38:41 2018.
Using localhost, /Users/username/Documents/projects/main/output/software/liftover/data/genomes/GCA_000004515.3_Glycine_max_v2.0/GCA_000004515.3_Glycine_max_v2.0.2bit and ../GCF_000004515.3_V1.1/GCF_000004515.3_V1.1.2bit
# chmod a+x /Users/username/Documents/projects/main/output/software/liftover/data/genomes/GCA_000004515.3_Glycine_max_v2.0/run.blat/job.csh
# chmod a+x /Users/username/Documents/projects/main/output/software/liftover/data/genomes/GCA_000004515.3_Glycine_max_v2.0/run.blat/doAlign.csh
# ssh -x -o 'StrictHostKeyChecking = no' -o 'BatchMode = yes' localhost nice /Users/username/Documents/projects/main/output/software/liftover/data/genomes/GCA_000004515.3_Glycine_max_v2.0/run.blat/doAlign.csh
cd /Users/username/Documents/projects/main/output/software/liftover/data/genomes/GCA_000004515.3_Glycine_max_v2.0/run.blat
rm -rf tParts
/Users/username/Documents/projects/main/output/software/liftover/data/scripts/partitionSequence.pl 58018742 0 /Users/username/Documents/projects/main/output/software/liftover/data/genomes/GCA_000004515.3_Glycine_max_v2.0/GCA_000004515.3_Glycine_max_v2.0.2bit /Users/username/Documents/projects/main/output/software/liftover/data/genomes/GCA_000004515.3_Glycine_max_v2.0/GCA_000004515.3_Glycine_max_v2.0.chrom.sizes 2000 -lstDir=tParts
rm -rf qParts
/Users/username/Documents/projects/main/output/software/liftover/data/scripts/partitionSequence.pl 10000000 0 ../GCF_000004515.3_V1.1/GCF_000004515.3_V1.1.2bit ../GCF_000004515.3_V1.1/GCF_000004515.3_V1.1.chrom.sizes 1000 -lstDir=qParts
Can't open seq size file ../GCF_000004515.3_V1.1/GCF_000004515.3_V1.1.chrom.sizes: No such file or directory
Command failed:
ssh -x -o 'StrictHostKeyChecking = no' -o 'BatchMode = yes' localhost nice /Users/username/Documents/projects/main/output/software/liftover/data/genomes/GCA_000004515.3_Glycine_max_v2.0/run.blat/doAlign.csh

I think the biggest problem is that the hgsql command is not found.

If I run hgsql on the command line, I find it even though I get the following error:

dyld: Library not loaded: /usr/local/opt/mysql/lib/libmysqlclient.20.dylib
  Referenced from: /usr/local/bin/hgsql
  Reason: image not found
[1]    34059 abort      hgsql

Location of the hgsql installation:

which hgsql
/Users/username/Documents/projects/main/output/software/liftover/data/bin/hgsql

Can anyone help me with using hgsql, please?

Answer 1

It is possible that the shell being used in the process is different from your standard command line shell. I am not familiar with the tool, and it's hard to say more without knowing your OSX version, but it looks like it is using zsh. I think that zsh only fairly recently became the standard OSX shell. I don't know the tool or the shell very well, but it looks like you might be able to explicitly set up hgsql in your PATH in your .zshrc.

It's also possible that the liftover script was not built to run via zsh. I notice that it is trying to run csh scripts as part of the workflow. Then again the localhost may insulate it.

So I think that my first step would be to sort out which shell I am using, whether it works with the liftover pipeline, and how it's running on the localhost. I notice that the ssh call running the csh script on the localhost is the step that actually fails. What shell is executing the csh script? Does that shell know about hgsql? It could be that the way that the error bubbles up it is hiding which shell it is actually using.

I'd recommend trying that ssh call in isolation from the rest of the pipeline, and diagnosing the error that way.

Hope that helps.