0
$\begingroup$

I am setting up a HiChIPseq pipeline, but installing software necessary for the HiC side of things has been unbecoming.

Both my PI and I have tried to install Hi-C Pro onto our Mac Pro that we use as our compute server (though in a couple of weeks we hope to migrate to an EPYC 64 Core computer server of our own!).

There are three approaches that I have taken to install Hi-C Pro.

  1. From Source

To install from source, you need to supply the paths to its dependencies. Believing we did a good job, we run it and find problems accessing bx-python.

No matter how many times we update pip and try to install bx-python and specify that we want it installed for python2.

  1. From singularity

This is the one I tried. From what I understand this creates a VM with the necessary dependencies installed, and allows you to use the program without building. However, it appears to abort when conda tries to install numpy.

+ conda install -y -c anaconda numpy=1.16.3
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): / Killed
ABORT: Aborting with RETVAL=255
Cleaning up...

I have seen the failed solving environment bug many times, but have no idea how to solve it. I have tried updating conda, but that doesn't solve it.

  1. Install Someone's Premade Conda Environment

Running conda.sh give me this:

Solving environment: failed

ResolvePackageNotFound:
<Lots of packages...>

Sorry if this is sloppy.

$\endgroup$
  • $\begingroup$ Is the conda environment running under python2 or 3? $\endgroup$ – ATpoint Sep 14 at 15:57
  • $\begingroup$ @ATpoint python2 $\endgroup$ – Jeff Sep 14 at 16:24
1
$\begingroup$

I could install HiC-Pro using conda when creating a separate environment (which I always do for side-projects), and then pull the conda build of HiC-Pro:

conda create --name HiCPro python=2.7
conda activate HiCPro
conda install -c davebx hicpro

Not sure what conda.sh is that you describe, sometimes tricky to get random code from GitHub running. Give it a go with a standard conda install as I described.

| improve this answer | |
$\endgroup$
1
$\begingroup$

If you're reading this, I can only assume that's because you had trouble installing HiC-Pro too. I was finally able to install from source and run the software, so here is how I did it, giving detail to the error that I was hit with and the clues I used to solve the puzzles and get through installing this program:

From Source To install from source, you need to supply the paths to its dependencies. Believing we did a good job, we run it and find problems accessing bx-python.

No matter how many times we update pip and try to install bx-python and specify that we want it installed for p

Setup

I downloaded from source once again using git clone.

The I cated the config file to find the programs I needed to find the path of, which I did using which. I ensured that I was using python2.

I also used pip2 to install pysam, numpy, and scipy. I needed to use sudo to install pysam, though. I also installed something called bx-python.

Make Configure

Here is the error I get when I run make configure

Checking Python libraries ...
Traceback (most recent call last):
File "../scripts/install/check_pythonlib.py", line 24, in <module>
      import bx.intervals
  ModuleNotFoundError: No module named 'bx'

  Traceback (most recent call last):
    File "../scripts/install/check_pythonlib.py", line 28, in <module>
      raise ImportError('Error - bx-python cannot be imported')  
ImportError: Error - bx-python cannot be imported
Can not proceed without the required Python libraries, please install them and re-run
make[1]: *** [configure] Error 1
make: *** [configure] Error 2

Looking at this error for the fifth time, I realize that the first part, and not the last part, is the important one. It's yelling at me that I don't have the bx module installed. It also gives me the path to the file that threw the exception, so I read it and saw that this was what was being run:

## bx
try:
 import bx.intervals
 if vcmp(bx.__version__, '0.5.0') == -1:
   raise ValueError('bx-python '+bx.__version__+' detected. Version >= 0.5.0 required')
 except ImportError:
   raise ImportError('Error - bx-python cannot be imported')

I notice that there's nothing fancy, the thing just needs to be imported, and its actual name is bx. Wonderful, I pip2 it. But... I hit another error in pip:

pyrsistent requires Python '>=3.5' but the running Python is 2.7.16

Apparently, pyrsistent dropped support for python2. Wonderful. I get around it by:

sudo pip2 install awsebcli pyrsistent==0.16.0

In the middle of this installation, Im reading the output and see:

bx-python 0.8.9 has requirement six>=1.13.0, but you'll have six 1.11.0 which is incompatible.

So:

sudo pip2 install six==1.13.0

At this point, I, without any more problems, sudo pip2 install bx.

I run make configure again, and I run into...

Checking Python libraries ...
Traceback (most recent call last):
File "../scripts/install/check_pythonlib.py", line 24, in <module>
      import bx.intervals
  ModuleNotFoundError: No module named 'bx'

  Traceback (most recent call last):
    File "../scripts/install/check_pythonlib.py", line 28, in <module>
      raise ImportError('Error - bx-python cannot be imported')  
ImportError: Error - bx-python cannot be imported
Can not proceed without the required Python libraries, please install them and re-run
make[1]: *** [configure] Error 1
make: *** [configure] Error 2

huh? Didn't I fix this?? Something seems fishy. For whatever reason, my mind then wanders to the Makefile to see if anything there pops out, cause... what else would be calling that python file? In the main Makefile I see:

make -f ./scripts/install/Makefile CONFIG_SYS=$(CONFIG_SYS)

So I head on over to ./scripts/install/Makefile and see:

./scripts/install/install_dependencies.sh -c $(CONFIG_SYS) -o  $(realpath $(PREFIX))/HiC-Proo    _$(VNUM) -q

Nothing like a goose chase. So I head on over to install_dependencies.sh, where I see:

# python
which python > /dev/null;
if [ $? != "0" ]; then
     echo -e "$RED""Can not proceed without Python, please install and re-run""$NORMAL"
     exit 1;
else
     pver=`python --version 2>&1 | cut -d" " -f2` #<- THIS LINE
     vercomp $pver "2.7.0"
     if [[ $? == 2 ]]; then
         echo -e "$RED""Python v2.7.0 or higher is needed [$pver detected].""$NORMAL"
         exit 1;
     fi
fi

I check that lines output, AND IT'S PYTHON3!! For some reason, it didn't give me that error message, it just kept going from that line.

So, I changed all instances of python with python2 in that file.

I reran make configure and got The required Python libraries appear to be already installed. Bliss.

Make Install

Next step was make install, and guess what? A roadblock:

Traceback (most recent call last):
File "setup.py", line 2, in <module>
from numpy.distutils.core import setup
  ModuleNotFoundError: No module named 'numpy'
make: *** [iced] Error 1

Hmm... well, it's probably the same thing. python3 being called where python2 should be. So I double checked the main Makefile and changed the setup.py call to use python2.

I run make install and it runs for awhile, chugging along using the g++ compiler. Then I hit an odd error:

cp: /Users/bosslab/Downloads/HiC-Pro-master/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4/HiC-Pro_2.11.4: name too long (not copied)

I'm guessing that the Makefile has a recursive path somewhere. I check the appropriate section and find:

cp -Ri $(MK_PATH) $(INSTALL_PATH)

I simply replace that line with the actual paths, with the install_path being adjacent instead of inside the mk_path.

Then I get hit with:

HiC-Pro installed in /Users/bosslab/Downloads/HiC-Pro-master/HiC-Pro_2.11.4 !

So running $install_dir/HiC-Pro --help returns the help. YAY!

Leftovers

I installed it, but how about when I actually run it?

Error: The 'ice' command is not in your path. Please check where the 'iced' python package has been installed and update your PATH !

Ok:

$ sudo pip2 install iced
Requirement already satisfied: iced in /Users/bosslab/Library/Python/2.7/lib/python/site-packages (0.4.2)

...so, I set up a test to see whether it actually is installed:

$ python2
Python 2.7.16 (default, Nov  9 2019, 05:55:08) 
[GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.32.4) (-macos10.15-objc-s on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> iimport iced
>>> quit()

It's already installed. But then I had an epiphany:

$ python3
Python 3.8.5 (default, Jul 21 2020, 10:48:26) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ipomport iced
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'iced'
>>> quit()

Aha! Somewhere (i don't know where though) python3 is being called. So I just installed that module for python3 with sudo pip3 install iced.

Then I had some trouble with pointing to the right bowtie index (just give the directory where the index exists) and then I hit my next error dealing with HiC-Pro itself:

Pairing of R1 and R2 tags ...
Logs: logs/dixon_2M/mergeSAM.log
make: *** [bowtie_pairing] Error 1

Checking the log file named in the error message:

/usr/local/bin/python /Users/bosslab/Downloads/HiC-Pro-master/HiC-Pro_2.11.4/scripts/mergeSAM. py -q 0 -t -v -f bowtie_results/bwt2/dixon_2M/SRR400264_00_R1_hg19.bwt2merged.bam -r bowtie_re sults/bwt2/dixon_2M/SRR400264_00_R2_hg19.bwt2merged.bam -o bowtie_results/bwt2/dixon_2M/SRR400 264_00_hg19.bwt2pairs.bam
   File "/Users/bosslab/Downloads/HiC-Pro-master/HiC-Pro_2.11.4/scripts/mergeSAM.py", line 26
   print "Usage : python mergeSAM.py"
    ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Usage : python mergeS AM.py")?

Missing parentheses? Oh my god, it's using python3 for some reason again. I grep the installation directory for a file that calls merge_sam.py, and find config-system.txt. I go in there and just need to change the python path from /usr/bin/local/ where python3 is installed to /usr/bin/ where python2 is installed.

Lastly, I get an error during the ICE normalization step that numpy is not installed. Using my big fat brain, I isntall numpy to my python3 and call it a day. Now it all works as intended.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Yikes, this sounds like an awful time troubleshooting. Glad you were able to solve your issue and post a solution here $\endgroup$ – James Hawley Sep 21 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.