1
$\begingroup$

I would like to try an use Singularity with CNV kit as the HPC I use seems to work better with Singularity:

Initial Attempt:

singularity exec ~/CNV_kit/cnvkit_latest.sif /bin/bash -c "ls /"
  • Result:
bin  boot  dev  environment  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  singularity  srv  sys  tmp  usr  var

Attempt 1:

singularity exec ~/CNV_kit/cnvkit_latest.sif /bin/bash -c "cd /proc/ && ls"

This results in a folder with a series of numbers but not the cnvkit py files.

Attempt 2:

singularity exec ~/CNV_kit/cnvkit_latest.sif /bin/bash -c "cd /singularity/ && ls"

This results in /bin/bash: line 1: cd: /singularity/: Not a directory

Attempt 3:

 singularity exec ~/CNV_kit/cnvkit_latest.sif /bin/bash -c "cd /media/ && ls"

This results in nothing being printed to the screen. I tried this command three times to be sure.

How can I find the Singularity path I would need to use with singularity exec to run from the Singularity image.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I like this question a lot. Firstly I don't know singularity, but I know Docker and it will be similar. docker exec is very similar.

So for example the reasons are likely as follows:

  • media could be empty and hence no output.
  • the Python script may not be in /proc
  • singularity is likely to be a file rather than a directory.

One solution is,

singularity exec ~/CNV_kit/cnvkit_latest.sif /bin/bash -c "ls -l /"

i.e. ask it which are the directories. Another solution is..

singularity exec ~/CNV_kit/cnvkit_latest.sif /bin/bash -c "find / -name 'cnvkit*'"

This will locate cnvkit.py, note the use of ' and " for this sort of command.

Personally, I would dive into the container and issue the command of interest, manually. This way you get a feel for how it's constructed and don't need to keep issuing commands from outside the container. In Docker this is easy and it's impossible to keep a third party out of a container (even if the developer tries). In singularity it is likely similar.

From the notes I would simply access the singularity container directly.

Accessing a singularity container is different from Docker (I looked it up) ...

singularity shell ~/CNV_kit/cnvkit_latest.sif

Once inside you'll be able to look around, try out the find command, look in the bin etc... There will be no need to issue commands from outside the container to get it working.

From further comments,

A node on an HPC is a group of usually 8 to 32+ CPUs. What you have is a 'master' or 'head' node, where you log into and keep your data, and the stuff you do the work on is called 'nodes'. It has a more specific name, but it's not very cool, so we don't use it. The HPC will comprise a large number of nodes, e.g. 100+, but can be smaller particularly with the massive CPU machines around now. A node concept is a bit different on a NVidia machine if its part of a cluster.

The issue is someone could have called a script 'node', which would be confusing. It's complicated, because someone could have their own research machine which is attached and maintained on. the HPC, but only they can access and called it e.g. 'Joes_node', of 'Jills_node'.

Personally, I would sit down with your administrator and ask them to explain singularity and this the comments herein and this. They will know that at admin level (thats their job).

From my point of view, if I'd access to a cluster using singularity I could figure it out. I use GCP now, it's geeky but cool, so my information is out of date on non-Cloud clusters.

RamRS comments What RamRS is saying and what I'm trying to assess whether you are within a singularity container. Again, I don't know the architecture.

Merging With Accepted Answer:

The main file in cnvkit is cnvkit.py in order to find that you need to look using singularity exec. The following command gives the path to the cnvkit.py file:

singularity exec cnvkit_latest.sif /bin/bash -c "cd /usr/local/bin/ && ls"

An example of how to use this path is below:

# Directory containing sample files:
directory="/path/to/bam/files/folder"

# Define path to singularity image:
cnvkit_container="/path/to/cnvkit_latest.sif"

cnvkit_path="/usr/local/bin/cnvkit.py"

singularity exec "$cnvkit_container" "$cnvkit_path" call "$directory/sample.cns" -o "$directory/sample.call.cns"
Improve this answer
$\endgroup$
23
  • 1
    $\begingroup$ Use singularity shell to find and test the executable, then use singularity exec to use it. $\endgroup$
    – Ram RS
    May 26 at 21:31
  • 1
    $\begingroup$ @Indira if you use the shell command it will give you access to the container - it operates just like an enclosed system (because it is). You will then be able to figure out how to run it via exec. If you try the shell command you will see after e.g. pwd. $\endgroup$
    – M__ ♦
    May 26 at 21:31
  • 1
    $\begingroup$ @M__ I was unaware that singularity shell is a thing, thank you so much! $\endgroup$
    – Ram RS
    May 26 at 21:34
  • 1
    $\begingroup$ Are you sure you're running the command from within the singularity shell? Compare the output to singularity exec file.sif /bin/bash -c "ls /" run from your HPC shell and a simple ls / from within singularity shell. The output should be identical $\endgroup$
    – Ram RS
    May 26 at 22:09
  • 1
    $\begingroup$ @M__ The nomenclature is now head and worker nodes. Even GitHub has now renamed the principal code branch to main, so the nomenclature you're censoring is outdated. $\endgroup$
    – Ram RS
    May 30 at 16:51
0
$\begingroup$

Thanks to OP for sharing that their image is from https://hub.docker.com/r/etal/cnvkit

I downloaded this and it seems pretty simple.

My tests:

singularity pull docker://etal/cnvkit:latest
$ singularity shell ./cnvkit_latest.sif
INFO:    Converting SIF file to temporary sandbox...
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Singularity> ls /
bin  boot  dev  environment  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  singularity  srv  sys  tmp  usr  var
Singularity> ls /singularity
/singularity
Singularity> cnvkit.py
usage: cnvkit.py [-h]
                 {batch,target,access,antitarget,autobin,coverage,reference,fix,segment,call,diagram,scatter,heatmap,breaks,genemetrics,gainloss,sex,gender,metrics,segmetrics,bintest,import-picard,import-seg,import-theta,import-rna,export,version}
                 ...

CNVkit, a command-line toolkit for copy number analysis.

positional arguments:
  {batch,target,access,antitarget,autobin,coverage,reference,fix,segment,call,diagram,scatter,heatmap,breaks,genemetrics,gainloss,sex,gender,metrics,segmetrics,bintest,import-picard,import-seg,import-theta,import-rna,export,version}
                        Sub-commands (use with -h for more info)
    batch               Run the complete CNVkit pipeline on one or more BAM files.
    target              Transform bait intervals into targets more suitable for CNVkit.
    access              List the locations of accessible sequence regions in a FASTA file.
    antitarget          Derive off-target ("antitarget") bins from target regions.
    autobin             Quickly calculate reasonable bin sizes from BAM read counts.
    coverage            Calculate coverage in the given regions from BAM read depths.
    reference           Compile a coverage reference from the given files (normal samples).
    fix                 Combine target and antitarget coverages and correct for biases. Adjust raw coverage data according to the given reference, correct potential biases and re-center.
    segment             Infer copy number segments from the given coverage table.
    call                Call copy number variants from segmented log2 ratios.
    diagram             Draw copy number (log2 coverages, segments) on chromosomes as a diagram. If both the raw probes and segments are given, show them side-by-side on each chromosome (segments on the left side, probes on the right
                        side).
    scatter             Plot probe log2 coverages and segmentation calls together.
    heatmap             Plot copy number for multiple samples as a heatmap.
    breaks              List the targeted genes in which a copy number breakpoint occurs.
    genemetrics         Identify targeted genes with copy number gain or loss.
    sex                 Guess samples' sex from the relative coverage of chromosomes X and Y.
    metrics             Compute coverage deviations and other metrics for self-evaluation.
    segmetrics          Compute segment-level metrics from bin-level log2 ratios.
    bintest             Test for single-bin copy number alterations.
    import-picard       Convert Picard CalculateHsMetrics tabular output to CNVkit .cnn files. The input file is generated by the PER_TARGET_COVERAGE option in the CalculateHsMetrics script in Picard tools. If 'antitarget' is in the
                        input filename, the generated output filename will have the suffix '.antitargetcoverage.cnn', otherwise '.targetcoverage.cnn'.
    import-seg          Convert a SEG file to CNVkit .cns files.
    import-theta        Convert THetA output to a BED-like, CNVkit-like tabular format. Equivalently, use the THetA results file to convert CNVkit .cns segments to integer copy number calls.
    import-rna          Convert a cohort of per-gene log2 ratios to CNVkit .cnr format.
    export              Convert CNVkit output files to another format.
    version             Display this program's version.

options:
  -h, --help            show this help message and exit

See the online manual for details: https://cnvkit.readthedocs.io
Singularity> which cnvkit.py
/usr/local/bin/cnvkit.py

As you can see, since cnvkit.py is in /usr/local/bin/, simply typing cnvkit.py within the shell works. Let's try an exec with no frills:

$ singularity exec cnvkit_latest.sif cnvkit.py
INFO:    Converting SIF file to temporary sandbox...
usage: cnvkit.py [-h]
                 {batch,target,access,antitarget,autobin,coverage,reference,fix,segment,call,diagram,scatter,heatmap,breaks,genemetrics,gainloss,sex,gender,metrics,segmetrics,bintest,import-picard,import-seg,import-theta,import-rna,export,version}
                 ...

CNVkit, a command-line toolkit for copy number analysis.

positional arguments:
  {batch,target,access,antitarget,autobin,coverage,reference,fix,segment,call,diagram,scatter,heatmap,breaks,genemetrics,gainloss,sex,gender,metrics,segmetrics,bintest,import-picard,import-seg,import-theta,import-rna,export,version}
                        Sub-commands (use with -h for more info)
    batch               Run the complete CNVkit pipeline on one or more BAM files.
    target              Transform bait intervals into targets more suitable for CNVkit.
    access              List the locations of accessible sequence regions in a FASTA file.
    antitarget          Derive off-target ("antitarget") bins from target regions.
    autobin             Quickly calculate reasonable bin sizes from BAM read counts.
    coverage            Calculate coverage in the given regions from BAM read depths.
    reference           Compile a coverage reference from the given files (normal samples).
    fix                 Combine target and antitarget coverages and correct for biases. Adjust raw coverage data according to the given reference, correct potential biases and re-center.
    segment             Infer copy number segments from the given coverage table.
    call                Call copy number variants from segmented log2 ratios.
    diagram             Draw copy number (log2 coverages, segments) on chromosomes as a diagram. If both the raw probes and segments are given, show them side-by-side on each chromosome (segments on the left side, probes on the right
                        side).
    scatter             Plot probe log2 coverages and segmentation calls together.
    heatmap             Plot copy number for multiple samples as a heatmap.
    breaks              List the targeted genes in which a copy number breakpoint occurs.
    genemetrics         Identify targeted genes with copy number gain or loss.
    sex                 Guess samples' sex from the relative coverage of chromosomes X and Y.
    metrics             Compute coverage deviations and other metrics for self-evaluation.
    segmetrics          Compute segment-level metrics from bin-level log2 ratios.
    bintest             Test for single-bin copy number alterations.
    import-picard       Convert Picard CalculateHsMetrics tabular output to CNVkit .cnn files. The input file is generated by the PER_TARGET_COVERAGE option in the CalculateHsMetrics script in Picard tools. If 'antitarget' is in the
                        input filename, the generated output filename will have the suffix '.antitargetcoverage.cnn', otherwise '.targetcoverage.cnn'.
    import-seg          Convert a SEG file to CNVkit .cns files.
    import-theta        Convert THetA output to a BED-like, CNVkit-like tabular format. Equivalently, use the THetA results file to convert CNVkit .cns segments to integer copy number calls.
    import-rna          Convert a cohort of per-gene log2 ratios to CNVkit .cnr format.
    export              Convert CNVkit output files to another format.
    version             Display this program's version.

options:
  -h, --help            show this help message and exit

See the online manual for details: https://cnvkit.readthedocs.io
INFO:    Cleaning up image...

The answer seems to be a lot easier than having to find cnvkit.py, as it is already in the PATH.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This did not work for me as it seems to work differently on the HPC I am on. $\endgroup$
    – Indira
    Jun 1 at 16:20
  • $\begingroup$ That does not make much sense. There could be something unusual with your HPC. Perhaps you should contact your sysadmin. $\endgroup$
    – Ram RS
    Jun 1 at 18:19
  • $\begingroup$ I have done so and I simply have to navigate around these issues using the path inside the singularity image. $\endgroup$
    – Indira
    Jun 2 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.