Skip to content

Using containers

Running commands inside a container

The exec command will allow you to execute any program within a given container. The run command performs the action defined by the %runscript section, which is the primary task of the container. Using the run command is the simpler approach for job submissions.

If you have the Singularity module loaded, you can even "execute" a container, which performs the same action as the singularity run command:

In this example we have a container called pandoc.img:

module load singularity
$ singularity inspect -r pandoc.img
#!/bin/sh
    pandoc "${@}"
$ ./pandoc.img --version | head -1
pandoc 1.16.0.2

Execute a script from outside the container

Using on the cluster

For typical use, you want to use runscripts or exec commands, especially when submitting the work via Grid Engine.

The following example runs a python script from the current directory.

$ singularity exec ./centos7-python-3.4.el7.img python hello_world.py
Hello, World!

Note that home directories and current directories are seamlessly available from within the container.

Using with Grid Engine

One of the major benefits of Singularity is the simplicity with which it can be used in an HPC environment. Your Grid Engine qsub file only needs to load the Singularity module and you can run your container. The resource requirements should be very similar to native code.

Simple example

Example:

#$ -S /bin/bash
#$ -cwd
#$ -j y
#$ -pe smp 1
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G

# create a PDF from a markdown file called example.md
# using pandoc inside a container
module load singularity
singularity run /data/containers/pandoc.img example.md -o output.pdf

Detailed example

To begin, a container is prepared to provide python 2.7.15 compiled from source. Containers can be useful to provide later versions of applications than are available from package repositories, particularly as there are often a lot of dependencies required for compilation.

Example container definition file, python-centos7.def

BootStrap: yum
OSVersion: 7
MirrorURL: https://mirror.centos.org/%{OSVERSION}/os/$basearch/
UpdateURL: https://mirror.centos.org/%{OSVERSION}/updates/$basearch/
Include: yum

%environment
    PATH=${PATH}:/usr/local/bin
    PYTHONPATH=/usr/local/lib/python2.7

%help
    Python 2.7.15 compiled from source on Centos7

%post
    yum -y groupinstall "Development Tools"
    yum -y install wget

    # Variables section
    VER="2.7.15"
    FTPSRV="https://www.python.org/ftp/python/${VER}"
    FTP_FILE="Python-${VER}.tgz"
    PYTHON_SHORT_VERSION=${VER%.*}

    # Download, compile and install
    /usr/bin/wget ${FTPSRV}/${FTP_FILE}
    tar xzf ${FTP_FILE}
    cd Python-${VER}
    ./configure && make && make install

%runscript
    /usr/local/bin/python "$@"

After creating the container, we can use it in a scheduler job, using the following submission script:

#!/bin/bash
#$ -cwd
#$ -l h_rt=12:0:0
#$ -l h_vmem=4G
module load singularity
singularity run python-centos7.img python_prime.py

Shell access to the container

We can open an interactive shell within the container. This can be useful for debugging container bootstraps or checking how a container is built.

$ singularity shell ./centos7-python-3.4.el7.img
Singularity: Invoking an interactive shell within container...

Singularity.centos7-python-3.4.el7.img> python3.4

Python 3.4.8 (default, Mar 23 2018, 10:04:27)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Running containers from external sources

Reproducibility

For reproducible results, we advise that you do not rely on externally-sourced containers for performing your research.

Containers created elsewhere can be copied or imported, and run on the cluster. In addition to copying container images around manually, you can import or run remote containers as a proof-of-concept.

Singularity hub will build a container specified by a github repository and allow you to pull an image, or run, exec and shell into an image.

In this example, a Github repository containing a Singularity definition file has been added to Singularity Hub, and a container has automatically been created. This allows rapid prototyping and development of containers.

singularity exec shub://sbutcher/container-R Rscript test.R

Documentation is available on the Singularity Hub Wiki

Support for docker images

You can also import Docker images via the Remote Docker API, or base your containers for ad-hoc development and testing of your containers.

Use of Docker containers for Research

For long term reproducibility of containers we recommend that you do not rely on 3rd party docker containers for your research, instead building your own native Singularity containers from definition files. This improves your chances of rebuilding the container from definition file in future, and reduces risk of upstream changes affecting your ability to recreate the container, such as Docker API changes, and removal or alterations to the original docker container by the creator.

To import the container tagged latest in the Docker Hub:

singularity create ubuntu.img
singularity import ubuntu.img docker://ubuntu:latest