Using modules

HPC cluster systems typically have a large number of software packages installed which often clash with each other: for example, the commands for OpenMPI and MPICH would overlap if simultaneously installed. We use the modules package to manage the user environment for the different packages which are installed. This makes it possible to use different packages or switch between versions of the same package without conflicts.

To get a list of available packages, use:

module avail

To get a list of currently loaded packages:

module list

To setup your environment to use a package you type:

module load <package>

replacing <package> with the relevant module name. For example to load Matlab use:

module load matlab

To unload a specific module, use:

module unload <package>

replacing <package> with the relevant module name. For example to unload Matlab use:

module unload matlab

To unload all loaded modules, use:

module purge

Default versions

Environment modules also support the concept of default versions. The default version, where it exists, is shown in the output of module avail e.g. module load matlab will load matlab/2014a, since it is marked as default. If you wish to load the 2015a version then you will need to specify module load matlab/2015a.

Sometimes we stick with an older version as the default, since a lot of people may still be using that version, and will require checking of scripts and communication to all users of the package. For popular packages, we rarely make the newest version the default one, due to the potential for introduction of incompatibilities or inconsistent results.

Conflicts

Loading multiple versions of the same module will result in a conflict:

$ module load intel/15.0.0
$ module load intel/16.0.0
intel/16.0.0(12):ERROR:150: Module 'intel/16.0.0' conflicts with the currently loaded module(s) 'intel/15.0.0'
intel/16.0.0(12):ERROR:102: Tcl command execution failed: conflict intel

Loading multiple versions at the same time would result in undefined behaviour and therefore is not possible.

Some modules may load other modules and thus may cause a conflict like:

$ module load intel/5.1.1
$ module load NAMD/2.11-mpi
intelmpi/5.1.3(12):ERROR:150: Module 'intelmpi/5.1.3' conflicts with the currently loaded module(s) 'intelmpi/5.1.1'
intelmpi/5.1.3(12):ERROR:102: Tcl command execution failed: conflict intelmpi

If your require the use of a specific version which conflicts with another modules dependency please send us an email at its-research-support@qmul.ac.uk.


Development Versions

When requesting a new version or software package you may be asked to test a development version before it is released.

To access development modules you can load the use.dev module:

$ module load use.dev
$ module avail
----------------------------------------- /share/apps/environmentmodules/dev ------------------------------------------
test/2.1.2      testing/2.3

$ module load test/2.1.2

Private modules

It is possible to have your own module files to enable you to install and manage specific versions of software without affecting existing installations.

This is mostly useful when you have very specific needs that will not be shared with other users.

Basic setup

Private modules are defined within modulefiles stored in ~/privatemodules. This directory is not created by default.

This example uses a python install done under ~/privatemodules/python/3.4.3

If your own installation is in a different directory the paths will need to be modified.

To setup a module you'll need to do the following:

# Make "privatemodules" directory in your home directory
mkdir ~/privatemodules

# using an editor (vim in this case) create a module file called "test_module"
vim ~/privatemodules/test_module

Module Files

Modulefiles handle the path adjustments required for an application to work, the full syntax is defined in the MODULEFILE(4) man page.

The basics are covered by this example modulefile for python 3.4.3:

#%Module########################################################################
#
# python 3.4.3 module file

proc ModulesHelp { } {
    puts stderr "\tAdds python 3.4.3 to your environment"
}

module-whatis "Loads Python 3.4.3"

prepend-path PATH ~/privatemodules/python/3.4.3/bin
prepend-path LD_LIBRARY_PATH ~/privatemodules/python/3.4.3/lib
prepend-path C_INCLUDE_PATH ~/privatemodules/python/3.4.3/include
prepend-path PKG_CONFIG_PATH ~/privatemodules/python/3.4.3/lib/pkgconfig
prepend-path MANPATH ~/privatemodules/python/3.4.3/share/man
prepend-path PYTHONPATH ~/privatemodules/python/3.4.3/lib/python3.4.3

Usage

Once a module is configured in a modulefile and the software has been installed you can use private modules with the following commands:

$ module load use.own
$ module load test_module

$ which python
/data/home/abc123/privatemodules/python/3.4.3/bin/python