Miniconda¶
Miniconda is a free open source minimal installer for Conda. It is a small,
bootstrap version of Anaconda that includes only Conda, Python,
the packages they depend on, and a small number of other useful packages,
including pip, zlib and a few others.
Miniconda is available as a module on Apocrita.
Miniconda contains support for the Mamba package manager
Miniconda versions on Apocrita contain the fast cross-platform
package manager Mamba. We strongly
recommend replacing conda with mamba when installing packages, to
significantly reduce the time taken for the package solver to run.
Examples below show use of the mamba command when installing packages.
Usage¶
Python distribution module file conflicts
To prevent errors when running the Python interpreter, attempting to load an additional Python distribution module after loading a Miniconda module will produce a module load conflict error.
To run the default installed version of Miniconda (Python 3), simply
load the miniconda module:
module load miniconda
You can then check your Python version:
$ which python
/share/apps/centos7/miniconda/X.Y.Z/bin/python
$ python -V
Python 3.X.Y
Environment setup
The variable OMP_NUM_THREADS
is set to the value of NSLOTS in serial jobs if the variable does not
already exist in the environment. You will see a confirmation message when
the module is loaded.
Conda package manager¶
In addition to the Conda package manager, Miniconda provides a number of package management tools including the familiar Python ones: pip and EasyInstall.
Environments¶
When working with Miniconda you'll probably want to update installed packages as well as installing some new ones. Since Miniconda is installed to shared storage and you don't have write access there, you'll need to create Miniconda environments somewhere where you do have write access (for example scratch directories or lab shares). You can use your home directory for this but beware: you have limited storage.
Python virtualenv environments can also be created using Miniconda because it also provides the Python programming language as well as Conda environments.
Listing environments¶
You can list existing Conda environments as follows:
$ conda env list
# conda environments:
#
base * /share/apps/centos7/miniconda/X.Y.Z
Let's set up a new location for environments, we do this by editing .condarc
which is in your home directory:
$ cat ~/.condarc
channels:
- defaults
ssl_verify: true
envs_dirs:
- /data/scratch/abc123/miniconda/envs
pkgs_dirs:
- /data/scratch/abc123/miniconda/pkgs
Here we have specified /data/scratch/abc123/miniconda as a location to
install to.
Creating a new environment¶
Let's create a new environment:
$ conda create --quiet --yes --name myenv
Solving environment: ...working... done
## Package Plan ##
environment location: /data/scratch/abc123/miniconda/envs/myenv
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Here we have specified that we want a new environment called "myenv". We've not requested any packages. Suppose we wanted a specific point release of python installed. We could do:
$ mamba create --quiet --yes --name myenv python=3
Package Version Build Channel Size
──────────────────────────────────────────────────────────────────────────────────
Install:
──────────────────────────────────────────────────────────────────────────────────
(package list redacted)
Summary:
Install: XX packages
Total download: XX MB
──────────────────────────────────────────────────────────────────────────────────
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Let's see which version of Python we have in our $PATH:
$ which python
/share/apps/centos7/miniconda/X.Y.Z/bin/python
$ python -V
Python 3.X.Y
Let's activate the new environment (the asterisk denotes the currently active environment):
$ conda activate myenv
$ conda env list
# conda environments:
#
myenv * /data/scratch/abc123/miniconda/envs/myenv
base /share/apps/centos7/miniconda/X.Y.Z
And let's check python again:
$ which python
/data/scratch/abc123/miniconda/envs/myenv/bin/python
$ python -V
Python 3.X.Y
The following command will remove an unwanted environment, to save disk space:
conda env remove -n myenv
Installing packages using Conda¶
You can install other packages from Conda into your environment.
Packages must be installed into activated environments
Packages must be installed into your personal environments. If package installs are attempted without first activating an environment, a permission error will be shown.
This example demonstrates installation of the scipy package, using mamba:
$ conda activate myenv
(myenv) $ mamba install --quiet --yes scipy
Package Version Build Channel Size
───────────────────────────────────────────────────────────────────────────────
Install:
───────────────────────────────────────────────────────────────────────────────
(package list redacted)
Summary:
Install: XX packages
Total download: XX MB
───────────────────────────────────────────────────────────────────────────────
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
$ python -c "import scipy; print(scipy.__version__)"
1.X.Y
The conda list command will show all of the packages installed into your
environment.
Example jobs¶
Threading with OpenMP
Some Python packages use OpenMP for threading. For serial jobs, the
module sets the variable OMP_NUM_THREADS to be the number of requested
slots in the current job, if the variable is unset when loading the module.
This avoids an issue where some packages incorrectly use too many threads
for OpenMP work.
If you are using OpenMP with such packages or in your own code, you should
check that the OMP_NUM_THREADS variable has been set correctly, or
override this value manually, either before or after loading the module.
Serial job¶
Here is an example job running on 1 core:
#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe smp 1
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
module load miniconda
conda activate myenv
python example.py
Serial job demonstrating environment switching¶
This example shows that you can switch Conda environments in a job script:
#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe smp 1
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
module load miniconda
conda env list
conda activate myenv
conda env list
If we view the job output:
# conda environments:
#
myenv /data/scratch/abc123/miniconda/envs/myenv
base * /share/apps/centos7/miniconda/X.Y.Z
# conda environments:
#
myenv * /data/scratch/abc123/miniconda/envs/myenv
base /share/apps/centos7/miniconda/X.Y.Z
Here we can see from the output of the conda env list commands,
that we are successfully switching environments inside a job script.