Anaconda¶
Anaconda is a free and open source distribution of the Python and R programming languages. It contains a number of useful packages which make it popular in fields like data science, machine learning and scientific computing.
Anaconda is available as a module on Apocrita.
Anaconda versions¶
There are multiple versions of Anaconda available as modules including support for Python 2 and 3.
While some code will work in both versions there are a number of incompatibilities so the version of Python you need may depend on the code you are running. For new code bases Python 3 is strongly recommended.
Usage¶
Loading the module¶
Python distribution module file conflicts
To prevent errors when running the Python interpreter, attempting to load an additional Python distribution module after loading an Anaconda module will produce a module load conflict error.
To run the default installed version of Anaconda (Python 3), simply
load the anaconda3 module:
module load anaconda3
You can then check your python version:
$ which python
/share/apps/centos7/anaconda3/2020.02/bin/python
$ python -V
Python 3.7.6
Environment setup in anaconda3 version >= 2020.02
Recent releases of anaconda recommend use of conda to activate your
environment rather than source. This takes effect in anaconda3
module version 2020.02 and later. The documentation on this page reflects
the newer approach used by the default module version.
The variable OMP_NUM_THREADS
is set to the value of NSLOTS in serial jobs if the variable does not
already exist in the environment. You will see a confirmation message when
the module is loaded.
Conda package manager¶
In addition to the Conda package manager, Anaconda provides a number of package management tools including the familiar Python ones: pip and EasyInstall.
Environments¶
When working with Anaconda you'll probably want to update installed packages as well as installing some new ones. Since Anaconda is installed to shared storage and you don't have write access there, you'll need to create Anaconda environments somewhere where you do have write access (for example scratch directories or lab shares). You can use your home directory for this but beware: you have limited storage.
Python virtualenv environments can also be created using Anaconda because it also provides the Python programming language as well as Conda environments.
Listing environments¶
You can list existing Conda environments as follows:
$ conda env list
# conda environments:
#
base * /share/apps/centos7/anaconda3/2020.02
Let's set up a new location for environments, we do this by editing .condarc
which is in your home directory:
$ cat ~/.condarc
channels:
- defaults
ssl_verify: true
envs_dirs:
- /data/scratch/abc123/anaconda/envs
pkgs_dirs:
- /data/scratch/abc123/anaconda/pkgs
Here we have specified /data/scratch/abc123/anaconda as a location to install
to.
Creating a new environment¶
Let's create a new environment:
$ conda create --quiet --yes --name myenv
Solving environment: ...working... done
## Package Plan ##
environment location: /data/scratch/abc123/anaconda/envs/myenv
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Here we have specified that we want a new environment called "myenv". We've not requested any packages. Suppose we wanted a specific point release of python installed. We could do:
$ conda create --quiet --yes --name myenv python=3.7.0
Solving environment: ...working... done
## Package Plan ##
environment location: /data/scratch/abc123/anaconda/envs/myenv
added / updated specs:
- python=3.7
The following packages will be downloaded:
(package list redacted)
The following NEW packages will be INSTALLED:
(package list redacted)
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Let's see which version of python we have in our $PATH:
$ which python
/share/apps/centos7/anaconda3/2020.02/bin/python
$ python -V
Python 3.7.6
Let's activate the new environment. The asterisk denotes the currently active environment:
$ conda activate myenv
$ conda env list
# conda environments:
#
myenv * /data/scratch/abc123/anaconda/envs/myenv
base /share/apps/centos7/anaconda3/2020.02
And let's check python again:
$ which python
/data/scratch/abc123/anaconda/envs/myenv/bin/python
$ python -V
Python 3.7.0 :: Anaconda, Inc.
The following command will remove an unwanted environment, to save disk space:
conda env remove -n myenv
Installing packages using conda¶
You can install other packages from Conda into your environment.
Packages must be installed into activated environments
Packages must be installed into your personal environments. If package installs are attempted without first activating an environment, a permission error will be shown.
This example demonstrates installation of the scipy package:
$ conda activate myenv
(myenv) $ conda install --quiet --yes scipy
Solving environment: ...working... done
## Package Plan ##
environment location: /data/scratch/abc123/anaconda/envs/myenv
added / updated specs:
- scipy
The following NEW packages will be INSTALLED:
(package list redacted for brevity)
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
$ python -c "import scipy; print(scipy.__version__)"
1.1.0
The conda list command will show all of the packages installed into your
environment.
Example jobs¶
Threading with OpenMP
Some Python packages use OpenMP for threading. For serial jobs, the
module sets the variable OMP_NUM_THREADS to be the number of requested
slots in the current job, if the variable is unset when loading the module.
This avoids an issue where some packages incorrectly use too many threads
for OpenMP work.
If you are using OpenMP with such packages or in your own code, you should
check that the OMP_NUM_THREADS variable has been set correctly, or
override this value manually, either before or after loading the module.
Serial job (< 2020.02)¶
Here is an example job running on 1 core:
#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe smp 1
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
module load anaconda3/5.2.0
source activate myenv
python example.py
Serial job (>= 2020.02)¶
Here is an example job running on 1 core:
#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe smp 1
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
module load anaconda3
conda activate myenv
python example.py
Serial job demonstrating environment switching¶
This example shows that you can switch Conda environments in a job script:
#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe smp 1
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
module load anaconda3
conda env list
conda activate myenv
conda env list
If we view the job output:
# conda environments:
#
myenv /data/scratch/abc123/anaconda/envs/myenv
base * /share/apps/centos7/anaconda3/2020.02
# conda environments:
#
myenv * /data/scratch/abc123/anaconda/envs/myenv
base /share/apps/centos7/anaconda3/2020.02
Here we can see from the output of the conda env list commands,
that we are successfully switching environments inside a job script.