Using Jupyter via OnDemand

Jupyter is an interactive web-based environment for creating and using Jupyter notebooks, documents containing live code and visualisations. While many languages are supported by Jupyter, Python notebooks are very commonly used. Please refer to the overview section for instructions on how to login to OnDemand.

Starting a Jupyter session

If your code does not specifically use GPU acceleration and/or you do not have access to GPU-enabled nodes, then choose "Jupyter Notebook CPU" from the Interactive Apps list.

Here you can choose the number of cores and quantity of RAM required.

Request 1 core for most Jupyter jobs

Many Jupyter notebooks will only use 1 core, therefore any additional cores requested will be allocated but not utilised.

Choosing a 1 hour maximum running time is the best option for getting a session quickly, unless you have access to owned nodes which may also provide sessions immediately for up to 240hrs if resources are available.

Once clicking Launch, the request will be queued, and when resources have been allocated, you will be presented with the option to open the session.

Here you can see the compute host that the session has been scheduled on, and time remaining for the session.

Clicking Connect to Jupyter will open the familiar Jupyter session window, from which you can open notebooks.

And in action:

Using environments

The Anaconda suite, which we are using, provides a lot of popular Python packages as standard. However, if you need you use modules outside of the default list provided by Anaconda, you will need to switch environments from within the Jupyter notebook.

We will demonstrate this by attempting to use the emoji module to display emojis. A default Anaconda environment does not contain this package.

Listing installed packages

You can inspect the list of packages provided by the default Anaconda installation from within a terminal session on Apocrita by loading the Anaconda module: module load anaconda3 followed by the conda list command

In a Jupyter notebook, attempting to load the emoji module will fail.

We will need to create a new conda environment containing our desired packages. Installing the emoji package into an activated environment will allow us to use it on the command line interface, but to also make it visible to Jupyter we need the ipykernel package, and a command to activate it.

We can perform this on Apocrita as follows:

module load anaconda3

# use -y to accept the y/n question, -q to reduce verbosity
conda create -n emoji_env -y -q
conda activate emoji_env

# add the desired packages to the env
# we need to specify the conda-forge channel to install the emoji package
conda install -c conda-forge emoji ipykernel -y -q

# make the env visible to Jupyter
python -m ipykernel install --user --name emoji_env

# deactivate your env
conda deactivate

Note that the ipykernel package and subsequent command are only required to make the environment visible to Jupyter.

If you already have an existing conda environment myenv but you wish to make it visible to Jupyter, simply activate your environment:

conda activate myenv

and run:

conda install ipykernel -y -q
python -m ipykernel install --user --name myenv

Now if you open a new Jupyter session from OnDemand, you will notice that the new environment is available under the Kernel -> Change Kernel menu, with the name we provided with the --name option. Similarly, it will be available as an option under the File -> New notebook menu in addition to the default Anaconda Python 3 environment.

Now if you open a new notebook, you can now use the emoji_env we created. Note that the currently active environment is displayed in the top right corner.

Using this method, if you have notebooks requiring packages in addition to the defaults, you can now maintain different environments to support use of those notebooks.

Installing the R kernel

do not use install.packages inside an R kernel

Due to the way the R kernel works within Anaconda, attempting to use the install.packages method inside an R kernel will install R packages outside of the Anaconda environment, which may cause unforeseen conflicts when running the native R application, via a module.

To install the R kernel, run the following commands as a one-off setup:

qlogin -l h_vmem=10G
module load anaconda3
conda create -n R -y -q
conda activate R
conda install r-recommended r-irkernel -y
R -e 'IRkernel::installspec()'

The R kernel will now be available as an option within Jupyter Notebook when starting a new notebook, or under the Kernel -> Change kernel menu:

To uninstall the R kernel, simply run conda env remove -n R (or delete the environment install directory) and also delete the kernel specification directory with rm -r ~/.local/share/jupyter/kernels/ir/.