Skip to content

Jupyter usage

Slurm

Word of advice

Jupyter notebooks are a great way to prototype or setup your more complex Python or even Tensorflow workflows. However, due to its interactive nature it is not suitable for running long calculations or train AI models in a JupyterHub/Lab session. Lucky for you, you can easily submit your Jupyter notebooks to Slurm.

Let us assume you finished your Tensorflow workflow for example as Jupyter notebook and you tested it for functionality. Now you are ready to submit the production run to the Slurm queue.

#!/bin/bash

#SBATCH --time=10:00:00
#SBATCH --mem=40G
#SBATCH --ntasks=1
#SBATCH --job-name=Tensorflow-training
#SBATCH --partition=clara
#SBATCH --gres=gpu:v100:1

module load JupyterLab/1.2.5-fosscuda-2019b-Python-3.7.4
module load TensorFlow/2.4.0-fosscuda-2019b-Python-3.7.4
jupyter nbconvert --to notebook --execute Tensorflow.ipynb --output tf-job-%j.ipynb

How to use your own environments and kernels

Following here are two quick methods (using conda and using pip+venv) you can use to set up your own individual environments.

Using conda

  1. Create a conda environment

    # load anaconda module
module load Anaconda3
# ensure conda initialisation
conda init bash
# create conda environment, e.g. containing tensorflow package
conda create -n tf2.1 tensorflow=2.1

  2. Install the ipykernel

    # activate environment
conda activate tf2.1
# install the ipykernel package for kernel management 
conda install ipykernel
# install kernel
python -m ipykernel install --user --name 'tf2.1' --display-name "Tensorflow 2.1"

  3. Log out of JupyterHub and back in again, maybe you need to restart your session

  4. Enjoy the new environment in JupyterHub

Using pip + venv

  1. Create a virtual python environment

    # activate up to date python
module load Python
# create a directory
mkdir venv
# create an environment within this directory
python -m venv venv/tf
# activate environment
source venv/tf/bin/activate
# install packages as needed, e.g. tensorflow
pip install tensorflow

  2. Install the ipykernel

    # activate environment, maybe duplicate
source venv/tf/bin/activate
# install the ipykernel package for kernel management
pip install ipykernel
# install kernel
python -m ipykernel install --user --name 'tf' --display-name "Tensorflow (user)"

  3. Log out of JupyterHub and back in again, maybe you need to restart your session

  4. Enjoy the new environment in JupyterHub

Using modules in a custom Jupyter environment

We provide a collection of commonly used software. This also includes packages that can be used with Jupyter and Python.

This tutorial takes you through the steps to set up your own version-specific virtual Python environment to use in Jupyter notebooks and the available software modules for that version.

Setup Python 3.7 environment

Connect to one of the login nodes, e.g.

ssh <sc-user>@login01.sc.uni-leipzig.de

Next, you need to load the Python module in the specific version you need. In this case it is version 3.7

module load Python/3.7

If this is your fist time setting up a virtual environment, you need to create a directory to store your environments in. The following command will create a directory named venv in your home directory.

mkdir -p ~/venv

Now you are ready to create your virtual environment. To do that issue the following command:

python -m venv ~/venv/py37

This will create a subdirectory py37 in the aboved created directory containing the basic files and links.
To use the newly created virtual environment it needs to be activated.

source ~/venv/py37/bin/activate

To check if everything works as expected check if you get similar results using these commands.

$ which python
~/venv/py37/bin/python
$ python --version
Python 3.7.4

Add virtual environment to Jupyter

To make Jupyter aware of your newly created virtual environment you need to install ipykernel and add the kernel to Jupyter.
Using pip you can easily install the required package and all its dependencies.

pip install ipykernel

Using the following command you create a new button in your JupyterLab Launcher named Python 3.7 (user).

python -m ipykernel install --user --name 'py37' --display-name "Python 3.7 (user)"

Using new environment in Jupyter

Now you can use your browser and login to JupterHub on https://lab.sc.uni-leipzig.de

Remember

Load the same modules you used to create your virtual environment, before opening a new notebook.

You can now click on the blue hexagon button in the sidebar (marked 1. Load software modules below) to open the software selection tool.

Here you can load the modules you need in your notebook.

Jupyter Launcher

NOTICE

Only Python modules with the same major and minor version of your environment will work. In this example you can load modules with Python 3.7.* in their name.

In your launcher you should also find a new button with the name you have chosen in the previous step. In the image above, marked with 2. Open new environment.

Double clicking this new button opens a notebook in this new environment.

Example PyTorch

If you want to use PyTorch in your notebook. First search for PyTorch in the software list.

Jupyter load modules

Then find the version you want to use with the appropriate Python version in the list of available modules and click load.

The modules and all needed dependencies are then loaded. They move up to the loaded modules section and are ready to be used in a new notebook.

Jupyter loaded modules

Julia on lab.sc.uni-leipzig.de

  • From the launcher, open a terminal and load the desired julia version
module load Julia/$VERSION
  • you can press TAB for autocompletion after module load Julia/ to see the available versions.
  • $VERSION is variable and refers to the version of Julia.
  • Now you can run Julia from the terminal 
    julia
  • In Julia, install IJulia in the base environment
using Pkg
Pkg.add("IJulia")
  • You should only have few packages installed in the base environment, e.g. IJulia, debuggers, benchmark tools, etc.
  • Install packages for analyses in separate environments.
  • Add a multithreaded kernel (change the number of threads to what you reserved)
using IJulia
installkernel("Julia 4c", env = Dict("JULIA_NUM_THREADS" => "4"))
  • You can have several kernel with different numbers of threads installed at the same time.
  • Wait 2 minutes.
  • Now you can choose the kernel Julia 4c $VERSION from the launcher.
  • For your analysis create a specific environment and load this at the beginning of your notebook:
  • Open a notebook with the Julia kernel and in the first cell execute
] activate --shared myanalysis
  • Now you can use all packages installed in the environment myanalysis.

NOTICE

Please read the documentation for Pkg.jl for details.

WebIO hinzufügen

Some packages, like WebIO, need an extension to Jupyter to work properly. How these kind of packages can be used, is explained here, using WebIO as an example.

Login via SSH to login01.sc.uni-leipzig.de or open a new "Terminal" in an already running JupyterLab session.
Check if pip from Anaconda is working. (It should if you followed the instructions above).
Type which pip and check whether the output looks like this:

$ which pip
/software/all/Anaconda3/2020.02/bin/pip

Now, you can just install the WebIO extension to Jupyter:

pip install webio_jupyter_extension --user

Very important is the --user parameter. This command installs the extension in your home directory. The extension is available after you reload or after your next login to Jupyter.

Now, you can install and use WebIO in Julia notebooks in Jupyter:

using Pkg
Pkg.add("WebIO")
using WebIO

Intalling the package via Pkg.add("WebIO") is only needed once, of course.