JupyterLab

JupyterLab is the latest web-based interactive development environment for notebooks, code, and data. Its flexible interface allows users to configure and arrange workflows in data science, scientific computing, computational journalism, and machine learning. A modular design invites extensions to expand and enrich functionality.

Users may start using JupyterLab using the HPC-one web portal.


JupyterLab Tips & Tricks

Convert a Jupyter Notebook to python code for production run with batch job

Interactive apps are primarily for visualization code development and hence up to 24 hours of wall time is allowed. For production run of developed codes, it is recommended to submit them as batch jobs to SLURM scheduler. Below is an example on how to convert a jupyter notebook into native python codes for batch executions.

# Convert a jupyter notebook into python code (.ipynb becomes .py)
$ jupyter nbconvert --to python cell2location_cpu.ipynb

#Put the following in a SLURM script and submit to SLURM to run as batch job
ml miniconda/py39/4.10.3
conda activate /share1/cell2location/0.0.6
python cell2location_cpu.py

Create custom kernel for Jupyter

If you would like to use your own set of python modules, you may create a conda environment and prepare a corresponding kernel for Jupyter.
For example, if you would like to use an alternative python version and pandas in Jupyter, you may run the following.

# load the miniconda environment module
ml miniconda/py39/4.10.3

# create a conda environment called “myproject”
conda create -n myproject

# activate the "myproject" conda environment and install python version 3.9.7 followed by pandas
conda activate myproject
conda install -c conda-forge python=3.9.7
conda install -c anaconda pandas

# install python kernel for Jupyter
# avoid installation of any jupyter related modules
pip install --user ipykernel

# added the kernel to Jupyter and name it as "Environment(myproject)"
python -m ipykernel install --user --name=myproject --display-name='Environment(myproject)'

Then, you may open a new notebook in JupyterLab, click the Kernel chooser in top right corner and choose “Environment(myproject)” in the Kernel menu.


Access files under /lustre1 in Jupyter

For the sake of security, JupyterLab only allows access to items under the folder where it starts (i.e ~/ ).

You may run the following commands to create symbolic links (aka shortcuts) under ~/ to point to the respective folders in Lustre.

PI_GROUP=$(groups | tr " " "\n" | grep _)
ln -s /lustre1/g/${PI_GROUP}/ ~/lustre_g_${PI_GROUP}
ln -s /lustre1/u/${USER}/ ~/lustre_u_${USER}

Take a user with login name “tmchan” under the PI Group “fin_davis” as example,  he may access ~/lustre1_u_tmchan and ~/lustre1_g_fin_davis in JupyterLab to access the folders at /lustre1/u/tmchan and /lustre1/g/fin_davis respectively.


Monitor resource usage with GPU Dashboard in Jupyter

A GPU dashboard is available for JupyterLab for showing the resource usage of GPU in real time, side by side with cells in your notebook. It is particularly useful in identifying any bottleneck or issues during code development.

You may activate your conda or python virtual environment for your kernel and run the following once to install the GPU dashboard.

pip install jupyterlab-nvdashboard

References:

  1. Jupyter Documentation
  2. IPython kernel
  3. NVIDIA GPU dashboard for Jupyter