Skip to content
Snippets Groups Projects
Commit df74dca3 authored by Jens Henrik Goebbert's avatar Jens Henrik Goebbert
Browse files

update kernel-create

parent 2f6f2383
Branches
No related tags found
No related merge requests found
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
![jsc-logo.jpg](attachment:67258d94-84e6-4a0c-ae8f-c74332ec082e.jpg) ![jsc-logo.jpg](attachment:67258d94-84e6-4a0c-ae8f-c74332ec082e.jpg)
Author: [Jens Henrik Göbbert](mailto:j.goebbert@fz-juelich.de) Author: [Jens Henrik Göbbert](mailto:j.goebbert@fz-juelich.de)
------------------------------------ ------------------------------------
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Create your own Jupyter Kernel # Create your own Jupyter Kernel
Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed. Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed.
With your own kernel you can overcome that problem easily and define your own environment, in which you work. With your own kernel you can overcome that problem easily and define your own environment, in which you work.
This notebook shows you how you can build your own kernel for a **python environment**. This notebook shows you how you can build your own kernel for a **python environment**.
<div class="alert alert-block alert-info">
<b>Attention:</b>
This notebook is meant to run out of a JupyterLab on JSC's HPC systems.</br>
</div>
------------------------- -------------------------
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Building your own Jupyter kernel is a three step process ## Building your own Jupyter kernel is a three step process
1. Create/Pimp new virtual Python environment 1. Create/Pimp new virtual Python environment
* venv * venv
2. Create/Edit launch script for the Jupyter kernel 2. Create/Edit launch script for the Jupyter kernel
* kernel.sh * kernel.sh
3. Create/Edit Jupyter kernel configuration 3. Create/Edit Jupyter kernel configuration
* kernel.json * kernel.json
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
### Settings ### Settings
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* Set kernel name #### Set kernel name
- must be lower case - must be lower case
- change if you like - change if you like
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
# INPUT NEEDED: # INPUT NEEDED:
KERNEL_NAME=$(whoami)_kernel_ecar2023 KERNEL_NAME=$(whoami)_kernel
export KERNEL_NAME=$(echo "${KERNEL_NAME}" | awk '{print tolower($0)}') export KERNEL_NAME=$(echo "${KERNEL_NAME}" | awk '{print tolower($0)}')
echo ${KERNEL_NAME} # double check echo ${KERNEL_NAME} # double check
``` ```
%% Output
jovyan_kernel_ecar2023
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* List directories where JupyterLab will search for kernels #### List directories where JupyterLab will search for kernels
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
# JUPYTER SEARCH PATH (for kernels-directory) # JUPYTER SEARCH PATH (for kernels-directory)
echo "jupyter search paths for kernels-directories" echo "jupyter search paths for kernel-directories"
if [ -z $JUPYTER_PATH ]; then if [ -z $JUPYTER_PATH ]; then
echo "$HOME/.local/share/jupyter" echo "$HOME/.local/share/jupyter"
else else
tr ':' '\n' <<< "$JUPYTER_PATH" tr ':' '\n' <<< "$JUPYTER_PATH"
fi fi
``` ```
%% Output
jupyter search paths for kernels-directories
/home/jovyan/.local/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Ruby/3.0.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-R/4.1.2-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-PyVisualization/1.0-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-PyQuantum/3.0-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-PyDeepLearning/1.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Octave/6.4.0-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Julia/1.7.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-JavaScript/5.2.1-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Cling/0.9-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/JupyterKernel-Bash/0.7.2-gcccoremkl-11.2.0-2021.4.0-2022.3.4/share/jupyter
/p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/share/jupyter
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
<div class="alert alert-block alert-info"> #### Set the kernel directory
<b>Attention:</b> - check that the kernel name is unique
Please choose 'private kernel' if you are unsure.</br> - print the location of the new kernel
Using 'project kernel's need to be enabled for your project first by our Jupyter-JSC admins.
</div>
* Set kernel type
- private kernel = "\${HOME}/.local/"
- project kernel = "\${PROJECT}/.local/"
- other kernel = "\<your-path\>" (ensure it is part of $JUPYTER_PATH or your kernel will not be found by JuypterLab)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
# INPUT NEEDED: # define KERNEL_SPECS_DIR
export KERNEL_TYPE=private # private, project or other
export KERNEL_SPECS_PREFIX=${HOME}/.local export KERNEL_SPECS_PREFIX=${HOME}/.local
if [ ! -d "$KERNEL_SPECS_PREFIX" ]; then
################### echo "ERROR: please create directory $KERNEL_SPECS_PREFIX"
# project kernel
if [ "${KERNEL_TYPE}" == "project" ]; then
export KERNEL_SPECS_PREFIX=${PROJECT}/.local
echo "project kernel"
# private kernel
elif [ "${KERNEL_TYPE}" == "private" ]; then
export KERNEL_SPECS_PREFIX=${HOME}/.local
echo "private kernel"
else
if [ ! -d "$KERNEL_SPECS_PREFIX" ]; then
echo "ERROR: please create directory $KERNEL_SPECS_PREFIX"
fi
echo "other kernel"
fi fi
export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels
# check if kernel name is unique # check if kernel name is unique
if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then
echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}" echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}"
echo " Rename kernel name or remove directory." echo " Rename kernel name or remove directory."
fi fi
echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME} # double check # print the location of the new kernel
echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME}
``` ```
%% Output
private kernel
/home/jovyan/.local/share/jupyter/kernels/jovyan_kernel_ecar2023
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* Set directory for kernels virtual environment #### Set the kernel's virtual environment
- change if you like - by default it is located at $PROJECT
- print the location of the new kernels virtual environment
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
# INPUT NEEDED: # define KERNEL_VENVS_DIR
export KERNEL_VENVS_DIR=${HOME}/jupyter_kernel_venvs export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels
###################
mkdir -p ${KERNEL_VENVS_DIR} mkdir -p ${KERNEL_VENVS_DIR}
if [ "${KERNEL_TYPE}" != "private" ] && [ "${KERNEL_TYPE}" != "other" ]; then
echo "Please check the permissions and ensure your project partners have read/execute permissions:"
namei -l ${KERNEL_VENVS_DIR}
fi
echo ${KERNEL_VENVS_DIR} # double check # print the location of the new kernels virtual environment
ls -lt ${KERNEL_VENVS_DIR} echo ${KERNEL_VENVS_DIR}
``` ```
%% Output
/home/jovyan/jupyter_kernel_venvs
total 0
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## 1. Create/Pimp new virtual Python environment ## 1. Create/Pimp new virtual Python environment
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 1.1 - Load basic Python module #### 1.1 - Load basic Python module
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
module purge module -q purge
module load Stages/2022 # any stage can be used module -q load Stages/2024 # any stage can be used
module -q load GCC
module -q load Python # only Python is required
# module list # double check
```
module load GCC/11.2.0 %% Cell type:code id: tags:
module load Python/3.9.6 # only Python is required
module load ParaStationMPI ``` bash
# get Python version
module list # double check export PYV=$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')
``` echo $PYV
```
%% Output
The following modules were not unloaded:
(Use "module --force purge" to unload all):
1) Stages/2022
Currently Loaded Modules:
1) Stages/2022 (S) 16) libxslt/.1.1.34 (H)
2) GCCcore/.11.2.0 (H) 17) libyaml/.0.2.5 (H)
3) zlib/.1.2.11 (H) 18) PostgreSQL/13.4
4) binutils/.2.37 (H) 19) gflags/.2.2.2 (H)
5) GCC/11.2.0 20) libspatialindex/.1.9.3 (H)
6) bzip2/.1.0.8 (H) 21) NASM/.2.15.05 (H)
7) ncurses/.6.2 (H) 22) libjpeg-turbo/.2.1.1 (H)
8) libreadline/.8.1 (H) 23) Python/3.9.6
9) Tcl/8.6.11 24) numactl/2.0.14
10) SQLite/.3.36 (H) 25) nvidia-driver/.default (H,g)
11) XZ/.5.2.5 (H) 26) CUDA/11.5 (g)
12) GMP/6.2.1 27) UCX/default (g)
13) libffi/.3.4.2 (H) 28) pscom/.5.4-default (H)
14) OpenSSL/1.1 29) mpi-settings/UCX
15) libxml2/.2.9.10 (H) 30) ParaStationMPI/5.5.0-1 (g)
Where:
H: Hidden Module
S: Module is Sticky, requires --force to unload or purge
g: built for GPU
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 1.2 - Load extra modules you need for your kernel #### 1.2 - Load extra modules you need for your kernel
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
# module load <module you need> # module load <module you need>
module load TensorFlow module load OpenCV # example
module load OpenCV
module load Jupyter
module list # module list # double check
``` ```
%% Output
Currently Loaded Modules:
1) Stages/2022 (S) 70) MPFR/4.1.0
2) GCCcore/.11.2.0 (H) 71) MPC/1.2.1
3) zlib/.1.2.11 (H) 72) gmpy2/2.1.0b5
4) binutils/.2.37 (H) 73) sympy/1.8
5) GCC/11.2.0 74) xarray/0.20.1
6) bzip2/.1.0.8 (H) 75) SciPy-Stack/2021b
7) ncurses/.6.2 (H) 76) x264/.20210613 (H)
8) libreadline/.8.1 (H) 77) LAME/.3.100 (H)
9) Tcl/8.6.11 78) x265/.3.4 (H)
10) SQLite/.3.36 (H) 79) libvpx/1.11.0
11) XZ/.5.2.5 (H) 80) FriBidi/1.0.10
12) GMP/6.2.1 81) FFmpeg/.4.4.1 (H,g)
13) libffi/.3.4.2 (H) 82) nettle/.3.7.3 (H)
14) OpenSSL/1.1 83) libdrm/.2.4.108 (H)
15) libxml2/.2.9.10 (H) 84) LLVM/13.0.0
16) libxslt/.1.1.34 (H) 85) libunwind/.1.5.0 (H)
17) libyaml/.0.2.5 (H) 86) OpenGL/2021b (g)
18) PostgreSQL/13.4 87) freeglut/.3.2.1 (H)
19) gflags/.2.2.2 (H) 88) JasPer/.2.0.33 (H)
20) libspatialindex/.1.9.3 (H) 89) gettext/.0.21 (H)
21) NASM/.2.15.05 (H) 90) PCRE/.8.45 (H)
22) libjpeg-turbo/.2.1.1 (H) 91) GLib/.2.69.1 (H)
23) Python/3.9.6 92) ATK/.2.36.0 (H)
24) numactl/2.0.14 93) DBus/.1.13.18 (H)
25) nvidia-driver/.default (H,g) 94) DB/.18.1.40 (H)
26) CUDA/11.5 (g) 95) Perl/5.34.0
27) UCX/default (g) 96) intltool/.0.51.0 (H)
28) pscom/.5.4-default (H) 97) AT-SPI2-core/.2.40.3 (H)
29) mpi-settings/UCX 98) AT-SPI2-ATK/.2.38.0 (H)
30) ParaStationMPI/5.5.0-1 (g) 99) Gdk-Pixbuf/.2.42.6 (H)
31) imkl/.2021.4.0 (H) 100) pixman/.0.40.0 (H)
32) cuDNN/8.3.1.22-CUDA-11.5 (g) 101) cairo/.1.16.0 (H)
33) NCCL/2.12.7-1-CUDA-11.5 (g) 102) HarfBuzz/.2.8.2 (H)
34) pybind11/.2.7.1 (H) 103) Pango/.1.48.8 (H)
35) SciPy-bundle/2021.10 104) libepoxy/.1.5.9 (H)
36) Szip/.2.1.1 (H) 105) libcroco/.0.6.13 (H)
37) HDF5/1.12.1-serial 106) Rust/1.54.0
38) h5py/3.5.0-serial 107) librsvg/.2.51.2 (H)
39) cURL/7.78.0 108) GTK+/.3.24.23 (H)
40) double-conversion/3.1.6 109) Eigen/3.3.9
41) flatbuffers/.2.0.0 (H) 110) OpenEXR/3.1.1
42) giflib/.5.2.1 (H) 111) OpenCV/4.5.4
43) libpciaccess/.0.16 (H) 112) Pandoc/2.16.1
44) hwloc/2.5.0 (g) 113) texlive/20200406
45) ICU/.70.1 (H) 114) SWIG/.4.0.2 (H)
46) JsonCpp/.1.9.4 (H) 115) FFTW/.3.3.10-nompi (H)
47) LMDB/.0.9.29 (H) 116) tbb/2020.3
48) nsync/.1.24.0 (H) 117) PCRE2/10.37
49) protobuf/.3.17.3 (H) 118) libevent/.2.1.12 (H)
50) protobuf-python/.3.17.3 (H) 119) NSPR/4.32
51) flatbuffers-python/2.0 120) NSS/3.73
52) typing-extensions/3.10.0.0 121) Qt5/.5.15.2 (H)
53) libpng/.1.6.37 (H) 122) ITK/5.2.1-nompi
54) snappy/.1.1.9 (H) 123) netCDF/4.8.1-serial
55) TensorFlow/2.6.0-CUDA-11.5 (g) 124) netcdf4-python/1.5.7-serial
56) Brotli/1.0.9 125) git/2.33.1-nodocs
57) freetype/.2.11.0 (H) 126) scikit-build/0.11.1
58) expat/.2.4.1 (H) 127) scikit-learn/1.0.1
59) util-linux/.2.37 (H) 128) PyYAML/5.4.1
60) fontconfig/.2.13.94 (H) 129) bokeh/2.4.1
61) xorg-macros/.1.19.3 (H) 130) dask/2021.9.1
62) X11/20210802 131) scikit-image/0.18.3
63) Tk/.8.6.11 (H) 132) numba/0.55.1 (g)
64) Tkinter/3.9.6 133) GEOS/3.9.1
65) LibTIFF/.4.3.0 (H) 134) Shapely/1.8.0
66) Pillow-SIMD/9.0.1 135) nodejs/.16.13.0 (H)
67) Qhull/.2020.2 (H) 136) yarn/1.22.17
68) matplotlib/3.4.3 137) Jupyter/2022.3.4
69) Seaborn/0.11.2
Where:
H: Hidden Module
S: Module is Sticky, requires --force to unload or purge
g: built for GPU
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 1.3 - Create and activate a virtual environment for the kernel #### 1.3 - Create and activate a virtual environment for the kernel
and ensure python packages installed in the virtual environment are always prefered and ensure python packages installed in the virtual environment are always prefered
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
if [ -d "${KERNEL_VENVS_DIR}/${KERNEL_NAME}" ]; then export VIRTUAL_ENV=${KERNEL_VENVS_DIR}/${KERNEL_NAME}
echo "ERROR: Directory for virtual environment already ${KERNEL_VENVS_DIR}/${KERNEL_NAME}" if [ -d "${VIRTUAL_ENV}" ]; then
echo "ERROR: Directory for virtual environment already ${VIRTUAL_ENV}"
echo " Rename kernel name or remove directory." echo " Rename kernel name or remove directory."
else else
python -m venv --system-site-packages ${KERNEL_VENVS_DIR}/${KERNEL_NAME} python -m venv --system-site-packages ${VIRTUAL_ENV}
source ${KERNEL_VENVS_DIR}/${KERNEL_NAME}/bin/activate source ${VIRTUAL_ENV}/bin/activate
export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.9/site-packages:${PYTHONPATH} export PYTHONPATH=${VIRTUAL_ENV}/lib/python${PYV}/site-packages:${PYTHONPATH}
echo ${VIRTUAL_ENV} # double check echo ${VIRTUAL_ENV} # double check
fi fi
``` ```
%% Output
/home/jovyan/jupyter_kernel_venvs/jovyan_kernel_ecar2023
(jovyan_kernel_ecar2023)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 1.4 - Install Python libraries required for communication with Jupyter #### 1.4 - Install Python libraries required for communication with Jupyter
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
which pip which pip
if [ -z "${VIRTUAL_ENV}" ]; then if [ -z "${VIRTUAL_ENV}" ]; then
echo "ERROR: Virtual environment not successfully initialized." echo "ERROR: Virtual environment not successfully initialized."
else else
pip install ipykernel pip install ipykernel
ls ${VIRTUAL_ENV}/lib/python3.9/site-packages/ # double check ls ${VIRTUAL_ENV}/lib/python${PYV}/site-packages/ # double check
fi fi
``` ```
%% Output
/home/jovyan/jupyter_kernel_venvs/jovyan_kernel_ecar2023/bin/pip
(jovyan_kernel_ecar2023) Requirement already satisfied: ipykernel in /p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/lib/python3.9/site-packages (6.13.0)
Requirement already satisfied: packaging in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipykernel) (20.9)
Requirement already satisfied: psutil in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipykernel) (5.8.0)
Requirement already satisfied: nest-asyncio in /p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/lib/python3.9/site-packages (from ipykernel) (1.5.5)
Requirement already satisfied: tornado>=6.1 in /p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/lib/python3.9/site-packages (from ipykernel) (6.2)
Requirement already satisfied: matplotlib-inline>=0.1 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipykernel) (0.1.3)
Requirement already satisfied: ipython>=7.23.1 in /p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/lib/python3.9/site-packages (from ipykernel) (7.34.0)
Requirement already satisfied: jupyter-client>=6.1.12 in /p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/lib/python3.9/site-packages (from ipykernel) (7.1.2)
Requirement already satisfied: traitlets>=5.1.0 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipykernel) (5.1.0)
Requirement already satisfied: debugpy>=1.0 in /p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/lib/python3.9/site-packages (from ipykernel) (1.6.0)
Requirement already satisfied: pickleshare in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel) (0.7.5)
Requirement already satisfied: setuptools>=18.5 in /home/jovyan/jupyter_kernel_venvs/jovyan_kernel_ecar2023/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel) (56.0.0)
Requirement already satisfied: pygments in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel) (2.9.0)
Requirement already satisfied: jedi>=0.16 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel) (0.18.0)
Requirement already satisfied: backcall in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel) (0.2.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel) (3.0.20)
Requirement already satisfied: decorator in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel) (5.0.9)
Requirement already satisfied: pexpect>4.3 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel) (4.8.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel) (0.8.2)
Requirement already satisfied: python-dateutil>=2.1 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel) (2.8.2)
Requirement already satisfied: pyzmq>=13 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel) (22.3.0)
Requirement already satisfied: entrypoints in /p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel) (0.4)
Requirement already satisfied: jupyter-core>=4.6.0 in /p/software/hdfcloud/stages/2022/software/Jupyter/2022.3.4-gcccoremkl-11.2.0-2021.4.0/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel) (4.9.1)
Requirement already satisfied: ptyprocess>=0.5 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from pexpect>4.3->ipython>=7.23.1->ipykernel) (0.7.0)
Requirement already satisfied: wcwidth in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=7.23.1->ipykernel) (0.2.5)
Requirement already satisfied: six>=1.5 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from python-dateutil>=2.1->jupyter-client>=6.1.12->ipykernel) (1.16.0)
Requirement already satisfied: pyparsing>=2.0.2 in /p/software/hdfcloud/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages (from packaging->ipykernel) (2.4.7)
_distutils_hack pip-21.1.3.dist-info setuptools-56.0.0.dist-info
distutils-precedence.pth pkg_resources
pip setuptools
(jovyan_kernel_ecar2023)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 1.5 - Install whatever else you need in your Python virtual environment (using pip) * 1.5 - Install whatever else you need in your Python virtual environment (using pip)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
#pip install <python-package you need> #pip install <python-package you need>
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## 2. Create/Edit launch script for the Jupyter kernel ## 2. Create/Edit launch script for the Jupyter kernel
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 2.1 - Create launch script, which loads your Python virtual environment and starts the ipykernel process inside: * 2.1 - Create launch script, which loads your Python virtual environment and starts the ipykernel process inside:
<div class="alert alert-block alert-info"> <div class="alert alert-block alert-info">
<b>Attention:</b> <b>Attention:</b>
You MUST load the exactly the same modules as you did above for your virtual Python environment. You MUST load the exactly the same modules as you did above for your virtual Python environment.
</div> </div>
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
echo '#!/bin/bash'" echo '#!/bin/bash'"
# Load basic Python module # Load basic Python module
module purge module purge
module load Stages/2022 module load Stages/2024
module load GCC
module load GCC/11.2.0 module load Python
module load Python/3.9.6
module load ParaStationMPI
# Load extra modules you need for your kernel (as you did in step 1.2) # Load extra modules you need for your kernel (as you did in step 1.2)
#module load <module you need> # module load <module you need>
module load TensorFlow
module load OpenCV module load OpenCV
module load Jupyter
# Activate your Python virtual environment # Activate your Python virtual environment
source ${KERNEL_VENVS_DIR}/${KERNEL_NAME}/bin/activate source ${VIRTUAL_ENV}/bin/activate
# Ensure python packages installed in the virtual environment are always prefered # Ensure python packages installed in the virtual environment are always prefered
export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.9/site-packages:"'${PYTHONPATH}'" export PYTHONPATH=${VIRTUAL_ENV}/lib/python${PYV}/site-packages:"'${PYTHONPATH}'"
exec python -m ipykernel "'$@' > ${VIRTUAL_ENV}/kernel.sh exec python -m ipykernel "'$@' > ${VIRTUAL_ENV}/kernel.sh
chmod +x ${VIRTUAL_ENV}/kernel.sh chmod +x ${VIRTUAL_ENV}/kernel.sh
cat ${VIRTUAL_ENV}/kernel.sh # double check cat ${VIRTUAL_ENV}/kernel.sh # double check
``` ```
%% Output
(jovyan_kernel_ecar2023) (jovyan_kernel_ecar2023) (jovyan_kernel_ecar2023) #!/bin/bash
# Load basic Python module
module purge
module load Stages/2022
module load GCC/11.2.0
module load Python/3.9.6
module load ParaStationMPI
# Load extra modules you need for your kernel (as you did in step 1.2)
#module load <module you need>
module load TensorFlow
module load OpenCV
module load Jupyter
# Activate your Python virtual environment
source /home/jovyan/jupyter_kernel_venvs/jovyan_kernel_ecar2023/bin/activate
# Ensure python packages installed in the virtual environment are always prefered
export PYTHONPATH=/home/jovyan/jupyter_kernel_venvs/jovyan_kernel_ecar2023/lib/python3.9/site-packages:${PYTHONPATH}
exec python -m ipykernel $@
(jovyan_kernel_ecar2023)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## 3. Create/Edit Jupyter kernel configuration ## 3. Create/Edit Jupyter kernel configuration
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 3.1 - Create Jupyter kernel configuration directory and files #### 3.1 - Create Jupyter kernel configuration directory and files
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
python -m ipykernel install --name=${KERNEL_NAME} --prefix ${VIRTUAL_ENV} python -m ipykernel install --name=${KERNEL_NAME} --prefix ${VIRTUAL_ENV}
export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels
``` ```
%% Output
Installed kernelspec jovyan_kernel_ecar2023 in /home/jovyan/jupyter_kernel_venvs/jovyan_kernel_ecar2023/share/jupyter/kernels/jovyan_kernel_ecar2023
(jovyan_kernel_ecar2023) (jovyan_kernel_ecar2023)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 3.2 - Adjust kernel.json file #### 3.2 - Adjust kernel.json file
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
mv ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json.orig mv ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json.orig
echo '{ echo '{
"argv": [ "argv": [
"'${KERNEL_VENVS_DIR}/${KERNEL_NAME}/kernel.sh'", "'${KERNEL_VENVS_DIR}/${KERNEL_NAME}/kernel.sh'",
"-m", "-m",
"ipykernel_launcher", "ipykernel_launcher",
"-f", "-f",
"{connection_file}" "{connection_file}"
], ],
"display_name": "'${KERNEL_NAME}'", "display_name": "'${KERNEL_NAME}'",
"language": "python", "language": "python",
"metadata": { "metadata": {
"debugger": true "debugger": true
} }
}' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json }' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json
cat ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json # double check cat ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json # double check
``` ```
%% Output
(jovyan_kernel_ecar2023) (jovyan_kernel_ecar2023) (jovyan_kernel_ecar2023) (jovyan_kernel_ecar2023) {
"argv": [
"/home/jovyan/jupyter_kernel_venvs/jovyan_kernel_ecar2023/kernel.sh",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "jovyan_kernel_ecar2023",
"language": "python",
"metadata": {
"debugger": true
}
}
(jovyan_kernel_ecar2023)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 3.3 - Create link to kernel specs #### 3.3 - Create link to kernel specs
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
mkdir -p ${KERNEL_SPECS_DIR} mkdir -p ${KERNEL_SPECS_DIR}
cd ${KERNEL_SPECS_DIR} cd ${KERNEL_SPECS_DIR}
ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} . ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .
echo -e "\n\nThe new kernel '${KERNEL_NAME}' was added to your kernels in '${KERNEL_SPECS_DIR}/'\n" echo -e "\n\nThe new kernel '${KERNEL_NAME}' was added to your kernels in '${KERNEL_SPECS_DIR}/'\n"
ls ${KERNEL_SPECS_DIR} # double check ls ${KERNEL_SPECS_DIR} # double check
``` ```
%% Output
(jovyan_kernel_ecar2023) (jovyan_kernel_ecar2023) (jovyan_kernel_ecar2023) (jovyan_kernel_ecar2023)
The new kernel 'jovyan_kernel_ecar2023' was added to your kernels in '/home/jovyan/.local/share/jupyter/kernels/'
(jovyan_kernel_ecar2023) jovyan_kernel_ecar2023
(jovyan_kernel_ecar2023)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## 4. Cleanup ## 4. Cleanup
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
deactivate deactivate
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
----------------- -----------------
## 5. Check your new kernel ## 5. Check your new kernel
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
#### Exercise 1: #### Exercise 1:
- Check for the kernel directory: `$HOME/.local/share/jupyter/kernels/jovyan_kernel_ecar/` - Check for the kernel directory
- Check for the kernel definition file: `$HOME/.local/share/jupyter/kernels/jovyan_kernel_ecar/kernel.json` - Check for the kernel definition file
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Exercise 2: Exercise 2:
- Read/understand the content of the `kernel.json` file. - Read/understand the content of the `kernel.json` file.
- Follow the path to `kernel.sh` and check/understand its content, too. - Follow the path to `kernel.sh` and check/understand its content, too.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Exercise 3: Exercise 3:
- Open a new Jupyter notebook to test the new Jupyter kernel. - Open a new Jupyter notebook to test the new Jupyter kernel.
- Run `import cv2` which loads the Python interface to OpenCV. - Run `import cv2` which loads the Python interface to OpenCV.
- Double-check if this package comes from the loaded module by running `cv2` - Double-check if this package comes from the loaded module by running `cv2`
%% Cell type:code id: tags:
``` bash
```
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment