Commit eddec700 authored by Jens Henrik Göbbert's avatar Jens Henrik Göbbert
Browse files

update with pull-request

parent e64aa9c6
%% Cell type:markdown id: tags:
# Install containerized Jupyter kernel at Jupyter-JSC
This Jupyter notebook will walk you through the installation of a containerized Jupyter kernel (for use at Jupyter-JSC, but it should actually work with any Jupyter server on a system where Singularity is installed). Considerable performance improvements (especially with respect to kernel start-up times) over e.g. conda-based Jupyter kernels on distributed filesystems, as are typically installed on HPC systems, might be experienced. In the example below, the `base-notebook` from the [Jupyter docker stacks](https://jupyter-docker-stacks.readthedocs.io/en/latest/) is used as an IPython kernel (already having the required `ipykernel` package installed), the approach presented here might be extended to any other [Jupyter kernel compatible programming language](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels), though.
Requirements:
* Python environment with an installed `ipykernel` package in a Docker (or Singularity) container
* `container` group access for the JSC systems as described [here](https://apps.fz-juelich.de/jsc/hps/juwels/container-runtime.html#getting-access) in the docs
%% Cell type:markdown id: tags:
Check that the Singularity container runtime is available via the JupyterLab environment,
%% Cell type:code id: tags:
``` bash
singularity --version
```
%% Output
singularity version 3.6.4-1.el8
%% Cell type:markdown id: tags:
Specify the filesystem location that stores the Singularity container image,
%% Cell type:code id: tags:
``` bash
IMAGE_TARGET_DIR=/p/project/cesmtst/hoeflich1/jupyter-base-notebook
```
%% Cell type:markdown id: tags:
Optional, if you already have a Singularity container image available at the above location: Convert a containerized Python environment (e.g. the Jupyter `base-notebook` that is [available via Dockerhub](https://hub.docker.com/r/jupyter/base-notebook)) into a Singularity container image to be used as an example here,
%% Cell type:code id: tags:
``` bash
mkdir -p ${IMAGE_TARGET_DIR}
```
%% Cell type:markdown id: tags:
Note that pulling and converting the Dockerhub image will take a bit of time,
%% Cell type:code id: tags:
``` bash
singularity pull ${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif docker://jupyter/base-notebook &> singularity.log
```
%% Cell type:code id: tags:
``` bash
cat singularity.log | grep -v warn
```
%% Output
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8
Copying blob sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d
Copying blob sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601
Copying blob sha256:e3cbfeece0aec396b6793a798ed1b2aed3ef8f8693cc9b3036df537c1f8e34a1
Copying blob sha256:48bd2a353bd8ed1ad4b841de108ae42bccecc44b3f05c3fcada8a2a6f5fa09cf
Copying blob sha256:235d93b8ccf12e8378784dc15c5bd0cb08ff128d61b856d32026c5a533ac3c89
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:b6c06056c45bc1da74604fcf368b02794fe4e36dcae881f4c6b4fa32b37a1385
Copying blob sha256:60918bcbe6d44988e4e48db436996106cc7569a4b880488be9cac90ea6883ae0
Copying blob sha256:762f9ebe4ddc05e56e33f7aba2cdd1be62f747ecd9c8f9eadcb379debf3ebe06
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:1df9d491a0390ecc3f9fac4484c92b2a5f71a79450017f2fca1849f2d6e7f949
Copying blob sha256:be84c8c720e3c53037ac2c5cbc53cf9a2a674503b2c995da1351e5560f60cc12
Copying blob sha256:28807e96859dc8c00c96255dfa51a0822380638a092803e7143473d1870970fb
Copying blob sha256:bcdaf848f29a8bf0efc18a5883dc65a4a7a6b2c6cf4094e5115188ed22165a00
Copying blob sha256:49777cff52f155a9ba35e58102ecec7029dddf52aa4947f2cffbd1af12848e81
Copying blob sha256:7fb3bffa2e730b052c0c7aabd715303cc5830a05b992f2d3d70afeffa0a9ed4f
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying config sha256:79f074439b14ae0634f2f217e5debc159c4e8c3a9ff2e0119e4dc88f9c7e21a5
Writing manifest to image destination
Storing signatures
2021/01/19 11:59:33 info unpack layer: sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8
2021/01/19 11:59:34 info unpack layer: sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d
2021/01/19 11:59:34 info unpack layer: sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601
2021/01/19 11:59:34 info unpack layer: sha256:e3cbfeece0aec396b6793a798ed1b2aed3ef8f8693cc9b3036df537c1f8e34a1
2021/01/19 11:59:34 info unpack layer: sha256:48bd2a353bd8ed1ad4b841de108ae42bccecc44b3f05c3fcada8a2a6f5fa09cf
2021/01/19 11:59:34 info unpack layer: sha256:235d93b8ccf12e8378784dc15c5bd0cb08ff128d61b856d32026c5a533ac3c89
2021/01/19 11:59:34 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2021/01/19 11:59:34 info unpack layer: sha256:b6c06056c45bc1da74604fcf368b02794fe4e36dcae881f4c6b4fa32b37a1385
2021/01/19 11:59:34 info unpack layer: sha256:60918bcbe6d44988e4e48db436996106cc7569a4b880488be9cac90ea6883ae0
2021/01/19 11:59:34 info unpack layer: sha256:762f9ebe4ddc05e56e33f7aba2cdd1be62f747ecd9c8f9eadcb379debf3ebe06
2021/01/19 11:59:34 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2021/01/19 11:59:34 info unpack layer: sha256:1df9d491a0390ecc3f9fac4484c92b2a5f71a79450017f2fca1849f2d6e7f949
2021/01/19 11:59:36 info unpack layer: sha256:be84c8c720e3c53037ac2c5cbc53cf9a2a674503b2c995da1351e5560f60cc12
2021/01/19 11:59:40 info unpack layer: sha256:28807e96859dc8c00c96255dfa51a0822380638a092803e7143473d1870970fb
2021/01/19 11:59:40 info unpack layer: sha256:bcdaf848f29a8bf0efc18a5883dc65a4a7a6b2c6cf4094e5115188ed22165a00
2021/01/19 11:59:40 info unpack layer: sha256:49777cff52f155a9ba35e58102ecec7029dddf52aa4947f2cffbd1af12848e81
2021/01/19 11:59:40 info unpack layer: sha256:7fb3bffa2e730b052c0c7aabd715303cc5830a05b992f2d3d70afeffa0a9ed4f
2021/01/19 11:59:40 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
INFO: Creating SIF file...
%% Cell type:markdown id: tags:
Check that the Singularity image is available,
%% Cell type:code id: tags:
``` bash
ls -lah ${IMAGE_TARGET_DIR}
```
%% Output
total 177M
drwxr-sr-x 2 hoeflich1 cesmtst 4.0K Jan 19 11:59 .
drwxr-sr-x 5 hoeflich1 cesmtst 4.0K Jan 19 11:59 ..
-rwxr-xr-x 1 hoeflich1 cesmtst 183M Jan 19 11:59 jupyter-base-notebook.sif
%% Cell type:markdown id: tags:
Now, setup a Jupyter kernel specification with the `install-jupyter-kernel.sh` script from this repository (which basically writes a `kernel.json` file to the home directory location that Jupyter expects for user-specific kernels),
%% Cell type:code id: tags:
``` bash
KERNEL_DISPLAY_NAME=Singularity-Python # don't use whitespaces here!
SINGULARITY_IMAGE=${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif
```
%% Cell type:code id: tags:
``` bash
./install-singularity-jupyter-kernel.sh ${KERNEL_DISPLAY_NAME} ${SINGULARITY_IMAGE}
```
%% Cell type:markdown id: tags:
Check that the Jupyter kernel specification was written,
%% Cell type:code id: tags:
``` bash
cat ${HOME}/.local/share/jupyter/kernels/${KERNEL_DISPLAY_NAME}/kernel.json
```
%% Output
{
"argv": [
"singularity",
"exec",
"--cleanenv",
"/p/project/cesmtst/hoeflich1/jupyter-base-notebook/jupyter-base-notebook.sif",
"python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"language": "python",
"display_name": "Singularity-Python"
}
%% Cell type:markdown id: tags:
And that the above Singularity-Python kernel is visible by the Jupyter server,
%% Cell type:code id: tags:
``` bash
jupyter kernelspec list
```
%% Output
Available kernels:
singularity-python /p/home/jusers/hoeflich1/juwels/.local/share/jupyter/kernels/Singularity-Python
ruby /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Ruby/2.6.3-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/ruby
ir35 /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-R/3.5.3-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/ir35
pyquantum-1.0 /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-PyQuantum/1.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/pyquantum-1.0
pyparaview-5.8 /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-PyParaView/5.8.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/pyparaview-5.8
octave /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Octave/5.1.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/octave
julia-1.4 /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Julia/1.4.2-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/julia-1.4
javascript /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-JavaScript/5.2.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/javascript
cling-cpp17 /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Cling/0.6-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/cling-cpp17
bash /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Bash/0.7.1-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/bash
python3 /p/software/juwels/stages/Devel-2019a/software/Jupyter/2019a.2.4-gcccoremkl-8.3.0-2019.3.199-Python-3.6.8/share/jupyter/kernels/python3
%% Cell type:markdown id: tags:
If so, you should be able to choose and connect to the containerized Python kernel from the drop down menu and/or the kernel launcher tab (a reload of the JupyterLab web page might be necessary).
#!/bin/bash
# Author: Katharina Höflich
# Repository: https://github.com/FZJ-JSC/jupyter-jsc-notebooks
KERNEL_NAME=${1}
SINGULARITY_IMAGE=${2}
[[ ! -z "$KERNEL_NAME" ]] || echo "Provide a Jupyter kernel name, please."
[[ ! -z "$SINGULARITY_IMAGE" ]] || echo "Provide a Singularity container image, please."
USER_KERNEL_DIR=${HOME}/.local/share/jupyter/kernels/${KERNEL_NAME}
mkdir -p ${USER_KERNEL_DIR} || exit
#
#echo '{
# "argv": [
# "'"${USER_KERNEL_DIR}"'/singularity-kernel.sh",
# "-f",
# "{connection_file}"
# ],
# "display_name": "'"${KERNEL_NAME}"'",
# "language": "python"
#}' > ${USER_KERNEL_DIR}/kernel.json || exit
#
#echo '#!/bin/bash
#module purge
#SINGULARITY_IMAGE='"${SINGULARITY_IMAGE}"'
#singularity run ${SINGULARITY_IMAGE} python -m ipykernel $@
#' > ${USER_KERNEL_DIR}/singularity-kernel.sh || exit
#
#chmod +x ${USER_KERNEL_DIR}/singularity-kernel.sh
#
echo '{
"argv": [
"singularity",
"exec",
"--cleanenv",
"'"${SINGULARITY_IMAGE}"'",
"python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"language": "python",
"display_name": "'"${KERNEL_NAME}"'"
}' > ${USER_KERNEL_DIR}/kernel.json || exit