"# Install containerized Jupyter kernel at Jupyter-JSC\n",
"\n",
"This Jupyter notebook will walk you through the installation of a containerized Jupyter kernel (for use at Jupyter-JSC, but it should actually work with any Jupyter server on a system where Singularity is installed). Considerable performance improvements (especially with respect to kernel start-up times) over e.g. conda-based Jupyter kernels on distributed filesystems, as are typically installed on HPC systems, might be experienced. In the example below, the `base-notebook` from the [Jupyter docker stacks](https://jupyter-docker-stacks.readthedocs.io/en/latest/) is used as an IPython kernel (already having the required `ipykernel` package installed), the approach presented here might be extended to any other [Jupyter kernel compatible programming language](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels), though.\n",
"\n",
"Requirements:\n",
"\n",
"* Python environment with an installed `ipykernel` package in a Docker (or Singularity) container\n",
"* `container` group access for the JSC systems as described [here](https://apps.fz-juelich.de/jsc/hps/juwels/container-runtime.html#getting-access) in the docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check that the Singularity container runtime is available via the JupyterLab environment,"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"singularity version 3.6.4-1.el8\n"
]
}
],
"source": [
"singularity --version"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Specify the filesystem location that stores the Singularity container image,"
"Optional, if you already have a Singularity container image available at the above location: Convert a containerized Python environment (e.g. the Jupyter `base-notebook` that is [available via Dockerhub](https://hub.docker.com/r/jupyter/base-notebook)) into a Singularity container image to be used as an example here,"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"mkdir -p ${IMAGE_TARGET_DIR}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that pulling and converting the Dockerhub image will take a bit of time,"
"2021/01/19 11:59:33 info unpack layer: sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8\n",
"2021/01/19 11:59:34 info unpack layer: sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d\n",
"2021/01/19 11:59:34 info unpack layer: sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601\n",
"2021/01/19 11:59:34 info unpack layer: sha256:e3cbfeece0aec396b6793a798ed1b2aed3ef8f8693cc9b3036df537c1f8e34a1\n",
"2021/01/19 11:59:34 info unpack layer: sha256:48bd2a353bd8ed1ad4b841de108ae42bccecc44b3f05c3fcada8a2a6f5fa09cf\n",
"2021/01/19 11:59:34 info unpack layer: sha256:235d93b8ccf12e8378784dc15c5bd0cb08ff128d61b856d32026c5a533ac3c89\n",
"2021/01/19 11:59:34 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\n",
"2021/01/19 11:59:34 info unpack layer: sha256:b6c06056c45bc1da74604fcf368b02794fe4e36dcae881f4c6b4fa32b37a1385\n",
"2021/01/19 11:59:34 info unpack layer: sha256:60918bcbe6d44988e4e48db436996106cc7569a4b880488be9cac90ea6883ae0\n",
"2021/01/19 11:59:34 info unpack layer: sha256:762f9ebe4ddc05e56e33f7aba2cdd1be62f747ecd9c8f9eadcb379debf3ebe06\n",
"2021/01/19 11:59:34 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\n",
"2021/01/19 11:59:34 info unpack layer: sha256:1df9d491a0390ecc3f9fac4484c92b2a5f71a79450017f2fca1849f2d6e7f949\n",
"2021/01/19 11:59:36 info unpack layer: sha256:be84c8c720e3c53037ac2c5cbc53cf9a2a674503b2c995da1351e5560f60cc12\n",
"2021/01/19 11:59:40 info unpack layer: sha256:28807e96859dc8c00c96255dfa51a0822380638a092803e7143473d1870970fb\n",
"2021/01/19 11:59:40 info unpack layer: sha256:bcdaf848f29a8bf0efc18a5883dc65a4a7a6b2c6cf4094e5115188ed22165a00\n",
"2021/01/19 11:59:40 info unpack layer: sha256:49777cff52f155a9ba35e58102ecec7029dddf52aa4947f2cffbd1af12848e81\n",
"2021/01/19 11:59:40 info unpack layer: sha256:7fb3bffa2e730b052c0c7aabd715303cc5830a05b992f2d3d70afeffa0a9ed4f\n",
"2021/01/19 11:59:40 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\n",
"INFO: Creating SIF file...\n"
]
}
],
"source": [
"cat singularity.log | grep -v warn"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check that the Singularity image is available,"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 177M\n",
"drwxr-sr-x 2 hoeflich1 cesmtst 4.0K Jan 19 11:59 .\n",
"drwxr-sr-x 5 hoeflich1 cesmtst 4.0K Jan 19 11:59 ..\n",
"-rwxr-xr-x 1 hoeflich1 cesmtst 183M Jan 19 11:59 jupyter-base-notebook.sif\n"
]
}
],
"source": [
"ls -lah ${IMAGE_TARGET_DIR}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, setup a Jupyter kernel specification with the `install-jupyter-kernel.sh` script from this repository (which basically writes a `kernel.json` file to the home directory location that Jupyter expects for user-specific kernels),"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"KERNEL_DISPLAY_NAME=Singularity-Python # don't use whitespaces here!\n",
"If so, you should be able to choose and connect to the containerized Python kernel from the drop down menu and/or the kernel launcher tab (a reload of the JupyterLab web page might be necessary)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Bash",
"language": "bash",
"name": "bash"
},
"language_info": {
"codemirror_mode": "shell",
"file_extension": ".sh",
"mimetype": "text/x-sh",
"name": "bash"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
%% Cell type:markdown id: tags:
# Install containerized Jupyter kernel at Jupyter-JSC
This Jupyter notebook will walk you through the installation of a containerized Jupyter kernel (for use at Jupyter-JSC, but it should actually work with any Jupyter server on a system where Singularity is installed). Considerable performance improvements (especially with respect to kernel start-up times) over e.g. conda-based Jupyter kernels on distributed filesystems, as are typically installed on HPC systems, might be experienced. In the example below, the `base-notebook` from the [Jupyter docker stacks](https://jupyter-docker-stacks.readthedocs.io/en/latest/) is used as an IPython kernel (already having the required `ipykernel` package installed), the approach presented here might be extended to any other [Jupyter kernel compatible programming language](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels), though.
Requirements:
* Python environment with an installed `ipykernel` package in a Docker (or Singularity) container
*`container` group access for the JSC systems as described [here](https://apps.fz-juelich.de/jsc/hps/juwels/container-runtime.html#getting-access) in the docs
%% Cell type:markdown id: tags:
Check that the Singularity container runtime is available via the JupyterLab environment,
%% Cell type:code id: tags:
``` bash
singularity --version
```
%% Output
singularity version 3.6.4-1.el8
%% Cell type:markdown id: tags:
Specify the filesystem location that stores the Singularity container image,
Optional, if you already have a Singularity container image available at the above location: Convert a containerized Python environment (e.g. the Jupyter `base-notebook` that is [available via Dockerhub](https://hub.docker.com/r/jupyter/base-notebook)) into a Singularity container image to be used as an example here,
%% Cell type:code id: tags:
``` bash
mkdir-p${IMAGE_TARGET_DIR}
```
%% Cell type:markdown id: tags:
Note that pulling and converting the Dockerhub image will take a bit of time,
2021/01/19 11:59:33 info unpack layer: sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8
2021/01/19 11:59:34 info unpack layer: sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d
2021/01/19 11:59:34 info unpack layer: sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601
2021/01/19 11:59:34 info unpack layer: sha256:e3cbfeece0aec396b6793a798ed1b2aed3ef8f8693cc9b3036df537c1f8e34a1
2021/01/19 11:59:34 info unpack layer: sha256:48bd2a353bd8ed1ad4b841de108ae42bccecc44b3f05c3fcada8a2a6f5fa09cf
2021/01/19 11:59:34 info unpack layer: sha256:235d93b8ccf12e8378784dc15c5bd0cb08ff128d61b856d32026c5a533ac3c89
2021/01/19 11:59:34 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2021/01/19 11:59:34 info unpack layer: sha256:b6c06056c45bc1da74604fcf368b02794fe4e36dcae881f4c6b4fa32b37a1385
2021/01/19 11:59:34 info unpack layer: sha256:60918bcbe6d44988e4e48db436996106cc7569a4b880488be9cac90ea6883ae0
2021/01/19 11:59:34 info unpack layer: sha256:762f9ebe4ddc05e56e33f7aba2cdd1be62f747ecd9c8f9eadcb379debf3ebe06
2021/01/19 11:59:34 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2021/01/19 11:59:34 info unpack layer: sha256:1df9d491a0390ecc3f9fac4484c92b2a5f71a79450017f2fca1849f2d6e7f949
2021/01/19 11:59:36 info unpack layer: sha256:be84c8c720e3c53037ac2c5cbc53cf9a2a674503b2c995da1351e5560f60cc12
2021/01/19 11:59:40 info unpack layer: sha256:28807e96859dc8c00c96255dfa51a0822380638a092803e7143473d1870970fb
2021/01/19 11:59:40 info unpack layer: sha256:bcdaf848f29a8bf0efc18a5883dc65a4a7a6b2c6cf4094e5115188ed22165a00
2021/01/19 11:59:40 info unpack layer: sha256:49777cff52f155a9ba35e58102ecec7029dddf52aa4947f2cffbd1af12848e81
2021/01/19 11:59:40 info unpack layer: sha256:7fb3bffa2e730b052c0c7aabd715303cc5830a05b992f2d3d70afeffa0a9ed4f
2021/01/19 11:59:40 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
INFO: Creating SIF file...
%% Cell type:markdown id: tags:
Check that the Singularity image is available,
%% Cell type:code id: tags:
``` bash
ls-lah${IMAGE_TARGET_DIR}
```
%% Output
total 177M
drwxr-sr-x 2 hoeflich1 cesmtst 4.0K Jan 19 11:59 .
drwxr-sr-x 5 hoeflich1 cesmtst 4.0K Jan 19 11:59 ..
-rwxr-xr-x 1 hoeflich1 cesmtst 183M Jan 19 11:59 jupyter-base-notebook.sif
%% Cell type:markdown id: tags:
Now, setup a Jupyter kernel specification with the `install-jupyter-kernel.sh` script from this repository (which basically writes a `kernel.json` file to the home directory location that Jupyter expects for user-specific kernels),
%% Cell type:code id: tags:
``` bash
KERNEL_DISPLAY_NAME=Singularity-Python # don't use whitespaces here!
If so, you should be able to choose and connect to the containerized Python kernel from the drop down menu and/or the kernel launcher tab (a reload of the JupyterLab web page might be necessary).
"# Setup containerized Jupyter server for Jupyter-JSC\n",
"\n",
"This Jupyter notebook will explain how to setup a containerized Jupyter server at Jupyter-JSC. It makes use of the expert features described on page 22 (as of November 25th, 2020) of the training material available [here](https://jupyter-jsc.fz-juelich.de/nbviewer/github/FZJ-JSC/jupyter-jsc-notebooks/blob/master/Jupyter-JSC_supercomputing-in-the-browser.pdf). Please note, that setting up a containerized Jupyter server for the JupyterHub at JSC might introduce certain drawbacks to your Jupyter-JSC experience. Specifically, you will be restricted to the software environment that is installed in your container environment only, which might introduce unwanted side-effects to your JupyterLab-based workflows on the JSC HPC systems. For example, usage of the SLURM batch scheduler commands is not possible, because the SLURM libraries are not visible from the container environment per default. Also, you won't be able to use the Lmod software environment modules provided by JSC. Please note, that if these kind of side-effects are not acceptable, you might rather use a containerized Jupyter kernel as [described here](install-singularity-jupyter-kernel.ipynb). You could also setup your own non-containerized JupyterLab server.\n",
"\n",
"Please note, you can switch back to the default Jupyter-JSC server environment anytime by deleting `$HOME/.jupyter/start_jupyter-jsc.sh` after [login to the JSC systems](https://apps.fz-juelich.de/jsc/hps/juwels/access.html) via SSH.\n",
"\n",
"Requirements:\n",
"\n",
"* Jupyter server environment in a Docker (or Singularity) container\n",
"* `container` group access for the JSC systems as described [here](https://apps.fz-juelich.de/jsc/hps/juwels/container-runtime.html#getting-access) in the docs\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Specify the filesystem location that stores the Singularity container image,"
"Convert the example Jupyter base-notebook (that is [available via Dockerhub](https://hub.docker.com/r/jupyter/base-notebook)) into a Singularity container image,"
