update with pull-request

eddec700 · Jens Henrik Göbbert · e64aa9c6 · eddec700 · eddec700 · eddec700
Commit eddec700 authored 4 years ago by Jens Henrik Göbbert
--- a/001-Jupyter/004-Create_JupyterKernel_container/install-singularity-jupyter-kernel.ipynb
+++ b/001-Jupyter/004-Create_JupyterKernel_container/install-singularity-jupyter-kernel.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Install containerized Jupyter kernel at Jupyter-JSC\n",
+    "\n",
+    "This Jupyter notebook will walk you through the installation of a containerized Jupyter kernel (for use at Jupyter-JSC, but it should actually work with any Jupyter server on a system where Singularity is installed). Considerable performance improvements (especially with respect to kernel start-up times) over e.g. conda-based Jupyter kernels on distributed filesystems, as are typically installed on HPC systems, might be experienced. In the example below, the `base-notebook` from the [Jupyter docker stacks](https://jupyter-docker-stacks.readthedocs.io/en/latest/) is used as an IPython kernel (already having the required `ipykernel` package installed), the approach presented here might be extended to any other [Jupyter kernel compatible programming language](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels), though.\n",
+    "\n",
+    "Requirements:\n",
+    "\n",
+    "* Python environment with an installed `ipykernel` package in a Docker (or Singularity) container\n",
+    "* `container` group access for the JSC systems as described [here](https://apps.fz-juelich.de/jsc/hps/juwels/container-runtime.html#getting-access) in the docs"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Check that the Singularity container runtime is available via the JupyterLab environment,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "singularity version 3.6.4-1.el8\n"
+     ]
+    }
+   ],
+   "source": [
+    "singularity --version"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Specify the filesystem location that stores the Singularity container image,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "IMAGE_TARGET_DIR=/p/project/cesmtst/hoeflich1/jupyter-base-notebook"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Optional, if you already have a Singularity container image available at the above location: Convert a containerized Python environment (e.g. the Jupyter `base-notebook` that is [available via Dockerhub](https://hub.docker.com/r/jupyter/base-notebook)) into a Singularity container image to be used as an example here,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mkdir -p ${IMAGE_TARGET_DIR}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that pulling and converting the Dockerhub image will take a bit of time,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "singularity pull ${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif docker://jupyter/base-notebook &> singularity.log"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "INFO:    Converting OCI blobs to SIF format\n",
+      "INFO:    Starting build...\n",
+      "Getting image source signatures\n",
+      "Copying blob sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8\n",
+      "Copying blob sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d\n",
+      "Copying blob sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601\n",
+      "Copying blob sha256:e3cbfeece0aec396b6793a798ed1b2aed3ef8f8693cc9b3036df537c1f8e34a1\n",
+      "Copying blob sha256:48bd2a353bd8ed1ad4b841de108ae42bccecc44b3f05c3fcada8a2a6f5fa09cf\n",
+      "Copying blob sha256:235d93b8ccf12e8378784dc15c5bd0cb08ff128d61b856d32026c5a533ac3c89\n",
+      "Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\n",
+      "Copying blob sha256:b6c06056c45bc1da74604fcf368b02794fe4e36dcae881f4c6b4fa32b37a1385\n",
+      "Copying blob sha256:60918bcbe6d44988e4e48db436996106cc7569a4b880488be9cac90ea6883ae0\n",
+      "Copying blob sha256:762f9ebe4ddc05e56e33f7aba2cdd1be62f747ecd9c8f9eadcb379debf3ebe06\n",
+      "Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\n",
+      "Copying blob sha256:1df9d491a0390ecc3f9fac4484c92b2a5f71a79450017f2fca1849f2d6e7f949\n",
+      "Copying blob sha256:be84c8c720e3c53037ac2c5cbc53cf9a2a674503b2c995da1351e5560f60cc12\n",
+      "Copying blob sha256:28807e96859dc8c00c96255dfa51a0822380638a092803e7143473d1870970fb\n",
+      "Copying blob sha256:bcdaf848f29a8bf0efc18a5883dc65a4a7a6b2c6cf4094e5115188ed22165a00\n",
+      "Copying blob sha256:49777cff52f155a9ba35e58102ecec7029dddf52aa4947f2cffbd1af12848e81\n",
+      "Copying blob sha256:7fb3bffa2e730b052c0c7aabd715303cc5830a05b992f2d3d70afeffa0a9ed4f\n",
+      "Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\n",
+      "Copying config sha256:79f074439b14ae0634f2f217e5debc159c4e8c3a9ff2e0119e4dc88f9c7e21a5\n",
+      "Writing manifest to image destination\n",
+      "Storing signatures\n",
+      "2021/01/19 11:59:33  info unpack layer: sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:e3cbfeece0aec396b6793a798ed1b2aed3ef8f8693cc9b3036df537c1f8e34a1\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:48bd2a353bd8ed1ad4b841de108ae42bccecc44b3f05c3fcada8a2a6f5fa09cf\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:235d93b8ccf12e8378784dc15c5bd0cb08ff128d61b856d32026c5a533ac3c89\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:b6c06056c45bc1da74604fcf368b02794fe4e36dcae881f4c6b4fa32b37a1385\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:60918bcbe6d44988e4e48db436996106cc7569a4b880488be9cac90ea6883ae0\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:762f9ebe4ddc05e56e33f7aba2cdd1be62f747ecd9c8f9eadcb379debf3ebe06\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\n",
+      "2021/01/19 11:59:34  info unpack layer: sha256:1df9d491a0390ecc3f9fac4484c92b2a5f71a79450017f2fca1849f2d6e7f949\n",
+      "2021/01/19 11:59:36  info unpack layer: sha256:be84c8c720e3c53037ac2c5cbc53cf9a2a674503b2c995da1351e5560f60cc12\n",
+      "2021/01/19 11:59:40  info unpack layer: sha256:28807e96859dc8c00c96255dfa51a0822380638a092803e7143473d1870970fb\n",
+      "2021/01/19 11:59:40  info unpack layer: sha256:bcdaf848f29a8bf0efc18a5883dc65a4a7a6b2c6cf4094e5115188ed22165a00\n",
+      "2021/01/19 11:59:40  info unpack layer: sha256:49777cff52f155a9ba35e58102ecec7029dddf52aa4947f2cffbd1af12848e81\n",
+      "2021/01/19 11:59:40  info unpack layer: sha256:7fb3bffa2e730b052c0c7aabd715303cc5830a05b992f2d3d70afeffa0a9ed4f\n",
+      "2021/01/19 11:59:40  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\n",
+      "INFO:    Creating SIF file...\n"
+     ]
+    }
+   ],
+   "source": [
+    "cat singularity.log | grep -v warn"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Check that the Singularity image is available,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "total 177M\n",
+      "drwxr-sr-x 2 hoeflich1 cesmtst 4.0K Jan 19 11:59 .\n",
+      "drwxr-sr-x 5 hoeflich1 cesmtst 4.0K Jan 19 11:59 ..\n",
+      "-rwxr-xr-x 1 hoeflich1 cesmtst 183M Jan 19 11:59 jupyter-base-notebook.sif\n"
+     ]
+    }
+   ],
+   "source": [
+    "ls -lah ${IMAGE_TARGET_DIR}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, setup a Jupyter kernel specification with the `install-jupyter-kernel.sh` script from this repository (which basically writes a `kernel.json` file to the home directory location that Jupyter expects for user-specific kernels),"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "KERNEL_DISPLAY_NAME=Singularity-Python # don't use whitespaces here!\n",
+    "SINGULARITY_IMAGE=${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "./install-singularity-jupyter-kernel.sh ${KERNEL_DISPLAY_NAME} ${SINGULARITY_IMAGE}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Check that the Jupyter kernel specification was written,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      " \"argv\": [\n",
+      "   \"singularity\",\n",
+      "   \"exec\",\n",
+      "   \"--cleanenv\",\n",
+      "   \"/p/project/cesmtst/hoeflich1/jupyter-base-notebook/jupyter-base-notebook.sif\",\n",
+      "   \"python\",\n",
+      "   \"-m\",\n",
+      "   \"ipykernel\",\n",
+      "   \"-f\",\n",
+      "   \"{connection_file}\"\n",
+      " ],\n",
+      " \"language\": \"python\",\n",
+      " \"display_name\": \"Singularity-Python\"\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "cat ${HOME}/.local/share/jupyter/kernels/${KERNEL_DISPLAY_NAME}/kernel.json"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And that the above Singularity-Python kernel is visible by the Jupyter server,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Available kernels:\n",
+      "  singularity-python    /p/home/jusers/hoeflich1/juwels/.local/share/jupyter/kernels/Singularity-Python\n",
+      "  ruby                  /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Ruby/2.6.3-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/ruby\n",
+      "  ir35                  /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-R/3.5.3-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/ir35\n",
+      "  pyquantum-1.0         /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-PyQuantum/1.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/pyquantum-1.0\n",
+      "  pyparaview-5.8        /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-PyParaView/5.8.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/pyparaview-5.8\n",
+      "  octave                /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Octave/5.1.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/octave\n",
+      "  julia-1.4             /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Julia/1.4.2-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/julia-1.4\n",
+      "  javascript            /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-JavaScript/5.2.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/javascript\n",
+      "  cling-cpp17           /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Cling/0.6-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/cling-cpp17\n",
+      "  bash                  /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Bash/0.7.1-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/bash\n",
+      "  python3               /p/software/juwels/stages/Devel-2019a/software/Jupyter/2019a.2.4-gcccoremkl-8.3.0-2019.3.199-Python-3.6.8/share/jupyter/kernels/python3\n"
+     ]
+    }
+   ],
+   "source": [
+    "jupyter kernelspec list"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If so, you should be able to choose and connect to the containerized Python kernel from the drop down menu and/or the kernel launcher tab (a reload of the JupyterLab web page might be necessary)."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Bash",
+   "language": "bash",
+   "name": "bash"
+  },
+  "language_info": {
+   "codemirror_mode": "shell",
+   "file_extension": ".sh",
+   "mimetype": "text/x-sh",
+   "name": "bash"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
+%% Cell type:markdown id: tags:
+# Install containerized Jupyter kernel at Jupyter-JSC
+This Jupyter notebook will walk you through the installation of a containerized Jupyter kernel (for use at Jupyter-JSC, but it should actually work with any Jupyter server on a system where Singularity is installed). Considerable performance improvements (especially with respect to kernel start-up times) over e.g. conda-based Jupyter kernels on distributed filesystems, as are typically installed on HPC systems, might be experienced. In the example below, the `base-notebook` from the [Jupyter docker stacks](https://jupyter-docker-stacks.readthedocs.io/en/latest/) is used as an IPython kernel (already having the required `ipykernel` package installed), the approach presented here might be extended to any other [Jupyter kernel compatible programming language](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels), though.
+Requirements:
+* Python environment with an installed `ipykernel` package in a Docker (or Singularity) container
+* `container` group access for the JSC systems as described [here](https://apps.fz-juelich.de/jsc/hps/juwels/container-runtime.html#getting-access) in the docs
+%% Cell type:markdown id: tags:
+Check that the Singularity container runtime is available via the JupyterLab environment,
+%% Cell type:code id: tags:
+``` bash
+singularity --version
+```
+%% Output
+    singularity version 3.6.4-1.el8
+%% Cell type:markdown id: tags:
+Specify the filesystem location that stores the Singularity container image,
+%% Cell type:code id: tags:
+``` bash
+IMAGE_TARGET_DIR=/p/project/cesmtst/hoeflich1/jupyter-base-notebook
+```
+%% Cell type:markdown id: tags:
+Optional, if you already have a Singularity container image available at the above location: Convert a containerized Python environment (e.g. the Jupyter `base-notebook` that is [available via Dockerhub](https://hub.docker.com/r/jupyter/base-notebook)) into a Singularity container image to be used as an example here,
+%% Cell type:code id: tags:
+``` bash
+mkdir -p ${IMAGE_TARGET_DIR}
+```
+%% Cell type:markdown id: tags:
+Note that pulling and converting the Dockerhub image will take a bit of time,
+%% Cell type:code id: tags:
+``` bash
+singularity pull ${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif docker://jupyter/base-notebook &> singularity.log
+```
+%% Cell type:code id: tags:
+``` bash
+cat singularity.log | grep -v warn
+```
+%% Output
+    INFO:    Converting OCI blobs to SIF format
+    INFO:    Starting build...
+    Getting image source signatures
+    Copying blob sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8
+    Copying blob sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d
+    Copying blob sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601
+    Copying blob sha256:e3cbfeece0aec396b6793a798ed1b2aed3ef8f8693cc9b3036df537c1f8e34a1
+    Copying blob sha256:48bd2a353bd8ed1ad4b841de108ae42bccecc44b3f05c3fcada8a2a6f5fa09cf
+    Copying blob sha256:235d93b8ccf12e8378784dc15c5bd0cb08ff128d61b856d32026c5a533ac3c89
+    Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
+    Copying blob sha256:b6c06056c45bc1da74604fcf368b02794fe4e36dcae881f4c6b4fa32b37a1385
+    Copying blob sha256:60918bcbe6d44988e4e48db436996106cc7569a4b880488be9cac90ea6883ae0
+    Copying blob sha256:762f9ebe4ddc05e56e33f7aba2cdd1be62f747ecd9c8f9eadcb379debf3ebe06
+    Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
+    Copying blob sha256:1df9d491a0390ecc3f9fac4484c92b2a5f71a79450017f2fca1849f2d6e7f949
+    Copying blob sha256:be84c8c720e3c53037ac2c5cbc53cf9a2a674503b2c995da1351e5560f60cc12
+    Copying blob sha256:28807e96859dc8c00c96255dfa51a0822380638a092803e7143473d1870970fb
+    Copying blob sha256:bcdaf848f29a8bf0efc18a5883dc65a4a7a6b2c6cf4094e5115188ed22165a00
+    Copying blob sha256:49777cff52f155a9ba35e58102ecec7029dddf52aa4947f2cffbd1af12848e81
+    Copying blob sha256:7fb3bffa2e730b052c0c7aabd715303cc5830a05b992f2d3d70afeffa0a9ed4f
+    Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
+    Copying config sha256:79f074439b14ae0634f2f217e5debc159c4e8c3a9ff2e0119e4dc88f9c7e21a5
+    Writing manifest to image destination
+    Storing signatures
+    2021/01/19 11:59:33  info unpack layer: sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8
+    2021/01/19 11:59:34  info unpack layer: sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d
+    2021/01/19 11:59:34  info unpack layer: sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601
+    2021/01/19 11:59:34  info unpack layer: sha256:e3cbfeece0aec396b6793a798ed1b2aed3ef8f8693cc9b3036df537c1f8e34a1
+    2021/01/19 11:59:34  info unpack layer: sha256:48bd2a353bd8ed1ad4b841de108ae42bccecc44b3f05c3fcada8a2a6f5fa09cf
+    2021/01/19 11:59:34  info unpack layer: sha256:235d93b8ccf12e8378784dc15c5bd0cb08ff128d61b856d32026c5a533ac3c89
+    2021/01/19 11:59:34  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
+    2021/01/19 11:59:34  info unpack layer: sha256:b6c06056c45bc1da74604fcf368b02794fe4e36dcae881f4c6b4fa32b37a1385
+    2021/01/19 11:59:34  info unpack layer: sha256:60918bcbe6d44988e4e48db436996106cc7569a4b880488be9cac90ea6883ae0
+    2021/01/19 11:59:34  info unpack layer: sha256:762f9ebe4ddc05e56e33f7aba2cdd1be62f747ecd9c8f9eadcb379debf3ebe06
+    2021/01/19 11:59:34  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
+    2021/01/19 11:59:34  info unpack layer: sha256:1df9d491a0390ecc3f9fac4484c92b2a5f71a79450017f2fca1849f2d6e7f949
+    2021/01/19 11:59:36  info unpack layer: sha256:be84c8c720e3c53037ac2c5cbc53cf9a2a674503b2c995da1351e5560f60cc12
+    2021/01/19 11:59:40  info unpack layer: sha256:28807e96859dc8c00c96255dfa51a0822380638a092803e7143473d1870970fb
+    2021/01/19 11:59:40  info unpack layer: sha256:bcdaf848f29a8bf0efc18a5883dc65a4a7a6b2c6cf4094e5115188ed22165a00
+    2021/01/19 11:59:40  info unpack layer: sha256:49777cff52f155a9ba35e58102ecec7029dddf52aa4947f2cffbd1af12848e81
+    2021/01/19 11:59:40  info unpack layer: sha256:7fb3bffa2e730b052c0c7aabd715303cc5830a05b992f2d3d70afeffa0a9ed4f
+    2021/01/19 11:59:40  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
+    INFO:    Creating SIF file...
+%% Cell type:markdown id: tags:
+Check that the Singularity image is available,
+%% Cell type:code id: tags:
+``` bash
+ls -lah ${IMAGE_TARGET_DIR}
+```
+%% Output
+    total 177M
+    drwxr-sr-x 2 hoeflich1 cesmtst 4.0K Jan 19 11:59 .
+    drwxr-sr-x 5 hoeflich1 cesmtst 4.0K Jan 19 11:59 ..
+    -rwxr-xr-x 1 hoeflich1 cesmtst 183M Jan 19 11:59 jupyter-base-notebook.sif
+%% Cell type:markdown id: tags:
+Now, setup a Jupyter kernel specification with the `install-jupyter-kernel.sh` script from this repository (which basically writes a `kernel.json` file to the home directory location that Jupyter expects for user-specific kernels),
+%% Cell type:code id: tags:
+``` bash
+KERNEL_DISPLAY_NAME=Singularity-Python # don't use whitespaces here!
+SINGULARITY_IMAGE=${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif
+```
+%% Cell type:code id: tags:
+``` bash
+./install-singularity-jupyter-kernel.sh ${KERNEL_DISPLAY_NAME} ${SINGULARITY_IMAGE}
+```
+%% Cell type:markdown id: tags:
+Check that the Jupyter kernel specification was written,
+%% Cell type:code id: tags:
+``` bash
+cat ${HOME}/.local/share/jupyter/kernels/${KERNEL_DISPLAY_NAME}/kernel.json
+```
+%% Output
+    {
+     "argv": [
+       "singularity",
+       "exec",
+       "--cleanenv",
+       "/p/project/cesmtst/hoeflich1/jupyter-base-notebook/jupyter-base-notebook.sif",
+       "python",
+       "-m",
+       "ipykernel",
+       "-f",
+       "{connection_file}"
+     ],
+     "language": "python",
+     "display_name": "Singularity-Python"
+    }
+%% Cell type:markdown id: tags:
+And that the above Singularity-Python kernel is visible by the Jupyter server,
+%% Cell type:code id: tags:
+``` bash
+jupyter kernelspec list
+```
+%% Output
+    Available kernels:
+      singularity-python    /p/home/jusers/hoeflich1/juwels/.local/share/jupyter/kernels/Singularity-Python
+      ruby                  /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Ruby/2.6.3-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/ruby
+      ir35                  /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-R/3.5.3-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/ir35
+      pyquantum-1.0         /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-PyQuantum/1.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/pyquantum-1.0
+      pyparaview-5.8        /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-PyParaView/5.8.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/pyparaview-5.8
+      octave                /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Octave/5.1.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/octave
+      julia-1.4             /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Julia/1.4.2-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/julia-1.4
+      javascript            /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-JavaScript/5.2.0-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/javascript
+      cling-cpp17           /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Cling/0.6-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/cling-cpp17
+      bash                  /p/software/juwels/stages/Devel-2019a/software/JupyterKernel-Bash/0.7.1-gcccoremkl-8.3.0-2019.3.199-2019a.2.4/share/jupyter/kernels/bash
+      python3               /p/software/juwels/stages/Devel-2019a/software/Jupyter/2019a.2.4-gcccoremkl-8.3.0-2019.3.199-Python-3.6.8/share/jupyter/kernels/python3
+%% Cell type:markdown id: tags:
+If so, you should be able to choose and connect to the containerized Python kernel from the drop down menu and/or the kernel launcher tab (a reload of the JupyterLab web page might be necessary).
--- a/001-Jupyter/004-Create_JupyterKernel_container/install-singularity-jupyter-kernel.sh
+++ b/001-Jupyter/004-Create_JupyterKernel_container/install-singularity-jupyter-kernel.sh
+#!/bin/bash
+# Author: Katharina Höflich
+# Repository: https://github.com/FZJ-JSC/jupyter-jsc-notebooks
+KERNEL_NAME=${1}
+SINGULARITY_IMAGE=${2}
+[[ ! -z "$KERNEL_NAME" ]] || echo "Provide a Jupyter kernel name, please."
+[[ ! -z "$SINGULARITY_IMAGE" ]] || echo "Provide a Singularity container image, please."
+USER_KERNEL_DIR=${HOME}/.local/share/jupyter/kernels/${KERNEL_NAME}
+mkdir -p ${USER_KERNEL_DIR} || exit
+#
+#echo '{
+# "argv": [
+#  "'"${USER_KERNEL_DIR}"'/singularity-kernel.sh",
+#  "-f",
+#  "{connection_file}"
+# ],
+# "display_name": "'"${KERNEL_NAME}"'",
+# "language": "python"
+#}' > ${USER_KERNEL_DIR}/kernel.json || exit
+#
+#echo '#!/bin/bash
+#module purge
+#SINGULARITY_IMAGE='"${SINGULARITY_IMAGE}"'
+#singularity run ${SINGULARITY_IMAGE} python -m ipykernel $@
+#' > ${USER_KERNEL_DIR}/singularity-kernel.sh || exit
+#
+#chmod +x ${USER_KERNEL_DIR}/singularity-kernel.sh
+#
+echo '{
+ "argv": [
+   "singularity",
+   "exec",
+   "--cleanenv",
+   "'"${SINGULARITY_IMAGE}"'",
+   "python",
+   "-m",
+   "ipykernel",
+   "-f",
+   "{connection_file}"
+ ],
+ "language": "python",
+ "display_name": "'"${KERNEL_NAME}"'"
+}' > ${USER_KERNEL_DIR}/kernel.json || exit
--- a/001-Jupyter/004-Create_JupyterKernel_container/setup-singularity-jupyter-server.ipynb
+++ b/001-Jupyter/004-Create_JupyterKernel_container/setup-singularity-jupyter-server.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Setup containerized Jupyter server for Jupyter-JSC\n",
+    "\n",
+    "This Jupyter notebook will explain how to setup a containerized Jupyter server at Jupyter-JSC. It makes use of the expert features described on page 22 (as of November 25th, 2020) of the training material available [here](https://jupyter-jsc.fz-juelich.de/nbviewer/github/FZJ-JSC/jupyter-jsc-notebooks/blob/master/Jupyter-JSC_supercomputing-in-the-browser.pdf). Please note, that setting up a containerized Jupyter server for the JupyterHub at JSC might introduce certain drawbacks to your Jupyter-JSC experience. Specifically, you will be restricted to the software environment that is installed in your container environment only, which might introduce unwanted side-effects to your JupyterLab-based workflows on the JSC HPC systems. For example, usage of the SLURM batch scheduler commands is not possible, because the SLURM libraries are not visible from the container environment per default. Also, you won't be able to use the Lmod software environment modules provided by JSC. Please note, that if these kind of side-effects are not acceptable, you might rather use a containerized Jupyter kernel as [described here](install-singularity-jupyter-kernel.ipynb). You could also setup your own non-containerized JupyterLab server.\n",
+    "\n",
+    "Please note, you can switch back to the default Jupyter-JSC server environment anytime by deleting `$HOME/.jupyter/start_jupyter-jsc.sh` after [login to the JSC systems](https://apps.fz-juelich.de/jsc/hps/juwels/access.html) via SSH.\n",
+    "\n",
+    "Requirements:\n",
+    "\n",
+    "* Jupyter server environment in a Docker (or Singularity) container\n",
+    "* `container` group access for the JSC systems as described [here](https://apps.fz-juelich.de/jsc/hps/juwels/container-runtime.html#getting-access) in the docs\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Specify the filesystem location that stores the Singularity container image,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "IMAGE_TARGET_DIR=/p/project/cesmtst/hoeflich1/jupyter-base-notebook"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Convert the example Jupyter base-notebook (that is [available via Dockerhub](https://hub.docker.com/r/jupyter/base-notebook)) into a Singularity container image,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mkdir -p ${IMAGE_TARGET_DIR}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "singularity pull --force ${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif docker://jupyter/base-notebook &> singularity.log"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "INFO:    Using cached SIF image\n"
+     ]
+    }
+   ],
+   "source": [
+    "cat singularity.log | grep -v warn"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Check that the Singularity image is available,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "total 177M\n",
+      "drwxr-sr-x 2 hoeflich1 cesmtst 4.0K Jan 19 18:50 .\n",
+      "drwxr-sr-x 5 hoeflich1 cesmtst 4.0K Jan 19 18:05 ..\n",
+      "-rwxr-xr-x 1 hoeflich1 cesmtst 183M Jan 19 18:50 jupyter-base-notebook.sif\n"
+     ]
+    }
+   ],
+   "source": [
+    "ls -lah ${IMAGE_TARGET_DIR}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, manually (!) specify the Singularity image filesystem location in the `start_jupyter-jsc.sh` script and check that the specified path is correct,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "#!/bin/bash\n",
+      "\n",
+      "# Author: Katharina Höflich\n",
+      "# Repository: https://github.com/FZJ-JSC/jupyter-jsc-notebooks\n",
+      "\n",
+      "SINGULARITY_IMAGE=/p/project/cesmtst/hoeflich1/jupyter-base-notebook/jupyter-base-notebook.sif\n",
+      "JUPYTERJSC_USER_CMD=\"singularity exec ${SINGULARITY_IMAGE} jupyterhub-singleuser --config ${JUPYTER_LOG_DIR}/.config.py\"\n"
+     ]
+    }
+   ],
+   "source": [
+    "cat start_jupyter-jsc.sh"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And copy the `start_jupyter-jsc.sh` script to the filesystem location expected by Jupyter-JSC,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cp start_jupyter-jsc.sh $HOME/.jupyter/start_jupyter-jsc.sh"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, opening a new Jupyter session via the Jupyter-JSC control panel should now load the containerized Jupyter server that was setup here."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Bash",
+   "language": "bash",
+   "name": "bash"
+  },
+  "language_info": {
+   "codemirror_mode": "shell",
+   "file_extension": ".sh",
+   "mimetype": "text/x-sh",
+   "name": "bash"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
+%% Cell type:markdown id: tags:
+# Setup containerized Jupyter server for Jupyter-JSC
+This Jupyter notebook will explain how to setup a containerized Jupyter server at Jupyter-JSC. It makes use of the expert features described on page 22 (as of November 25th, 2020) of the training material available [here](https://jupyter-jsc.fz-juelich.de/nbviewer/github/FZJ-JSC/jupyter-jsc-notebooks/blob/master/Jupyter-JSC_supercomputing-in-the-browser.pdf). Please note, that setting up a containerized Jupyter server for the JupyterHub at JSC might introduce certain drawbacks to your Jupyter-JSC experience. Specifically, you will be restricted to the software environment that is installed in your container environment only, which might introduce unwanted side-effects to your JupyterLab-based workflows on the JSC HPC systems. For example, usage of the SLURM batch scheduler commands is not possible, because the SLURM libraries are not visible from the container environment per default. Also, you won't be able to use the Lmod software environment modules provided by JSC. Please note, that if these kind of side-effects are not acceptable, you might rather use a containerized Jupyter kernel as [described here](install-singularity-jupyter-kernel.ipynb). You could also setup your own non-containerized JupyterLab server.
+Please note, you can switch back to the default Jupyter-JSC server environment anytime by deleting `$HOME/.jupyter/start_jupyter-jsc.sh` after [login to the JSC systems](https://apps.fz-juelich.de/jsc/hps/juwels/access.html) via SSH.
+Requirements:
+* Jupyter server environment in a Docker (or Singularity) container
+* `container` group access for the JSC systems as described [here](https://apps.fz-juelich.de/jsc/hps/juwels/container-runtime.html#getting-access) in the docs
+%% Cell type:markdown id: tags:
+Specify the filesystem location that stores the Singularity container image,
+%% Cell type:code id: tags:
+``` bash
+IMAGE_TARGET_DIR=/p/project/cesmtst/hoeflich1/jupyter-base-notebook
+```
+%% Cell type:markdown id: tags:
+Convert the example Jupyter base-notebook (that is [available via Dockerhub](https://hub.docker.com/r/jupyter/base-notebook)) into a Singularity container image,
+%% Cell type:code id: tags:
+``` bash
+mkdir -p ${IMAGE_TARGET_DIR}
+```
+%% Cell type:code id: tags:
+``` bash
+singularity pull --force ${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif docker://jupyter/base-notebook &> singularity.log
+```
+%% Cell type:code id: tags:
+``` bash
+cat singularity.log | grep -v warn
+```
+%% Output
+    INFO:    Using cached SIF image
+%% Cell type:markdown id: tags:
+Check that the Singularity image is available,
+%% Cell type:code id: tags:
+``` bash
+ls -lah ${IMAGE_TARGET_DIR}
+```
+%% Output
+    total 177M
+    drwxr-sr-x 2 hoeflich1 cesmtst 4.0K Jan 19 18:50 .
+    drwxr-sr-x 5 hoeflich1 cesmtst 4.0K Jan 19 18:05 ..
+    -rwxr-xr-x 1 hoeflich1 cesmtst 183M Jan 19 18:50 jupyter-base-notebook.sif
+%% Cell type:markdown id: tags:
+Now, manually (!) specify the Singularity image filesystem location in the `start_jupyter-jsc.sh` script and check that the specified path is correct,
+%% Cell type:code id: tags:
+``` bash
+cat start_jupyter-jsc.sh
+```
+%% Output
+    #!/bin/bash
+    # Author: Katharina Höflich
+    # Repository: https://github.com/FZJ-JSC/jupyter-jsc-notebooks
+    SINGULARITY_IMAGE=/p/project/cesmtst/hoeflich1/jupyter-base-notebook/jupyter-base-notebook.sif
+    JUPYTERJSC_USER_CMD="singularity exec ${SINGULARITY_IMAGE} jupyterhub-singleuser --config ${JUPYTER_LOG_DIR}/.config.py"
+%% Cell type:markdown id: tags:
+And copy the `start_jupyter-jsc.sh` script to the filesystem location expected by Jupyter-JSC,
+%% Cell type:code id: tags:
+``` bash
+cp start_jupyter-jsc.sh $HOME/.jupyter/start_jupyter-jsc.sh
+```
+%% Cell type:markdown id: tags:
+Finally, opening a new Jupyter session via the Jupyter-JSC control panel should now load the containerized Jupyter server that was setup here.
--- a/001-Jupyter/004-Create_JupyterKernel_container/start_jupyter-jsc.sh
+++ b/001-Jupyter/004-Create_JupyterKernel_container/start_jupyter-jsc.sh
+#!/bin/bash
+# Author: Katharina Höflich
+# Repository: https://github.com/FZJ-JSC/jupyter-jsc-notebooks
+SINGULARITY_IMAGE=/p/project/cesmtst/hoeflich1/jupyter-base-notebook/jupyter-base-notebook.sif
+JUPYTERJSC_USER_CMD="singularity exec ${SINGULARITY_IMAGE} jupyterhub-singleuser --config ${JUPYTER_LOG_DIR}/.config.py"
--- a/README.md
+++ b/README.md
 # Jupyter4JSC
+_This is a dual repo, which is tracked in https://github.com/FZJ-JSC/jupyter-jsc-notebooks and in https://gitlab.version.fz-juelich.de/jupyter4jsc/j4j_notebooks. If you have suggestions or bug reports, feel free to reach out via either repository.__
 Selected Jupyter Notebooks from
 https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks