update to software stage 2025

b0930251 · Jens Henrik Goebbert · 62da1da7 · b0930251
Commit b0930251 authored 1 month ago by Jens Henrik Goebbert
--- a/03-HowTos/Create_JupyterKernel_general.ipynb
+++ b/03-HowTos/Create_JupyterKernel_general.ipynb
@@ -72,7 +72,7 @@
   "outputs": [],
   "source": [
    "# INPUT NEEDED:\n",
-    "KERNEL_NAME=${USER}_kernel\n",
+    "KERNEL_NAME=${USER}_kernel_2025new2\n",
    "\n",
    "export KERNEL_NAME=$(echo \"${KERNEL_NAME}\" | awk '{print tolower($0)}')\n",
    "echo ${KERNEL_NAME} # double check"
@@ -161,9 +161,10 @@
   "outputs": [],
   "source": [
    "module purge\n",
-    "module load Stages/2024 # any stage can be used\n",
+    "module load Stages/2025 # any stage can be used\n",
    "module load GCC\n",
-    "module load Python      # only Python is mandatory"
+    "module load Python      # only Python is mandatory\n",
+    "module load jupyter-server # provides ipykernel"
   ]
  },
  {
@@ -223,41 +224,24 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "#### 1.4 - Install Python libraries required for communication with Jupyter"
+    "#### 1.4 - Install whatever else you need in your Python virtual environment (using pip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
-   "metadata": {
-    "scrolled": true
-   },
+   "metadata": {},
   "outputs": [],
   "source": [
-    "which pip\n",
    "if [ -z \"${VIRTUAL_ENV}\" ]; then\n",
    "  echo \"ERROR: Virtual environment not successfully initialized.\"\n",
    "else\n",
-    "  pip install ipykernel\n",
+    "  echo \"Installing custom Python packages using pip from the virtual environment:\"\n",
+    "  which pip\n",
+    "  # pip install <python-package you need>\n",
    "fi"
   ]
  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "#### 1.5 - Install whatever else you need in your Python virtual environment (using pip)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#pip install <python-package you need>"
-   ]
-  },
  {
   "cell_type": "markdown",
   "metadata": {},
@@ -294,12 +278,13 @@
    "\n",
    "# Load basic Python module\n",
    "module purge\n",
-    "module load Stages/2024\n",
+    "module load Stages/2025\n",
    "module load GCC\n",
    "module load Python\n",
+    "module load jupyter-server  # provides ipykernel\n",
    "\n",
    "# Load extra modules you need for your kernel (as you did in step 1.2)\n",
-    "#module load <module you need>\n",
+    "# module load <module you need>\n",
    "\n",
    "# Activate your Python virtual environment\n",
    "source ${VIRTUAL_ENV}/bin/activate\n",
@@ -307,7 +292,7 @@
    "# Ensure python packages installed in the virtual environment are always prefered\n",
    "export PYTHONPATH=${VIRTUAL_ENV}/lib/python${PYV}/site-packages:\"'${PYTHONPATH}'\"\n",
    "    \n",
-    "exec python -m ipykernel \"'$@' > ${VIRTUAL_ENV}/kernel.sh\n",
+    "exec python -Xfrozen_modules=off -m ipykernel \"'$@' > ${VIRTUAL_ENV}/kernel.sh\n",
    "chmod +x ${VIRTUAL_ENV}/kernel.sh\n",
    "\n",
    "cat ${VIRTUAL_ENV}/kernel.sh # double check"

 %% Cell type:markdown id: tags:

 ![header.png](attachment:9f53dcb1-00d6-4245-955a-b527f1540865.png)
 <h5 style="text-align: right">Author: <a href="mailto:j.goebbert@fz-juelich.de?subject=Jupyter-JSC%20documentation">Jens Henrik Göbbert</a></h5>
 <h5><a href="../index.ipynb">Index</a></h5>
 <h1 style="text-align: center">Create your own Jupyter Kernel</h1>

 %% Cell type:markdown id: tags:

 Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed.
 With your own kernel you can overcome that problem easily and define your own environment, in which you work.

 This notebook shows you how you can build your own kernel for a **python environment**.

 <div class="alert alert-block alert-info">
 <b>Attention:</b>
 This notebook is meant to run out of a JupyterLab on JSC's HPC systems.</br>
 </div>

 -------------------------

 %% Cell type:markdown id: tags:

 ## Building your own Jupyter kernel is a three step process
 1. Create/Pimp new virtual Python environment
   * venv
 2. Create/Edit launch script for the Jupyter kernel
   * kernel.sh
 3. Create/Edit Jupyter kernel configuration
   * kernel.json

 %% Cell type:markdown id: tags:

 ### Settings

 %% Cell type:markdown id: tags:

 #### Set the kernel name
  - must be lower case
  - change if you like

 %% Cell type:code id: tags:

 ``` bash
 # INPUT NEEDED:
-KERNEL_NAME=${USER}_kernel
+KERNEL_NAME=${USER}_kernel_2025new2

 export KERNEL_NAME=$(echo "${KERNEL_NAME}" | awk '{print tolower($0)}')
 echo ${KERNEL_NAME} # double check
 ```

 %% Cell type:markdown id: tags:

 #### Set the kernel directory
  - check that the kernel name is unique
  - print the location of the new kernel

 %% Cell type:code id: tags:

 ``` bash
 # define KERNEL_SPECS_DIR
 export KERNEL_SPECS_PREFIX=${HOME}/.local
 if [ ! -d "$KERNEL_SPECS_PREFIX" ]; then
  echo "ERROR: please create directory $KERNEL_SPECS_PREFIX"
 fi
 export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels

 # check if kernel name is unique
 if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then
  echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}"
  echo "       Rename kernel name or remove directory."
 fi

 # print the location of the new kernel
 echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME}
 ```

 %% Cell type:markdown id: tags:

 #### Set the kernel's virtual environment
  - by default it is located at $PROJECT
  - print the location of the new kernels virtual environment

 %% Cell type:code id: tags:

 ``` bash
 # define KERNEL_VENVS_DIR
 export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels
 mkdir -p ${KERNEL_VENVS_DIR}

 # print the location of the new kernels virtual environment
 echo ${KERNEL_VENVS_DIR}
 ```

 %% Cell type:markdown id: tags:

 ---

 %% Cell type:markdown id: tags:

 ## 1. Create/Pimp new virtual Python environment

 %% Cell type:markdown id: tags:

 #### 1.1 - Load basic Python module

 %% Cell type:code id: tags:

 ``` bash
 module purge
-module load Stages/2024 # any stage can be used
+module load Stages/2025 # any stage can be used
 module load GCC
 module load Python      # only Python is mandatory
+module load jupyter-server # provides ipykernel
 ```

 %% Cell type:code id: tags:

 ``` bash
 # get Python version
 export PYV=$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')
 echo $PYV
 ```

 %% Cell type:markdown id: tags:

 #### 1.2 - Load extra modules you need for your kernel

 %% Cell type:code id: tags:

 ``` bash
 # module load <module you need>
 ```

 %% Cell type:markdown id: tags:

 #### 1.3 - Create and activate a virtual environment for the kernel
 and ensure python packages installed in the virtual environment are always prefered

 %% Cell type:code id: tags:

 ``` bash
 export VIRTUAL_ENV=${KERNEL_VENVS_DIR}/${KERNEL_NAME}
 if [ -d "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Directory for virtual environment already ${VIRTUAL_ENV}"
  echo "       Rename kernel name or remove directory."
 else
  python -m venv --system-site-packages ${VIRTUAL_ENV}
  source ${VIRTUAL_ENV}/bin/activate
  export PYTHONPATH=${VIRTUAL_ENV}/lib/python${PYV}/site-packages:${PYTHONPATH}
  echo ${VIRTUAL_ENV} # double check
 fi
 ```

 %% Cell type:markdown id: tags:

-#### 1.4 - Install Python libraries required for communication with Jupyter
+#### 1.4 - Install whatever else you need in your Python virtual environment (using pip)

 %% Cell type:code id: tags:

 ``` bash
-which pip
 if [ -z "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Virtual environment not successfully initialized."
 else
-  pip install ipykernel
+  echo "Installing custom Python packages using pip from the virtual environment:"
+  which pip
+  # pip install <python-package you need>
 fi
 ```

 %% Cell type:markdown id: tags:

-#### 1.5 - Install whatever else you need in your Python virtual environment (using pip)
-
-%% Cell type:code id: tags:
-
-``` bash
-#pip install <python-package you need>
-```
-
-%% Cell type:markdown id: tags:
-
 ---

 %% Cell type:markdown id: tags:

 ## 2. Create/Edit launch script for the Jupyter kernel

 %% Cell type:markdown id: tags:

 #### 2.1 - Create launch script, which loads your Python virtual environment and starts the ipykernel process inside:

 <div class="alert alert-block alert-info">
 <b>Attention:</b>
 You MUST load the exactly the same modules as you did above for your virtual Python environment.
 </div>

 %% Cell type:code id: tags:

 ``` bash
 echo '#!/bin/bash'"

 # Load basic Python module
 module purge
-module load Stages/2024
+module load Stages/2025
 module load GCC
 module load Python
+module load jupyter-server  # provides ipykernel

 # Load extra modules you need for your kernel (as you did in step 1.2)
-#module load <module you need>
+# module load <module you need>

 # Activate your Python virtual environment
 source ${VIRTUAL_ENV}/bin/activate

 # Ensure python packages installed in the virtual environment are always prefered
 export PYTHONPATH=${VIRTUAL_ENV}/lib/python${PYV}/site-packages:"'${PYTHONPATH}'"

-exec python -m ipykernel "'$@' > ${VIRTUAL_ENV}/kernel.sh
+exec python -Xfrozen_modules=off -m ipykernel "'$@' > ${VIRTUAL_ENV}/kernel.sh
 chmod +x ${VIRTUAL_ENV}/kernel.sh

 cat ${VIRTUAL_ENV}/kernel.sh # double check
 ```

 %% Cell type:markdown id: tags:

 ---

 %% Cell type:markdown id: tags:

 ## 3. Create/Edit Jupyter kernel configuration

 %% Cell type:markdown id: tags:

 #### 3.1 - Create Jupyter kernel configuration directory and files

 %% Cell type:code id: tags:

 ``` bash
 python -m ipykernel install --name=${KERNEL_NAME} --prefix ${VIRTUAL_ENV}
 export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels
 ```

 %% Cell type:markdown id: tags:

 #### 3.2 - Adjust kernel.json file

 %% Cell type:code id: tags:

 ``` bash
 mv ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json.orig

 echo '{
  "argv": [
    "'${KERNEL_VENVS_DIR}/${KERNEL_NAME}/kernel.sh'",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ],
  "display_name": "'${KERNEL_NAME}'",
  "language": "python",
  "metadata": {
   "debugger": true
  }
 }' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json

 cat ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json # double check
 ```

 %% Cell type:markdown id: tags:

 #### 3.3 - Create link to kernel specs

 %% Cell type:code id: tags:

 ``` bash
 mkdir -p ${KERNEL_SPECS_DIR}
 cd ${KERNEL_SPECS_DIR}
 ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .

 echo -e "\n\nThe new kernel '${KERNEL_NAME}' was added to your kernels in '${KERNEL_SPECS_DIR}/'\n"
 ls ${KERNEL_SPECS_DIR} # double check
 ```

 %% Cell type:markdown id: tags:

 #### 3.4 - Use the kernel
 - You can select the new kernel in the top right corner of your notebook or from JupyterLab's Launchpad
 - The kernel icon will be added to your launcher, after a while by JupyterLab automatically or once you've restarted the JupyterLab

 %% Cell type:markdown id: tags:

 ---

 %% Cell type:markdown id: tags:

 ## 4. Cleanup

 %% Cell type:code id: tags:

 ``` bash
 deactivate
 ```