Skip to content
Snippets Groups Projects
Commit 8e1e513e authored by Jens Henrik Göbbert's avatar Jens Henrik Göbbert
Browse files

update to Stage/2020

parent eddec700
No related branches found
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
001-Jupyter/Create_JupyterKernel_general.ipynb
+ 16
− 11
View file @ 8e1e513e
%% Cell type:markdown id: tags:
![jsc-logo.jpg](attachment:67258d94-84e6-4a0c-ae8f-c74332ec082e.jpg)
Author: [Jens Henrik Göbbert](mailto:j.goebbert@fz-juelich.de)
------------------------------------
%% Cell type:markdown id: tags:
# Create your own Jupyter Kernel
Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed.
With your own kernel you can overcome that problem easily and define your own environment, in which you work.
This notebook shows you how you can build your own kernel for a **python environment**.
-------------------------
%% Cell type:markdown id: tags:
## Building your own Jupyter kernel is a three step process
1. Create/Pimp new virtual Python environment
* venv
2. Create/Edit launch script for the Jupyter kernel
* kernel.sh
3. Create/Edit Jupyter kernel configuration
* kernel.json
%% Cell type:markdown id: tags:
### Settings
%% Cell type:markdown id: tags:
* Set kernel name
- must be lower case
- change if you like
%% Cell type:code id: tags:
``` bash
# INPUT NEEDED:
KERNEL_NAME=${USER}_kernel
export KERNEL_NAME=$(echo "${KERNEL_NAME}" | awk '{print tolower($0)}')
echo ${KERNEL_NAME} # double check
```
%% Cell type:markdown id: tags:
* List directories where JupyterLab will search for kernels
%% Cell type:code id: tags:
``` bash
# JUPYTER SEARCH PATH (for kernels-directory)
echo "jupyter search paths for kernels-directories"
if [ -z $JUPYTER_PATH ]; then
echo "$HOME/.local/share/jupyter"
else
tr ':' '\n' <<< "$JUPYTER_PATH"
fi
```
%% Cell type:markdown id: tags:
* Set kernel type
- private kernel = "\${HOME}/.local/"
- project kernel = "\${PROJECT}/.local/"
- other kernel = "\<your-path\>" (ensure it is part of $JUPYTER_PATH or your kernel will not be found by JuypterLab)
%% Cell type:code id: tags:
``` bash
# INPUT NEEDED:
export KERNEL_TYPE=private # private, project or other
export KERNEL_SPECS_PREFIX=/p/home/jusers/$USER/jureca/.local
export KERNEL_SPECS_PREFIX=${HOME}/.local
###################
# project kernel
if [ "${KERNEL_TYPE}" == "project" ]; then
export KERNEL_SPECS_PREFIX=${PROJECT}/.local
echo "project kernel"
# private kernel
elif [ "${KERNEL_TYPE}" == "private" ]; then
export KERNEL_SPECS_PREFIX=${HOME}/.local
echo "private kernel"
else
if [ ! -d "$KERNEL_SPECS_PREFIX" ]; then
echo "ERROR: please create directory $KERNEL_SPECS_PREFIX"
fi
echo "other kernel"
fi
export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels
# check if kernel name is unique
if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then
echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}"
echo " Rename kernel name or remove directory."
fi
echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME} # double check
```
%% Cell type:markdown id: tags:
* Set directory for kernels virtual environment
- change if you like
%% Cell type:code id: tags:
``` bash
# INPUT NEEDED:
export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels
###################
mkdir -p ${KERNEL_VENVS_DIR}
if [ "${KERNEL_TYPE}" != "private" ] && [ "${KERNEL_TYPE}" != "other" ]; then
echo "Please check the permissions and ensure your project partners have read/execute permissions:"
namei -l ${KERNEL_VENVS_DIR}
fi
echo ${KERNEL_VENVS_DIR} # double check
ls -lt ${KERNEL_VENVS_DIR}
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 1. Create/Pimp new virtual Python environment
%% Cell type:markdown id: tags:
* 1.1 - Load required modules
%% Cell type:code id: tags:
``` bash
module -q purge
module -q use $OTHERSTAGES
module -q load Stages/Devel-2019a 2> /dev/null # any stage can be used
module -q load GCCcore/.8.3.0 2> /dev/null
module -q load Python/3.6.8 # only Python is required
module -q load Stages/2020 2> /dev/null # any stage can be used
module -q load GCCcore/.9.3.0 2> /dev/null
module -q load Python/3.8.5 # only Python is required
module list # double check
```
%% Cell type:markdown id: tags:
* 1.2 - Load extra modules you need for your kernel
%% Cell type:code id: tags:
``` bash
# module load <module you need>
```
%% Cell type:markdown id: tags:
* 1.3 - Create and activate a virtual environment for the kernel
and ensure python packages installed in the virtual environment are always prefered
%% Cell type:code id: tags:
``` bash
if [ -d "${KERNEL_VENVS_DIR}/${KERNEL_NAME}" ]; then
echo "ERROR: Directory for virtual environment already ${KERNEL_VENVS_DIR}/${KERNEL_NAME}"
echo " Rename kernel name or remove directory."
else
python -m venv --system-site-packages ${KERNEL_VENVS_DIR}/${KERNEL_NAME}
source ${KERNEL_VENVS_DIR}/${KERNEL_NAME}/bin/activate
export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.6/site-packages:${PYTHONPATH}
export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.8/site-packages:${PYTHONPATH}
echo ${VIRTUAL_ENV} # double check
fi
```
%% Cell type:markdown id: tags:
* 1.4 - Install Python libraries required for communication with Jupyter
%% Cell type:code id: tags:
``` bash
which pip
pip install --ignore-installed ipykernel
ls ${VIRTUAL_ENV}/lib/python3.6/site-packages/ # double check
if [ -z "${VIRTUAL_ENV}" ]; then
echo "ERROR: Virtual environment not successfully initialized."
else
pip install --ignore-installed ipykernel
ls ${VIRTUAL_ENV}/lib/python3.8/site-packages/ # double check
fi
```
%% Cell type:markdown id: tags:
* 1.5 - Install whatever else you need in your Python virtual environment (using pip)
%% Cell type:code id: tags:
``` bash
#pip install <python-package you need>
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 2. Create/Edit launch script for the Jupyter kernel
%% Cell type:markdown id: tags:
* 2.1 - Create launch script, which loads your Python virtual environment and starts the ipykernel process inside:
%% Cell type:code id: tags:
``` bash
echo '#!/bin/bash'"
# Load required modules
module purge
module use "'$OTHERSTAGES'"
module load Stages/Devel-2019a
module load GCCcore/.8.3.0
module load Python/3.6.8
module load Stages/2020
module load GCCcore/.9.3.0
module load Python/3.8.5
# Load extra modules you need for your kernel (as you did in step 1.2)
#module load <module you need>
# Activate your Python virtual environment
source ${KERNEL_VENVS_DIR}/${KERNEL_NAME}/bin/activate
# Ensure python packages installed in the virtual environment are always prefered
export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.6/site-packages:"'${PYTHONPATH}'"
export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.8/site-packages:"'${PYTHONPATH}'"
exec python -m ipykernel "'$@' > ${VIRTUAL_ENV}/kernel.sh
chmod +x ${VIRTUAL_ENV}/kernel.sh
cat ${VIRTUAL_ENV}/kernel.sh # double check
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 3. Create/Edit Jupyter kernel configuration
%% Cell type:markdown id: tags:
* 3.1 - Create Jupyter kernel configuration directory and files
%% Cell type:code id: tags:
``` bash
python -m ipykernel install --name=${KERNEL_NAME} --prefix ${VIRTUAL_ENV}
export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels
```
%% Cell type:markdown id: tags:
* 3.2 - Adjust kernel.json file
%% Cell type:code id: tags:
``` bash
mv ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json.orig
echo '{
"argv": [
"'${KERNEL_VENVS_DIR}/${KERNEL_NAME}/kernel.sh'",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "'${KERNEL_NAME}'",
"language": "python"
}' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json
cat ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json # double check
```
%% Cell type:markdown id: tags:
* 3.3 - Create link to kernel specs
%% Cell type:code id: tags:
``` bash
cd ${KERNEL_SPECS_DIR}
ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .
echo -e "\n\nThe new kernel '${KERNEL_NAME}' was added to your kernels in '${KERNEL_SPECS_DIR}/'\n"
ls ${KERNEL_SPECS_DIR} # double check
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 4. Cleanup
%% Cell type:code id: tags:
``` bash
deactivate
```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment