Skip to content
Snippets Groups Projects
Commit ed7c8607 authored by Jens Henrik Göbbert's avatar Jens Henrik Göbbert
Browse files

update kernel-build

parent 55cc79f0
No related branches found
No related tags found
No related merge requests found
%% Cell type:markdown id: tags:
# - Create your own Jupyter Kernel -
%% Cell type:markdown id: tags:
---
## Building your own Jupyter kernel is a three step process
1. Create/Pimp new virtual Python environment
* venv
2. Create/Edit launch script for the Jupyter kernel
* kernel.sh
3. Create/Edit Jupyter kernel configuration
* kernel.json
%% Cell type:markdown id: tags:
### Settings
%% Cell type:markdown id: tags:
* Set new kernel name
* Set kernel name
- change if you like
%% Cell type:code id: tags:
``` bash
# INPUT NEEDED:
export KERNEL_NAME=${USER}_kernel
echo ${KERNEL_NAME}
echo ${KERNEL_NAME} # double check
```
%% Cell type:markdown id: tags:
* Set directory for kernels virtual environment
- change if you like
* Set kernel type
- private kernel = "\${HOME}/.local/"
- project kernel = "\${PROJECT}/.local/"
%% Cell type:code id: tags:
``` bash
export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels/
mkdir -p ${KERNEL_VENVS_DIR}
echo ${KERNEL_VENVS_DIR}
# INPUT NEEDED:
export KERNEL_TYPE=private # private or project
###################
# private kernel
if [ "${KERNEL_TYPE}" == "private" ]; then
export KERNEL_SPECS_PREFIX=${HOME}/.local
echo "private kernel"
# project kernel
else
export KERNEL_SPECS_PREFIX=${PROJECT}/.local
echo "project kernel"
fi
export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels
# check if kernel name is unique
if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then
echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}"
echo " Rename kernel name or remove directory."
fi
echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME} # double check
```
%% Cell type:markdown id: tags:
* Set location of kernel spec
- select one:
- personal kernel = "\${HOME}/.local/"
- project kernel = "\${PROJECT}/.local/"
* Set directory for kernels virtual environment
- change if you like
%% Cell type:code id: tags:
``` bash
export KERNEL_SPECS_DIR=${HOME}/.local/
#export KERNEL_SPECS_DIR=${PROJECT}/.local/
echo ${KERNEL_SPECS_DIR}
# INPUT NEEDED:
export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels
###################
mkdir -p ${KERNEL_VENVS_DIR}
if [ "${KERNEL_TYPE}" != "private" ]; then
echo "Please check the permissions and ensure your project partners have read/execute permissions:"
namei -l ${KERNEL_VENVS_DIR}
fi
echo ${KERNEL_VENVS_DIR} # double check
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 1. Create/Pimp new virual Python environment
%% Cell type:markdown id: tags:
* 1.1 - Load required modules
%% Cell type:code id: tags:
``` bash
module -q purge
module -q use $OTHERSTAGES
module -q load Stages/Devel-2019a 2> /dev/null
module -q load Stages/Devel-2019a 2> /dev/null # any stage can be used
module -q load GCCcore/.8.3.0 2> /dev/null
module -q load Jupyter
module -q load Python/3.6.8 # only Python is required
module list # double check
```
%% Cell type:markdown id: tags:
* 1.2 - Load extra modules you need for your kernel
%% Cell type:code id: tags:
``` bash
# module load <module you need>
```
%% Cell type:markdown id: tags:
* 1.3 - Create and activate a virtual environment for the kernel
and ensure python packages installed in the virtual environment are always prefered
%% Cell type:code id: tags:
``` bash
python -m venv --system-site-packages ${KERNEL_VENVS_DIR}/${KERNEL_NAME}
source ${KERNEL_VENVS_DIR}/${KERNEL_NAME}/bin/activate
export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.6/site-packages:${PYTHONPATH}
echo ${VIRTUAL_ENV}
echo ${VIRTUAL_ENV} # double check
```
%% Cell type:markdown id: tags:
* 1.4 - Install Python libraries required for communication with Jupyter
%% Cell type:code id: tags:
``` bash
which pip
pip install --ignore-installed ipykernel
ls ${VIRTUAL_ENV}/lib/python3.6/site-packages/ # double check
```
%% Cell type:markdown id: tags:
* 1.5 - Install whatever else you need in your Python virtual environment (using pip)
%% Cell type:code id: tags:
``` bash
#pip install <python-package you need>
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 2. Create/Edit launch script for the Jupyter kernel
%% Cell type:markdown id: tags:
* 2.1 - Create launch script, which loads your Python virtual environment and starts the ipykernel process inside:
%% Cell type:code id: tags:
``` bash
echo '#!/bin/bash
# Load required modules
module purge
module load $OTHERSTAGES'"
module load Stages/Devel-2019a
module load GCCcore/.8.3.0
module load Jupyter
module load Python/3.6.8
# Load extra modules you need for your kernel (as you did in step 1.2)
#module load <module you need>
# Activate your Python virtual environment
source ${KERNEL_VENVS_DIR}/${KERNEL_NAME}/bin/activate
# Ensure python packages installed in the virtual environment are always prefered
export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.6/site-packages:"'${PYTHONPATH}'"
exec python -m ipykernel "'$@' > ${VIRTUAL_ENV}/kernel.sh
chmod +x ${VIRTUAL_ENV}/kernel.sh
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 3. Create/Edit Jupyter kernel configuration
%% Cell type:markdown id: tags:
* 3.1 - Create Jupyter kernel configuration directory and files
%% Cell type:code id: tags:
``` bash
python -m ipykernel install --name=${KERNEL_NAME} --prefix ${KERNEL_SPECS_DIR}
python -m ipykernel install --name=${KERNEL_NAME} --prefix ${VIRTUAL_ENV}
export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels
```
%% Cell type:markdown id: tags:
* 3.2 - Adjust kernel.json file
%% Cell type:code id: tags:
``` bash
mv ${KERNEL_SPECS_DIR}/share/jupyter/kernels/${KERNEL_NAME}/kernel.json ${KERNEL_SPECS_DIR}/share/jupyter/kernels/${KERNEL_NAME}/kernel.json.orig
mv ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json.orig
echo '
{
echo '{
"argv": [
"'${KERNEL_VENVS_DIR}/${KERNEL_NAME}/kernel.sh'",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "'${KERNEL_NAME}'",
"language": "python"
}' > ${KERNEL_SPECS_DIR}/share/jupyter/kernels/${KERNEL_NAME}/kernel.json
}' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json
```
%% Cell type:markdown id: tags:
* 3.3 - Create link to kernel specs
%% Cell type:code id: tags:
``` bash
cd ${KERNEL_SPECS_DIR}
ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .
ls ${KERNEL_SPECS_DIR} # double check
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 4. Cleanup
%% Cell type:code id: tags:
``` bash
deactivate
```
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment