Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
J
j4j_notebooks
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container registry
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
GitLab community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Tim Kreuzer
j4j_notebooks
Commits
ed7c8607
Commit
ed7c8607
authored
5 years ago
by
Jens Henrik Göbbert
Browse files
Options
Downloads
Patches
Plain Diff
update kernel-build
parent
55cc79f0
No related branches found
No related tags found
No related merge requests found
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
Create_JupyterKernel.ipynb
+89
-24
89 additions, 24 deletions
Create_JupyterKernel.ipynb
with
89 additions
and
24 deletions
Create_JupyterKernel.ipynb
+
89
−
24
View file @
ed7c8607
...
...
@@ -32,7 +32,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"* Set
new
kernel name\n",
"* Set kernel name\n",
" - change if you like"
]
},
...
...
@@ -42,16 +42,19 @@
"metadata": {},
"outputs": [],
"source": [
"# INPUT NEEDED:\n",
"export KERNEL_NAME=${USER}_kernel\n",
"echo ${KERNEL_NAME}"
"\n",
"echo ${KERNEL_NAME} # double check"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Set directory for kernels virtual environment\n",
" - change if you like"
"* Set kernel type\n",
" - private kernel = \"\\${HOME}/.local/\" \n",
" - project kernel = \"\\${PROJECT}/.local/\" "
]
},
{
...
...
@@ -60,19 +63,36 @@
"metadata": {},
"outputs": [],
"source": [
"export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels/\n",
"mkdir -p ${KERNEL_VENVS_DIR}\n",
"echo ${KERNEL_VENVS_DIR}"
"# INPUT NEEDED:\n",
"export KERNEL_TYPE=private # private or project\n",
"\n",
"###################\n",
"# private kernel\n",
"if [ \"${KERNEL_TYPE}\" == \"private\" ]; then\n",
" export KERNEL_SPECS_PREFIX=${HOME}/.local\n",
" echo \"private kernel\"\n",
"# project kernel\n",
"else\n",
" export KERNEL_SPECS_PREFIX=${PROJECT}/.local\n",
" echo \"project kernel\"\n",
"fi\n",
"export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels\n",
"\n",
"# check if kernel name is unique\n",
"if [ -d \"${KERNEL_SPECS_DIR}/${KERNEL_NAME}\" ]; then\n",
" echo \"ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}\"\n",
" echo \" Rename kernel name or remove directory.\"\n",
"fi\n",
"\n",
"echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME} # double check"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Set location of kernel spec\n",
" - select one:\n",
" - personal kernel = \"\\${HOME}/.local/\" \n",
" - project kernel = \"\\${PROJECT}/.local/\" "
"* Set directory for kernels virtual environment\n",
" - change if you like"
]
},
{
...
...
@@ -81,9 +101,17 @@
"metadata": {},
"outputs": [],
"source": [
"export KERNEL_SPECS_DIR=${HOME}/.local/\n",
"#export KERNEL_SPECS_DIR=${PROJECT}/.local/\n",
"echo ${KERNEL_SPECS_DIR}"
"# INPUT NEEDED:\n",
"export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels\n",
"\n",
"###################\n",
"mkdir -p ${KERNEL_VENVS_DIR}\n",
"if [ \"${KERNEL_TYPE}\" != \"private\" ]; then\n",
" echo \"Please check the permissions and ensure your project partners have read/execute permissions:\"\n",
" namei -l ${KERNEL_VENVS_DIR}\n",
"fi\n",
"\n",
"echo ${KERNEL_VENVS_DIR} # double check"
]
},
{
...
...
@@ -115,9 +143,10 @@
"source": [
"module -q purge\n",
"module -q use $OTHERSTAGES \n",
"module -q load Stages/Devel-2019a 2> /dev/null\n",
"module -q load Stages/Devel-2019a 2> /dev/null
# any stage can be used
\n",
"module -q load GCCcore/.8.3.0 2> /dev/null\n",
"module -q load Jupyter"
"module -q load Python/3.6.8 # only Python is required\n",
"module list # double check"
]
},
{
...
...
@@ -153,7 +182,7 @@
"python -m venv --system-site-packages ${KERNEL_VENVS_DIR}/${KERNEL_NAME}\n",
"source ${KERNEL_VENVS_DIR}/${KERNEL_NAME}/bin/activate\n",
"export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.6/site-packages:${PYTHONPATH}\n",
"echo ${VIRTUAL_ENV}"
"echo ${VIRTUAL_ENV}
# double check
"
]
},
{
...
...
@@ -169,7 +198,9 @@
"metadata": {},
"outputs": [],
"source": [
"pip install --ignore-installed ipykernel"
"which pip\n",
"pip install --ignore-installed ipykernel\n",
"ls ${VIRTUAL_ENV}/lib/python3.6/site-packages/ # double check"
]
},
{
...
...
@@ -222,7 +253,7 @@
"module load $OTHERSTAGES'\"\n",
"module load Stages/Devel-2019a\n",
"module load GCCcore/.8.3.0\n",
"module load
Jupyter
\n",
"module load
Python/3.6.8
\n",
"\n",
"# Load extra modules you need for your kernel (as you did in step 1.2)\n",
"#module load <module you need>\n",
...
...
@@ -264,7 +295,8 @@
"metadata": {},
"outputs": [],
"source": [
"python -m ipykernel install --name=${KERNEL_NAME} --prefix ${KERNEL_SPECS_DIR}"
"python -m ipykernel install --name=${KERNEL_NAME} --prefix ${VIRTUAL_ENV}\n",
"export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels"
]
},
{
...
...
@@ -280,10 +312,9 @@
"metadata": {},
"outputs": [],
"source": [
"mv ${
KERNEL_SPECS_DIR}/share/jupyter/kernels/${KERNEL_NAME}/kernel.json ${KERNEL_SPECS_DIR}/share/jupyter/kernels
/${KERNEL_NAME}/kernel.json.orig\n",
"mv ${
VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json ${VIRTUAL_ENV_KERNELS}
/${KERNEL_NAME}/kernel.json.orig\n",
"\n",
"echo '\n",
"{\n",
"echo '{\n",
" \"argv\": [\n",
" \"'${KERNEL_VENVS_DIR}/${KERNEL_NAME}/kernel.sh'\",\n",
" \"-m\",\n",
...
...
@@ -293,7 +324,25 @@
" ],\n",
" \"display_name\": \"'${KERNEL_NAME}'\",\n",
" \"language\": \"python\"\n",
"}' > ${KERNEL_SPECS_DIR}/share/jupyter/kernels/${KERNEL_NAME}/kernel.json"
"}' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* 3.3 - Create link to kernel specs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cd ${KERNEL_SPECS_DIR}\n",
"ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .\n",
"ls ${KERNEL_SPECS_DIR} # double check"
]
},
{
...
...
@@ -302,6 +351,22 @@
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Cleanup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"deactivate"
]
}
],
"metadata": {
...
...
%% Cell type:markdown id: tags:
# - Create your own Jupyter Kernel -
%% Cell type:markdown id: tags:
---
## Building your own Jupyter kernel is a three step process
1.
Create/Pimp new virtual Python environment
*
venv
2.
Create/Edit launch script for the Jupyter kernel
*
kernel.sh
3.
Create/Edit Jupyter kernel configuration
*
kernel.json
%% Cell type:markdown id: tags:
### Settings
%% Cell type:markdown id: tags:
*
Set
new
kernel name
*
Set kernel name
-
change if you like
%% Cell type:code id: tags:
```
bash
# INPUT NEEDED:
export
KERNEL_NAME
=
${
USER
}
_kernel
echo
${
KERNEL_NAME
}
echo
${
KERNEL_NAME
}
# double check
```
%% Cell type:markdown id: tags:
*
Set directory for kernels virtual environment
-
change if you like
*
Set kernel type
-
private kernel = "
\$
{HOME}/.local/"
-
project kernel = "
\$
{PROJECT}/.local/"
%% Cell type:code id: tags:
```
bash
export
KERNEL_VENVS_DIR
=
${
PROJECT
}
/
${
USER
}
/jupyter/kernels/
mkdir
-p
${
KERNEL_VENVS_DIR
}
echo
${
KERNEL_VENVS_DIR
}
# INPUT NEEDED:
export
KERNEL_TYPE
=
private
# private or project
###################
# private kernel
if
[
"
${
KERNEL_TYPE
}
"
==
"private"
]
;
then
export
KERNEL_SPECS_PREFIX
=
${
HOME
}
/.local
echo
"private kernel"
# project kernel
else
export
KERNEL_SPECS_PREFIX
=
${
PROJECT
}
/.local
echo
"project kernel"
fi
export
KERNEL_SPECS_DIR
=
${
KERNEL_SPECS_PREFIX
}
/share/jupyter/kernels
# check if kernel name is unique
if
[
-d
"
${
KERNEL_SPECS_DIR
}
/
${
KERNEL_NAME
}
"
]
;
then
echo
"ERROR: Kernel already exists in
${
KERNEL_SPECS_DIR
}
/
${
KERNEL_NAME
}
"
echo
" Rename kernel name or remove directory."
fi
echo
${
KERNEL_SPECS_DIR
}
/
${
KERNEL_NAME
}
# double check
```
%% Cell type:markdown id: tags:
*
Set location of kernel spec
-
select one:
-
personal kernel = "
\$
{HOME}/.local/"
-
project kernel = "
\$
{PROJECT}/.local/"
*
Set directory for kernels virtual environment
-
change if you like
%% Cell type:code id: tags:
```
bash
export
KERNEL_SPECS_DIR
=
${
HOME
}
/.local/
#export KERNEL_SPECS_DIR=${PROJECT}/.local/
echo
${
KERNEL_SPECS_DIR
}
# INPUT NEEDED:
export
KERNEL_VENVS_DIR
=
${
PROJECT
}
/
${
USER
}
/jupyter/kernels
###################
mkdir
-p
${
KERNEL_VENVS_DIR
}
if
[
"
${
KERNEL_TYPE
}
"
!=
"private"
]
;
then
echo
"Please check the permissions and ensure your project partners have read/execute permissions:"
namei
-l
${
KERNEL_VENVS_DIR
}
fi
echo
${
KERNEL_VENVS_DIR
}
# double check
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 1. Create/Pimp new virual Python environment
%% Cell type:markdown id: tags:
*
1.1 - Load required modules
%% Cell type:code id: tags:
```
bash
module
-q
purge
module
-q
use
$OTHERSTAGES
module
-q
load Stages/Devel-2019a 2> /dev/null
module
-q
load Stages/Devel-2019a 2> /dev/null
# any stage can be used
module
-q
load GCCcore/.8.3.0 2> /dev/null
module
-q
load Jupyter
module
-q
load Python/3.6.8
# only Python is required
module list
# double check
```
%% Cell type:markdown id: tags:
*
1.2 - Load extra modules you need for your kernel
%% Cell type:code id: tags:
```
bash
# module load <module you need>
```
%% Cell type:markdown id: tags:
*
1.3 - Create and activate a virtual environment for the kernel
and ensure python packages installed in the virtual environment are always prefered
%% Cell type:code id: tags:
```
bash
python
-m
venv
--system-site-packages
${
KERNEL_VENVS_DIR
}
/
${
KERNEL_NAME
}
source
${
KERNEL_VENVS_DIR
}
/
${
KERNEL_NAME
}
/bin/activate
export
PYTHONPATH
=
${
VIRTUAL_ENV
}
/lib/python3.6/site-packages:
${
PYTHONPATH
}
echo
${
VIRTUAL_ENV
}
echo
${
VIRTUAL_ENV
}
# double check
```
%% Cell type:markdown id: tags:
*
1.4 - Install Python libraries required for communication with Jupyter
%% Cell type:code id: tags:
```
bash
which pip
pip
install
--ignore-installed
ipykernel
ls
${
VIRTUAL_ENV
}
/lib/python3.6/site-packages/
# double check
```
%% Cell type:markdown id: tags:
*
1.5 - Install whatever else you need in your Python virtual environment (using pip)
%% Cell type:code id: tags:
```
bash
#pip install <python-package you need>
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 2. Create/Edit launch script for the Jupyter kernel
%% Cell type:markdown id: tags:
*
2.1 - Create launch script, which loads your Python virtual environment and starts the ipykernel process inside:
%% Cell type:code id: tags:
```
bash
echo
'#!/bin/bash
# Load required modules
module purge
module load $OTHERSTAGES'
"
module load Stages/Devel-2019a
module load GCCcore/.8.3.0
module load
Jupyter
module load
Python/3.6.8
# Load extra modules you need for your kernel (as you did in step 1.2)
#module load <module you need>
# Activate your Python virtual environment
source
${
KERNEL_VENVS_DIR
}
/
${
KERNEL_NAME
}
/bin/activate
# Ensure python packages installed in the virtual environment are always prefered
export PYTHONPATH=
${
VIRTUAL_ENV
}
/lib/python3.6/site-packages:"
'${PYTHONPATH}'
"
exec python -m ipykernel "
'$@'
>
${
VIRTUAL_ENV
}
/kernel.sh
chmod
+x
${
VIRTUAL_ENV
}
/kernel.sh
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 3. Create/Edit Jupyter kernel configuration
%% Cell type:markdown id: tags:
*
3.1 - Create Jupyter kernel configuration directory and files
%% Cell type:code id: tags:
```
bash
python
-m
ipykernel
install
--name
=
${
KERNEL_NAME
}
--prefix
${
KERNEL_SPECS_DIR
}
python
-m
ipykernel
install
--name
=
${
KERNEL_NAME
}
--prefix
${
VIRTUAL_ENV
}
export
VIRTUAL_ENV_KERNELS
=
${
VIRTUAL_ENV
}
/share/jupyter/kernels
```
%% Cell type:markdown id: tags:
*
3.2 - Adjust kernel.json file
%% Cell type:code id: tags:
```
bash
mv
${
KERNEL_SPECS_DIR
}
/share/jupyter/kernels/
${
KERNEL_NAME
}
/kernel.json
${
KERNEL_SPECS_DIR
}
/share/jupyter/kernels
/
${
KERNEL_NAME
}
/kernel.json.orig
mv
${
VIRTUAL_ENV_KERNELS
}
/
${
KERNEL_NAME
}
/kernel.json
${
VIRTUAL_ENV_KERNELS
}
/
${
KERNEL_NAME
}
/kernel.json.orig
echo
'
{
echo
'{
"argv": [
"'
${
KERNEL_VENVS_DIR
}
/
${
KERNEL_NAME
}
/kernel.sh
'",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "'
${
KERNEL_NAME
}
'",
"language": "python"
}'
>
${
KERNEL_SPECS_DIR
}
/share/jupyter/kernels/
${
KERNEL_NAME
}
/kernel.json
}'
>
${
VIRTUAL_ENV_KERNELS
}
/
${
KERNEL_NAME
}
/kernel.json
```
%% Cell type:markdown id: tags:
*
3.3 - Create link to kernel specs
%% Cell type:code id: tags:
```
bash
cd
${
KERNEL_SPECS_DIR
}
ln
-s
${
VIRTUAL_ENV_KERNELS
}
/
${
KERNEL_NAME
}
.
ls
${
KERNEL_SPECS_DIR
}
# double check
```
%% Cell type:markdown id: tags:
---
%% Cell type:markdown id: tags:
## 4. Cleanup
%% Cell type:code id: tags:
```
bash
deactivate
```
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment