Skip to content
Snippets Groups Projects

Pyenv kernel tutorial

Merged Pyenv kernel tutorial
guimaraes1/j4j_notebooks:master into master
Merged
Filipe Guimaraesrequested to merge
guimaraes1/j4j_notebooks:master into master
Compare
and
1 file
+ 303
− 0
Compare changes
  • Side-by-side
  • Inline
001-Jupyter/Create_JupyterKernel_pyenv.ipynb 0 → 100644
+ 303
− 0
%% Cell type:markdown id: tags:
![jsc-logo.jpg](attachment:500ad0ee-8923-43bb-9336-e838a26dc2f1.jpg)
Author: [Filipe Guimaraes](mailto:f.guimaraes@fz-juelich.de)
--------------------------------------
%% Cell type:markdown id: tags:
# Create your own Jupyter pyenv-Kernel
Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed.
With your own kernel you can overcome that problem easily and define your own environment, in which you work.
This notebook shows you how you can build your own kernel for a **pyenv environment**.
--------------------------------------
%% Cell type:markdown id: tags:
## Building your own Jupyter pyenv-kernel is a four-step process
1. **[Download/Install pyenv](#install)**: To start from scratch, and run the full installation.
2. **[Create and setup environment](#environment)**: To create an(other) environment in an existing pyenv setup.
If `pyenv` is already installed, start here.
3. **[Create/Edit launch script for the Jupyter kernel](#kernel)**: To setup an environment to be run via Jupyter.
If the environment already exists, start here.
4. **[Create/Edit Jupyter kernel configuration](#json)**: To attach your user to an existing environment via Jupyter.
If the kernel launch script was already created (e.g., by some other user in the project), start here.
%% Cell type:markdown id: tags:
<a id='settings'></a>
### Settings
%% Cell type:markdown id: tags:
To simplifly the process, it is convenient to define a **PYENV_ROOT** path for the central pyenv installation and put on the PATH.
%% Cell type:markdown id: tags:
**Important**: It is recommended to use a folder inside the `$PROJECT` file system, as the `$HOME` quota is low. It is also useful to share installation for different users in a single project.
%% Cell type:code id: tags:
``` bash
export PYENV_ROOT=${PROJECT_<projectid>}/${USER}/.pyenv
export PATH="$PYENV_ROOT/bin:$PATH"
```
%% Cell type:markdown id: tags:
Also the environment name can be set in an environment variable **PYENV_ENV** to simplify the process:
%% Cell type:code id: tags:
``` bash
export PYENV_ENV=my_env
```
%% Cell type:markdown id: tags:
---
<a id='install'></a>
## 1. Download/Install pyenv
%% Cell type:markdown id: tags:
Installing and setting up pyenv from scratch is very simple. A simple command is needed to install pyenv in **$PYENV_ROOT**:
%% Cell type:code id: tags:
``` bash
curl https://pyenv.run | bash
```
%% Cell type:markdown id: tags:
---
<a id='environment'></a>
## 2. Create and setup pyenv environment
%% Cell type:markdown id: tags:
The following steps describe how to create and setup a new pyenv environment to be used as a jupyter kernel. They can be repeated if multiple environments are required.
%% Cell type:markdown id: tags:
* 2.1 - Activate pyenv and virtual envs
%% Cell type:markdown id: tags:
Activate pyenv and the virtual environments by running:
%% Cell type:code id: tags:
``` bash
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
```
%% Cell type:markdown id: tags:
* 2.2 - Install a python version (e.g. `3.10.1`)
%% Cell type:code id: tags:
``` bash
pyenv install 3.10.1
```
%% Cell type:markdown id: tags:
Note: This step may take a few minutes to complete the installation.
%% Cell type:markdown id: tags:
* 2.3 - Create a new environment **$PYENV_ENV**
%% Cell type:markdown id: tags:
The environment is created using the python version installed above
%% Cell type:code id: tags:
``` bash
pyenv virtualenv 3.10.1 $PYENV_ENV
```
%% Cell type:markdown id: tags:
* 2.4 - Activate and setup the environment
%% Cell type:markdown id: tags:
Jupyter requires the `ipykernel` module and its dependencies. To install them, first activate the environment:
%% Cell type:code id: tags:
``` bash
pyenv activate $PYENV_ENV
```
%% Cell type:markdown id: tags:
When the environment is successfully activated, the name of the environment is shown between parenthesis in the command line, e.g. `(my_env)`. (To deactivate the environment, use `pyenv deactivate $PYENV_ENV`.)
%% Cell type:markdown id: tags:
The python version can be checked using
%% Cell type:code id: tags:
``` bash
(my_env)$ python --version
Python 3.10.1
```
%% Cell type:markdown id: tags:
The list of python modules is still empty
%% Cell type:code id: tags:
``` bash
(my_env)$ pip list
Package Version
---------- -------
pip 21.2.4
setuptools 58.1.0
```
%% Cell type:markdown id: tags:
To create a Jupyter kernel, the `ipykernel` and its dependencies are required. `pip` can be used to install it:
%% Cell type:code id: tags:
``` bash
pip install ipykernel
```
%% Cell type:markdown id: tags:
Many modules are installed:
%% Cell type:code id: tags:
``` bash
(my_env)$ pip list
Package Version
----------------- -------
backcall 0.2.0
debugpy 1.5.1
decorator 5.1.1
entrypoints 0.3
ipykernel 6.6.1
ipython 7.31.0
jedi 0.18.1
jupyter-client 7.1.0
jupyter-core 4.9.1
matplotlib-inline 0.1.3
nest-asyncio 1.5.4
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
pip 21.1.1
prompt-toolkit 3.0.24
ptyprocess 0.7.0
Pygments 2.11.2
python-dateutil 2.8.2
pyzmq 22.3.0
setuptools 56.0.0
six 1.16.0
tornado 6.1
traitlets 5.1.1
wcwidth 0.2.5
```
%% Cell type:markdown id: tags:
---
<a id='kernel'></a>
## 3. Create/Edit launch script for the Jupyter kernel
%% Cell type:markdown id: tags:
The following steps describe how to create and configure the launch script of a new Jupyter kernel using a pyenv environment. If the environment was created in the $PROJECT folder, many users of the project can follow these steps to create the kernel. The steps assume the variables described in the **[Settings section](#settings)** are set up.
%% Cell type:markdown id: tags:
<a id='launch'></a>
* 3.1 - Create kernel script to allow access to the pyenv environment
%% Cell type:code id: tags:
``` bash
echo '#!/bin/bash
module purge
export PYENV_ROOT='"$PYENV_ROOT"'
export PATH='"$PYENV_ROOT"'/bin:'"$PATH"'
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
# Activate your Python virtual environment
pyenv activate '"${PYENV_ENV}"'
exec python -m ipykernel $@' > ${PYENV_ROOT}/versions/${PYENV_ENV}/kernel.sh
```
%% Cell type:markdown id: tags:
Add executable permission to the script:
%% Cell type:code id: tags:
``` bash
chmod +x ${PYENV_ROOT}/versions/${PYENV_ENV}/kernel.sh
```
%% Cell type:markdown id: tags:
---
## 4. Create/Edit Jupyter kernel configuration
%% Cell type:markdown id: tags:
These steps describe how to create a Jupyter kernel configuration file, to be able to access the environment via a Jupyter notebook. To access an existing pyenv environment located in **$PROJECT**, only these steps are necessary. The steps assume the variables described in the **[Settings section](#settings)** are set up.
%% Cell type:markdown id: tags:
* 4.1 - Create a folder for the kernel
%% Cell type:code id: tags:
``` bash
mkdir -p $HOME/.local/share/jupyter/kernels/pyenv_${PYENV_ENV}
```
%% Cell type:markdown id: tags:
* 4.2 - Create and adjust the kernel.json file
%% Cell type:code id: tags:
``` bash
echo '{
"argv": [
"'"${PYENV_ROOT}"'/versions/'"${PYENV_ENV}"'/kernel.sh",
"-f",
"{connection_file}"
],
"display_name": "pyenv_'"${PYENV_ENV}"'",
"language": "python"
}' > $HOME/.local/share/jupyter/kernels/pyenv_${PYENV_ENV}/kernel.json
```
%% Cell type:markdown id: tags:
Restart of JupyterLab might be necessary to see the kernel in the kernel selection overview.