Commit 8d2555eb authored by Filipe Guimaraes's avatar Filipe Guimaraes
Browse files

pyenv kernel tutorial: Removed any mention to changes in .bash_profile;...

pyenv kernel tutorial: Removed any mention to changes in .bash_profile; Changed folder to ${PROJECT}/${USER}.
parent e88c853e
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
![jsc-logo.jpg](attachment:500ad0ee-8923-43bb-9336-e838a26dc2f1.jpg) ![jsc-logo.jpg](attachment:500ad0ee-8923-43bb-9336-e838a26dc2f1.jpg)
Author: [Filipe Guimaraes](mailto:f.guimaraes@fz-juelich.de) Author: [Filipe Guimaraes](mailto:f.guimaraes@fz-juelich.de)
-------------------------------------- --------------------------------------
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Create your own Jupyter pyenv-Kernel # Create your own Jupyter pyenv-Kernel
Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed. Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed.
With your own kernel you can overcome that problem easily and define your own environment, in which you work. With your own kernel you can overcome that problem easily and define your own environment, in which you work.
This notebook shows you how you can build your own kernel for a **pyenv environment**. This notebook shows you how you can build your own kernel for a **pyenv environment**.
-------------------------------------- --------------------------------------
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Building your own Jupyter pyenv-kernel is a four-step process ## Building your own Jupyter pyenv-kernel is a four-step process
1. **[Download/Install pyenv](#install)**: To start from scratch, and run the full installation. 1. **[Download/Install pyenv](#install)**: To start from scratch, and run the full installation.
2. **[Create and setup environment](#environment)**: To create an(other) environment in an existing pyenv setup. 2. **[Create and setup environment](#environment)**: To create an(other) environment in an existing pyenv setup.
If `pyenv` is already installed, start here.
3. **[Create/Edit launch script for the Jupyter kernel](#kernel)**: To setup an environment to be run via Jupyter. 3. **[Create/Edit launch script for the Jupyter kernel](#kernel)**: To setup an environment to be run via Jupyter.
If the environment already exists, start here.
4. **[Create/Edit Jupyter kernel configuration](#json)**: To attach your user to an existing environment via Jupyter. 4. **[Create/Edit Jupyter kernel configuration](#json)**: To attach your user to an existing environment via Jupyter.
If the kernel launch script was already created (e.g., by some other user in the project), start here.
%% Cell type:markdown id: tags:
If `pyenv` is already installed, start at 2.
If the environment already exists, start at 3.
If the kernel launch script was already created (e.g., by some other user in the project), start at 4.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
<a id='settings'></a> <a id='settings'></a>
### Settings ### Settings
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
To simplifly the process, it is convenient to define a **PYENV_ROOT** path for the central pyenv installation and put on the PATH. To simplifly the process, it is convenient to define a **PYENV_ROOT** path for the central pyenv installation and put on the PATH.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
**Important**: It is recommended to use a folder inside the `$PROJECT` file system, as the `$HOME` quota is low. It is also useful to share installation for different users in a single project. **Important**: It is recommended to use a folder inside the `$PROJECT` file system, as the `$HOME` quota is low. It is also useful to share installation for different users in a single project.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
export PYENV_ROOT=${PROJECT_<projectid>}/.pyenv export PYENV_ROOT=${PROJECT_<projectid>}/${USER}/.pyenv
export PATH="$PYENV_ROOT/bin:$PATH" export PATH="$PYENV_ROOT/bin:$PATH"
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Also the environment name can be set in an environment variable **PYENV_ENV** to simplify the process: Also the environment name can be set in an environment variable **PYENV_ENV** to simplify the process:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
export PYENV_ENV=my_env export PYENV_ENV=my_env
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
These lines can be added to `~/.bash_profile` (and then `source ~/.bash_profile`) to automatically set the PATH in case of restarting the shell.
%% Cell type:markdown id: tags:
--- ---
<a id='install'></a> <a id='install'></a>
## 1. Download/Install pyenv ## 1. Download/Install pyenv
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Installing and setting up pyenv from scratch is very simple. A simple command is needed to install pyenv in **$PYENV_ROOT**: Installing and setting up pyenv from scratch is very simple. A simple command is needed to install pyenv in **$PYENV_ROOT**:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
curl https://pyenv.run | bash curl https://pyenv.run | bash
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
<a id='environment'></a> <a id='environment'></a>
## 2. Create and setup pyenv environment ## 2. Create and setup pyenv environment
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The following steps describe how to create and setup a new pyenv environment to be used as a jupyter kernel. They can be repeated if multiple environments are required. The following steps describe how to create and setup a new pyenv environment to be used as a jupyter kernel. They can be repeated if multiple environments are required.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 2.1 - Activate pyenv and virtual envs * 2.1 - Activate pyenv and virtual envs
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Since it is necessary to activate the environment to install the required modules, the following commands should be added to `~/.bash_profile` to activate pyenv and the virtual environments: Activate pyenv and the virtual environments by running:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
eval "$(pyenv init --path)" eval "$(pyenv init --path)"
eval "$(pyenv init -)" eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)" eval "$(pyenv virtualenv-init -)"
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
And activate the lines
%% Cell type:code id: tags:
``` bash
source ~/.bash_profile
```
%% Cell type:markdown id: tags:
* 2.2 - Install a python version (e.g. `3.10.1`) * 2.2 - Install a python version (e.g. `3.10.1`)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
pyenv install 3.10.1 pyenv install 3.10.1
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Note: This step may take a few minutes to complete the installation. Note: This step may take a few minutes to complete the installation.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 2.3 - Create a new environment **$PYENV_ENV** * 2.3 - Create a new environment **$PYENV_ENV**
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The environment is created using the python version installed above The environment is created using the python version installed above
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
pyenv virtualenv 3.10.1 $PYENV_ENV pyenv virtualenv 3.10.1 $PYENV_ENV
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 2.4 - Activate and setup the environment * 2.4 - Activate and setup the environment
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Jupyter requires the `ipykernel` module and its dependencies. To install them, first activate the environment: Jupyter requires the `ipykernel` module and its dependencies. To install them, first activate the environment:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
pyenv activate $PYENV_ENV pyenv activate $PYENV_ENV
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
When the environment is successfully activated, the name of the environment is shown between parenthesis in the command line, e.g. `(my_env)`. (To deactivate the environment, use `pyenv deactivate $PYENV_ENV`.) When the environment is successfully activated, the name of the environment is shown between parenthesis in the command line, e.g. `(my_env)`. (To deactivate the environment, use `pyenv deactivate $PYENV_ENV`.)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The python version can be checked using The python version can be checked using
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
(my_env)$ python --version (my_env)$ python --version
Python 3.10.1 Python 3.10.1
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The list of python modules is still empty The list of python modules is still empty
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
(my_env)$ pip list (my_env)$ pip list
Package Version Package Version
---------- ------- ---------- -------
pip 21.2.4 pip 21.2.4
setuptools 58.1.0 setuptools 58.1.0
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
To create a Jupyter kernel, the `ipykernel` and its dependencies are required. `pip` can be used to install it: To create a Jupyter kernel, the `ipykernel` and its dependencies are required. `pip` can be used to install it:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
pip install ipykernel pip install ipykernel
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Many modules are installed: Many modules are installed:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
(my_env)$ pip list (my_env)$ pip list
Package Version Package Version
----------------- ------- ----------------- -------
backcall 0.2.0 backcall 0.2.0
debugpy 1.5.1 debugpy 1.5.1
decorator 5.1.1 decorator 5.1.1
entrypoints 0.3 entrypoints 0.3
ipykernel 6.6.1 ipykernel 6.6.1
ipython 7.31.0 ipython 7.31.0
jedi 0.18.1 jedi 0.18.1
jupyter-client 7.1.0 jupyter-client 7.1.0
jupyter-core 4.9.1 jupyter-core 4.9.1
matplotlib-inline 0.1.3 matplotlib-inline 0.1.3
nest-asyncio 1.5.4 nest-asyncio 1.5.4
parso 0.8.3 parso 0.8.3
pexpect 4.8.0 pexpect 4.8.0
pickleshare 0.7.5 pickleshare 0.7.5
pip 21.1.1 pip 21.1.1
prompt-toolkit 3.0.24 prompt-toolkit 3.0.24
ptyprocess 0.7.0 ptyprocess 0.7.0
Pygments 2.11.2 Pygments 2.11.2
python-dateutil 2.8.2 python-dateutil 2.8.2
pyzmq 22.3.0 pyzmq 22.3.0
setuptools 56.0.0 setuptools 56.0.0
six 1.16.0 six 1.16.0
tornado 6.1 tornado 6.1
traitlets 5.1.1 traitlets 5.1.1
wcwidth 0.2.5 wcwidth 0.2.5
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
<a id='kernel'></a> <a id='kernel'></a>
## 3. Create/Edit launch script for the Jupyter kernel ## 3. Create/Edit launch script for the Jupyter kernel
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The following steps describe how to create and configure the launch script of a new Jupyter kernel using a pyenv environment. If the environment was created in the $PROJECT folder, many users of the project can follow these steps to create the kernel. The steps assume the variables described in the **[Settings section](#settings)** are set up. Here, they don't need to be set in `~/.bash_profile` or `~/.bashrc`. The following steps describe how to create and configure the launch script of a new Jupyter kernel using a pyenv environment. If the environment was created in the $PROJECT folder, many users of the project can follow these steps to create the kernel. The steps assume the variables described in the **[Settings section](#settings)** are set up.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
<a id='launch'></a> <a id='launch'></a>
* 3.1 - Create kernel script to allow access to the pyenv environment * 3.1 - Create kernel script to allow access to the pyenv environment
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
echo '#!/bin/bash echo '#!/bin/bash
module purge module purge
export PYENV_ROOT='"$PYENV_ROOT"' export PYENV_ROOT='"$PYENV_ROOT"'
export PATH='"$PYENV_ROOT"'/bin:'"$PATH"' export PATH='"$PYENV_ROOT"'/bin:'"$PATH"'
eval "$(pyenv init --path)" eval "$(pyenv init --path)"
eval "$(pyenv init -)" eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)" eval "$(pyenv virtualenv-init -)"
# Activate your Python virtual environment # Activate your Python virtual environment
pyenv activate '"${PYENV_ENV}"' pyenv activate '"${PYENV_ENV}"'
exec python -m ipykernel $@' > ${PYENV_ROOT}/versions/${PYENV_ENV}/kernel.sh exec python -m ipykernel $@' > ${PYENV_ROOT}/versions/${PYENV_ENV}/kernel.sh
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Note: the `export` and `eval` lines are repeated here as they are not required to be in `~/.bash_profile` on these steps.
%% Cell type:markdown id: tags:
Add executable permission to the script: Add executable permission to the script:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
chmod +x ${PYENV_ROOT}/versions/${PYENV_ENV}/kernel.sh chmod +x ${PYENV_ROOT}/versions/${PYENV_ENV}/kernel.sh
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
## 4. Create/Edit Jupyter kernel configuration ## 4. Create/Edit Jupyter kernel configuration
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
These steps describe how to create a Jupyter kernel configuration file, to be able to access the environment via a Jupyter notebook. To access an existing pyenv environment located in **$PROJECT**, only these steps are necessary. The steps assume the variables described in the **[Settings section](#settings)** are set up. These steps describe how to create a Jupyter kernel configuration file, to be able to access the environment via a Jupyter notebook. To access an existing pyenv environment located in **$PROJECT**, only these steps are necessary. The steps assume the variables described in the **[Settings section](#settings)** are set up.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 4.1 - Create a folder for the kernel * 4.1 - Create a folder for the kernel
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
mkdir -p $HOME/.local/share/jupyter/kernels/pyenv_${PYENV_ENV} mkdir -p $HOME/.local/share/jupyter/kernels/pyenv_${PYENV_ENV}
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
* 4.2 - Create and adjust the kernel.json file * 4.2 - Create and adjust the kernel.json file
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` bash ``` bash
echo '{ echo '{
"argv": [ "argv": [
"'"${PYENV_ROOT}"'/versions/'"${PYENV_ENV}"'/kernel.sh", "'"${PYENV_ROOT}"'/versions/'"${PYENV_ENV}"'/kernel.sh",
"-f", "-f",
"{connection_file}" "{connection_file}"
], ],
"display_name": "pyenv_'"${PYENV_ENV}"'", "display_name": "pyenv_'"${PYENV_ENV}"'",
"language": "python" "language": "python"
}' > $HOME/.local/share/jupyter/kernels/pyenv_${PYENV_ENV}/kernel.json }' > $HOME/.local/share/jupyter/kernels/pyenv_${PYENV_ENV}/kernel.json
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Restart of JupyterLab might be necessary to see the kernel in the kernel selection overview. Restart of JupyterLab might be necessary to see the kernel in the kernel selection overview.
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment