"Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed.\n",
"With your own kernel you can overcome that problem easily and define your own environment, in which you work.\n",
"\n",
"This notebook shows you how you can build your own kernel for a **pyenv environment**.\n",
"\n",
"--------------------------------------"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Building your own Jupyter pyenv-kernel is a four-step process\n",
"\n",
"1. **[Download/Install pyenv](#install)**: To start from scratch, and run the full installation.\n",
"2. **[Create and setup environment](#environment)**: To create an(other) environment in an existing pyenv setup. \n",
"If `pyenv` is already installed, start here.\n",
"3. **[Create/Edit launch script for the Jupyter kernel](#kernel)**: To setup an environment to be run via Jupyter. \n",
"If the environment already exists, start here.\n",
"4. **[Create/Edit Jupyter kernel configuration](#json)**: To attach your user to an existing environment via Jupyter. \n",
"If the kernel launch script was already created (e.g., by some other user in the project), start here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='settings'></a>\n",
"### Settings"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To simplifly the process, it is convenient to define a **PYENV_ROOT** path for the central pyenv installation and put on the PATH. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Important**: It is recommended to use a folder inside the `$PROJECT` file system, as the `$HOME` quota is low. It is also useful to share installation for different users in a single project."
"Also the environment name can be set in an environment variable **PYENV_ENV** to simplify the process:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"export PYENV_ENV=my_env"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"<a id='install'></a>\n",
"## 1. Download/Install pyenv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Installing and setting up pyenv from scratch is very simple. A simple command is needed to install pyenv in **$PYENV_ROOT**:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"curl https://pyenv.run | bash"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"<a id='environment'></a>\n",
"## 2. Create and setup pyenv environment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following steps describe how to create and setup a new pyenv environment to be used as a jupyter kernel. They can be repeated if multiple environments are required."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For these steps, make sure to have a clean environment before starting this process. This can be obtained by running `module purge`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"$ module purge\n",
"$ module list\n",
"\n",
"Currently Loaded Modules:\n",
" 1) Stages/2020 (S)\n",
"\n",
" Where:\n",
" S: Module is Sticky, requires --force to unload or purge"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* 2.1 - Activate pyenv and virtual envs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Activate pyenv and the virtual environments by running:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"eval \"$(pyenv init --path)\"\n",
"eval \"$(pyenv init -)\"\n",
"eval \"$(pyenv virtualenv-init -)\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* 2.2 - Install a python version (e.g. `3.10.1`)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pyenv install 3.10.1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note: This step may take a few minutes to complete the installation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* 2.3 - Create a new environment **$PYENV_ENV** (defined in **[Settings section](#settings)**)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The environment is created using the python version installed above"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pyenv virtualenv 3.10.1 $PYENV_ENV"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* 2.4 - Activate and setup the environment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Jupyter requires the `ipykernel` module and its dependencies. To install them, first activate the environment:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pyenv activate $PYENV_ENV"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When the environment is successfully activated, the name of the environment is shown between parenthesis in the command line, e.g. `(my_env)`. (To deactivate the environment, use `pyenv deactivate $PYENV_ENV`.)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The python version can be checked using"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"(my_env)$ python --version\n",
"Python 3.10.1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The list of python modules is still empty"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"(my_env)$ pip list\n",
"Package Version\n",
"---------- -------\n",
"pip 21.2.4\n",
"setuptools 58.1.0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To create a Jupyter kernel, the `ipykernel` and its dependencies are required. `pip` can be used to install it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pip install ipykernel"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Many modules are installed:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"(my_env)$ pip list\n",
"Package Version\n",
"----------------- -------\n",
"backcall 0.2.0\n",
"debugpy 1.5.1\n",
"decorator 5.1.1\n",
"entrypoints 0.3\n",
"ipykernel 6.6.1\n",
"ipython 7.31.0\n",
"jedi 0.18.1\n",
"jupyter-client 7.1.0\n",
"jupyter-core 4.9.1\n",
"matplotlib-inline 0.1.3\n",
"nest-asyncio 1.5.4\n",
"parso 0.8.3\n",
"pexpect 4.8.0\n",
"pickleshare 0.7.5\n",
"pip 21.1.1\n",
"prompt-toolkit 3.0.24\n",
"ptyprocess 0.7.0\n",
"Pygments 2.11.2\n",
"python-dateutil 2.8.2\n",
"pyzmq 22.3.0\n",
"setuptools 56.0.0\n",
"six 1.16.0\n",
"tornado 6.1\n",
"traitlets 5.1.1\n",
"wcwidth 0.2.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"<a id='kernel'></a>\n",
"## 3. Create/Edit launch script for the Jupyter kernel"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following steps describe how to create and configure the launch script of a new Jupyter kernel using a pyenv environment. If the environment was created in the $PROJECT folder, many users of the project can follow these steps to create the kernel. The steps assume the variables described in the **[Settings section](#settings)** are set up."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='launch'></a>\n",
"* 3.1 - Create kernel script to allow access to the pyenv environment"
"These steps describe how to create a Jupyter kernel configuration file, to be able to access the environment via a Jupyter notebook. To access an existing pyenv environment located in **$PROJECT**, only these steps are necessary. The steps assume the variables described in the **[Settings section](#settings)** are set up."
Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed.
With your own kernel you can overcome that problem easily and define your own environment, in which you work.
This notebook shows you how you can build your own kernel for a **pyenv environment**.
--------------------------------------
%% Cell type:markdown id: tags:
## Building your own Jupyter pyenv-kernel is a four-step process
1.**[Download/Install pyenv](#install)**: To start from scratch, and run the full installation.
2.**[Create and setup environment](#environment)**: To create an(other) environment in an existing pyenv setup.
If `pyenv` is already installed, start here.
3.**[Create/Edit launch script for the Jupyter kernel](#kernel)**: To setup an environment to be run via Jupyter.
If the environment already exists, start here.
4.**[Create/Edit Jupyter kernel configuration](#json)**: To attach your user to an existing environment via Jupyter.
If the kernel launch script was already created (e.g., by some other user in the project), start here.
%% Cell type:markdown id: tags:
<aid='settings'></a>
### Settings
%% Cell type:markdown id: tags:
To simplifly the process, it is convenient to define a **PYENV_ROOT** path for the central pyenv installation and put on the PATH.
%% Cell type:markdown id: tags:
**Important**: It is recommended to use a folder inside the `$PROJECT` file system, as the `$HOME` quota is low. It is also useful to share installation for different users in a single project.
Also the environment name can be set in an environment variable **PYENV_ENV** to simplify the process:
%% Cell type:code id: tags:
``` bash
export PYENV_ENV=my_env
```
%% Cell type:markdown id: tags:
---
<aid='install'></a>
## 1. Download/Install pyenv
%% Cell type:markdown id: tags:
Installing and setting up pyenv from scratch is very simple. A simple command is needed to install pyenv in **$PYENV_ROOT**:
%% Cell type:code id: tags:
``` bash
curl https://pyenv.run | bash
```
%% Cell type:markdown id: tags:
---
<aid='environment'></a>
## 2. Create and setup pyenv environment
%% Cell type:markdown id: tags:
The following steps describe how to create and setup a new pyenv environment to be used as a jupyter kernel. They can be repeated if multiple environments are required.
%% Cell type:markdown id: tags:
For these steps, make sure to have a clean environment before starting this process. This can be obtained by running `module purge`:
%% Cell type:code id: tags:
``` bash
$ module purge
$ module list
Currently Loaded Modules:
1) Stages/2020 (S)
Where:
S: Module is Sticky, requires --force to unload or purge
```
%% Cell type:markdown id: tags:
* 2.1 - Activate pyenv and virtual envs
%% Cell type:markdown id: tags:
Activate pyenv and the virtual environments by running:
%% Cell type:code id: tags:
``` bash
eval"$(pyenv init --path)"
eval"$(pyenv init -)"
eval"$(pyenv virtualenv-init -)"
```
%% Cell type:markdown id: tags:
* 2.2 - Install a python version (e.g. `3.10.1`)
%% Cell type:code id: tags:
``` bash
pyenv install 3.10.1
```
%% Cell type:markdown id: tags:
Note: This step may take a few minutes to complete the installation.
%% Cell type:markdown id: tags:
* 2.3 - Create a new environment **$PYENV_ENV** (defined in **[Settings section](#settings)**)
%% Cell type:markdown id: tags:
The environment is created using the python version installed above
%% Cell type:code id: tags:
``` bash
pyenv virtualenv 3.10.1 $PYENV_ENV
```
%% Cell type:markdown id: tags:
* 2.4 - Activate and setup the environment
%% Cell type:markdown id: tags:
Jupyter requires the `ipykernel` module and its dependencies. To install them, first activate the environment:
%% Cell type:code id: tags:
``` bash
pyenv activate $PYENV_ENV
```
%% Cell type:markdown id: tags:
When the environment is successfully activated, the name of the environment is shown between parenthesis in the command line, e.g. `(my_env)`. (To deactivate the environment, use `pyenv deactivate $PYENV_ENV`.)
%% Cell type:markdown id: tags:
The python version can be checked using
%% Cell type:code id: tags:
``` bash
(my_env)$ python --version
Python 3.10.1
```
%% Cell type:markdown id: tags:
The list of python modules is still empty
%% Cell type:code id: tags:
``` bash
(my_env)$ pip list
Package Version
-----------------
pip 21.2.4
setuptools 58.1.0
```
%% Cell type:markdown id: tags:
To create a Jupyter kernel, the `ipykernel` and its dependencies are required. `pip` can be used to install it:
%% Cell type:code id: tags:
``` bash
pip install ipykernel
```
%% Cell type:markdown id: tags:
Many modules are installed:
%% Cell type:code id: tags:
``` bash
(my_env)$ pip list
Package Version
------------------------
backcall 0.2.0
debugpy 1.5.1
decorator 5.1.1
entrypoints 0.3
ipykernel 6.6.1
ipython 7.31.0
jedi 0.18.1
jupyter-client 7.1.0
jupyter-core 4.9.1
matplotlib-inline 0.1.3
nest-asyncio 1.5.4
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
pip 21.1.1
prompt-toolkit 3.0.24
ptyprocess 0.7.0
Pygments 2.11.2
python-dateutil 2.8.2
pyzmq 22.3.0
setuptools 56.0.0
six 1.16.0
tornado 6.1
traitlets 5.1.1
wcwidth 0.2.5
```
%% Cell type:markdown id: tags:
---
<aid='kernel'></a>
## 3. Create/Edit launch script for the Jupyter kernel
%% Cell type:markdown id: tags:
The following steps describe how to create and configure the launch script of a new Jupyter kernel using a pyenv environment. If the environment was created in the $PROJECT folder, many users of the project can follow these steps to create the kernel. The steps assume the variables described in the **[Settings section](#settings)** are set up.
%% Cell type:markdown id: tags:
<aid='launch'></a>
* 3.1 - Create kernel script to allow access to the pyenv environment
These steps describe how to create a Jupyter kernel configuration file, to be able to access the environment via a Jupyter notebook. To access an existing pyenv environment located in **$PROJECT**, only these steps are necessary. The steps assume the variables described in the **[Settings section](#settings)** are set up.
