update conda howto

5e8708b4 · Jens Henrik Göbbert · 6230249d · 5e8708b4
Commit 5e8708b4 authored 2 years ago by Jens Henrik Göbbert
--- a/03-HowTos/Create_JupyterKernel_conda.ipynb
+++ b/03-HowTos/Create_JupyterKernel_conda.ipynb
@@ -56,7 +56,8 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "Selectable **CONDA_TARGET_DIR** path for the central conda installation, should be in the project filesystem"
+    "Selectable **CONDA_ENV** name, will be used to specify the environment name\n",
+    "  - must be lowercase"
   ]
  },
  {
@@ -65,15 +66,17 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "export CONDA_TARGET_DIR=${HOME}/PROJECT_training2005/testdir/miniconda3"
+    "CONDA_ENV=my_condaenv\n",
+    "\n",
+    "export CONDA_ENV=$(echo \"${CONDA_ENV}\" | awk '{print tolower($0)}')\n",
+    "echo ${CONDA_ENV} # double check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "Selectable **CONDA_ENV** name, will be used to specify the environment name\n",
-    "  - must be lowercase"
+    "Selectable **CONDA_TARGET_DIR** path for the central conda installation, should be in the project filesystem"
   ]
  },
  {
@@ -82,10 +85,7 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "CONDA_ENV=my_env\n",
-    "\n",
-    "export CONDA_ENV=$(echo \"${CONDA_ENV}\" | awk '{print tolower($0)}')\n",
-    "echo ${CONDA_ENV} # double check"
+    "export CONDA_TARGET_DIR=${PROJECT}/${USER}/miniconda3/${CONDA_ENV}"
   ]
  },
  {
@@ -133,7 +133,8 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "mkdir -p ${CONDA_TARGET_DIR}"
+    "mkdir -p ${CONDA_TARGET_DIR}\n",
+    "echo ${CONDA_TARGET_DIR}"
   ]
  },
  {
@@ -158,7 +159,7 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "${CONDA_TARGET_DIR}/bin/conda init bash"
+    "${CONDA_TARGET_DIR}/bin/conda init --no-rc bash"
   ]
  },
  {
@@ -198,7 +199,7 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "${CONDA_TARGET_DIR}/bin/conda create -n ${CONDA_ENV} -y python=3.6.8 ipykernel"
+    "${CONDA_TARGET_DIR}/bin/conda create -n ${CONDA_ENV} -y python=3.10.4 ipykernel"
   ]
  },
  {
@@ -230,7 +231,7 @@
    "source '\"${CONDA_TARGET_DIR}\"'/bin/activate '\"${CONDA_ENV}\"'\n",
    "    \n",
    "# Ensure python packages installed in conda are always prefered, not necessary if module purge is used\n",
-    "export PYTHONPATH=${CONDA_PREFIX}/lib/python3.6/site-packages:${PYTHONPATH}\n",
+    "export PYTHONPATH=${CONDA_PREFIX}/lib/python3.10/site-packages:${PYTHONPATH}\n",
    "    \n",
    "exec python -m ipykernel $@' > ${CONDA_TARGET_DIR}/envs/${CONDA_ENV}/kernel.sh"
   ]
@@ -241,7 +242,8 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "chmod +x ${CONDA_TARGET_DIR}/envs/${CONDA_ENV}/kernel.sh"
+    "chmod +x ${CONDA_TARGET_DIR}/envs/${CONDA_ENV}/kernel.sh\n",
+    "echo ${CONDA_TARGET_DIR}/envs/${CONDA_ENV}/kernel.sh"
   ]
  },
  {
@@ -288,7 +290,10 @@
    "  \"{connection_file}\"\n",
    " ],\n",
    " \"display_name\": \"conda_'\"${CONDA_ENV}\"'\",\n",
-    " \"language\": \"python\"\n",
+    " \"language\": \"python\",\n",
+    " \"metadata\": {\n",
+    "   \"debugger\": true\n",
+    "  }\n",
    "}' > $HOME/.local/share/jupyter/kernels/conda_${CONDA_ENV}/kernel.json"
   ]
  },

 %% Cell type:markdown id: tags:

 ![header.png](attachment:364f4d26-8fd6-45ed-a6c3-1ab27dad25c4.png)
 <h5 style="text-align: right">Author: <a href="mailto:s.luehrs@fz-juelich.de?subject=Jupyter-JSC%20documentation">Sebastian Lührs</a></h5>
 <h5><a href="../index.ipynb">Index</a></h5>
 <h1 style="text-align: center">Create your own Jupyter CONDA-Kernel</h1>

 %% Cell type:markdown id: tags:

 Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed.
 With your own kernel you can overcome that problem easily and define your own environment, in which you work.

 This notebook shows you how you can build your own kernel for a **conda environment**.

 --------------------------------------

 %% Cell type:markdown id: tags:

 ## Building your own Jupyter CONDA-kernel is a three step process
 Download Minconda installer
 1. Download/Install Miniconda
   * Miniconda3.sh
 2. Create Conda Environment
   * conda create
 2. Create/Edit launch script for the Jupyter kernel
   * kernel.sh
 3. Create/Edit Jupyter kernel configuration
   * kernel.json

 %% Cell type:markdown id: tags:

 ### Settings

 %% Cell type:markdown id: tags:

-Selectable **CONDA_TARGET_DIR** path for the central conda installation, should be in the project filesystem
+Selectable **CONDA_ENV** name, will be used to specify the environment name
+  - must be lowercase

 %% Cell type:code id: tags:

 ``` bash
-export CONDA_TARGET_DIR=${HOME}/PROJECT_training2005/testdir/miniconda3
+CONDA_ENV=my_condaenv
+
+export CONDA_ENV=$(echo "${CONDA_ENV}" | awk '{print tolower($0)}')
+echo ${CONDA_ENV} # double check
 ```

 %% Cell type:markdown id: tags:

-Selectable **CONDA_ENV** name, will be used to specify the environment name
-  - must be lowercase
+Selectable **CONDA_TARGET_DIR** path for the central conda installation, should be in the project filesystem

 %% Cell type:code id: tags:

 ``` bash
-CONDA_ENV=my_env
-
-export CONDA_ENV=$(echo "${CONDA_ENV}" | awk '{print tolower($0)}')
-echo ${CONDA_ENV} # double check
+export CONDA_TARGET_DIR=${PROJECT}/${USER}/miniconda3/${CONDA_ENV}
 ```

 %% Cell type:markdown id: tags:

 ---
 ## 1. Download/Install Miniconda

 %% Cell type:markdown id: tags:

 Start here if you want to run the full installation.
 If you want to create another environment in an existing conda setup go to **create environment**. If you want to attach yourself to an existing environment go to **create user kernel**.

 %% Cell type:markdown id: tags:

 * 1.1 - Download Minconda installer

 %% Cell type:code id: tags:

 ``` bash
 wget --output-document=$HOME/Miniconda3.sh https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
 ```

 %% Cell type:markdown id: tags:

 * 1.2 - Create target directory

 %% Cell type:code id: tags:

 ``` bash
 mkdir -p ${CONDA_TARGET_DIR}
+echo ${CONDA_TARGET_DIR}
 ```

 %% Cell type:markdown id: tags:

 * 1.3 - Install Miniconda

 %% Cell type:code id: tags:

 ``` bash
 bash $HOME/Miniconda3.sh -b -u -p ${CONDA_TARGET_DIR}
 ```

 %% Cell type:code id: tags:

 ``` bash
-${CONDA_TARGET_DIR}/bin/conda init bash
+${CONDA_TARGET_DIR}/bin/conda init --no-rc bash
 ```

 %% Cell type:markdown id: tags:

 * 1.4 - Disable automatic activation

 %% Cell type:code id: tags:

 ``` bash
 ${CONDA_TARGET_DIR}/bin/conda config --set auto_activate_base false
 ```

 %% Cell type:markdown id: tags:

 ---
 ## 2. Create conda environment

 %% Cell type:markdown id: tags:

 Create new conda environment. The following steps can be repeated if multiple environments should be created. If the Python version differ towards the external Python version, a mix of Conda modules and external modules will not be possible

 %% Cell type:code id: tags:

 ``` bash
-${CONDA_TARGET_DIR}/bin/conda create -n ${CONDA_ENV} -y python=3.6.8 ipykernel
+${CONDA_TARGET_DIR}/bin/conda create -n ${CONDA_ENV} -y python=3.10.4 ipykernel
 ```

 %% Cell type:markdown id: tags:

 ---
 ## 3. Create/Edit launch script for the Jupyter kernel

 %% Cell type:markdown id: tags:

 * 3.1 - Create kernel to allow access to the conda environment. Adapte `module purge` and `PYTHONPATH` according to the comments.

 %% Cell type:code id: tags:

 ``` bash
 echo '#!/bin/bash

 # module purge # optional to disable the external environment, necessary, if python version is different

 # Activate your Python virtual environment
 source '"${CONDA_TARGET_DIR}"'/bin/activate '"${CONDA_ENV}"'

 # Ensure python packages installed in conda are always prefered, not necessary if module purge is used
-export PYTHONPATH=${CONDA_PREFIX}/lib/python3.6/site-packages:${PYTHONPATH}
+export PYTHONPATH=${CONDA_PREFIX}/lib/python3.10/site-packages:${PYTHONPATH}

 exec python -m ipykernel $@' > ${CONDA_TARGET_DIR}/envs/${CONDA_ENV}/kernel.sh
 ```

 %% Cell type:code id: tags:

 ``` bash
 chmod +x ${CONDA_TARGET_DIR}/envs/${CONDA_ENV}/kernel.sh
+echo ${CONDA_TARGET_DIR}/envs/${CONDA_ENV}/kernel.sh
 ```

 %% Cell type:markdown id: tags:

 ---
 ## 4. Create/Edit Jupyter kernel configuration

 %% Cell type:markdown id: tags:

 * 4.1 - Create user kernel, if you want to access the conda environment of a colleague, only these steps are necessary

 %% Cell type:code id: tags:

 ``` bash
 mkdir -p $HOME/.local/share/jupyter/kernels/conda_${CONDA_ENV}
 ```

 %% Cell type:markdown id: tags:

 * 4.2 - Adjust kernel.json file

 %% Cell type:code id: tags:

 ``` bash
 echo '{
 "argv": [
  "'"${CONDA_TARGET_DIR}"'/envs/'"${CONDA_ENV}"'/kernel.sh",
  "-f",
  "{connection_file}"
 ],
 "display_name": "conda_'"${CONDA_ENV}"'",
- "language": "python"
+ "language": "python",
+ "metadata": {
+   "debugger": true
+  }
 }' > $HOME/.local/share/jupyter/kernels/conda_${CONDA_ENV}/kernel.json
 ```

 %% Cell type:markdown id: tags:

 Restart of JupyterLab might be necessary to see the kernel in the kernel selection overview.