ensure lowercase kernelnames

d1855ea6 · Jens Henrik Goebbert · 969fa097 · d1855ea6 · d1855ea6
Commit d1855ea6 authored 5 years ago by Jens Henrik Goebbert
--- a/001-Jupyter/Create_JupyterKernel_conda.ipynb
+++ b/001-Jupyter/Create_JupyterKernel_conda.ipynb
@@ -51,7 +51,8 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "Selectable **CONDA_ENV** name, will be used to specify the environment name"
+    "Selectable **CONDA_ENV** name, will be used to specify the environment name\n",
+    "  - must be lowercase"
   ]
  },
  {
@@ -60,7 +61,10 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "export CONDA_ENV=my_env"
+    "CONDA_ENV=my_env\n",
+    "\n",
+    "export CONDA_ENV=$(echo \"${CONDA_ENV}\" | awk '{print tolower($0)}')\n",
+    "echo ${KERNEL_NAME} # double check"
   ]
  },
  {

 %% Cell type:markdown id: tags:

 # Create your own Jupyter CONDA-Kernel

 %% Cell type:markdown id: tags:

 ---
 ## Building your own Jupyter CONDA-kernel is a three step process
 Download Minconda installer
 1. Download/Install Miniconda
   * Miniconda3.sh
 2. Create Conda Environment
   * conda create
 2. Create/Edit launch script for the Jupyter kernel
   * kernel.sh
 3. Create/Edit Jupyter kernel configuration
   * kernel.json

 %% Cell type:markdown id: tags:

 ### Settings

 %% Cell type:markdown id: tags:

 Selectable **CONDA_TARGET_DIR** path for the central conda installation, should be in the project filesystem

 %% Cell type:code id: tags:

 ``` bash
 export CONDA_TARGET_DIR=${HOME}/PROJECT_training2005/testdir/miniconda3
 ```

 %% Cell type:markdown id: tags:

 Selectable **CONDA_ENV** name, will be used to specify the environment name
+  - must be lowercase

 %% Cell type:code id: tags:

 ``` bash
-export CONDA_ENV=my_env
+CONDA_ENV=my_env
+
+export CONDA_ENV=$(echo "${CONDA_ENV}" | awk '{print tolower($0)}')
+echo ${KERNEL_NAME} # double check
 ```

 %% Cell type:markdown id: tags:

 ---
 ## 1. Download/Install Miniconda

 %% Cell type:markdown id: tags:

 Start here if you want to run the full installation.
 If you want to create another environment in an existing conda setup go to **create environment**. If you want to attach yourself to an existing environment go to **create user kernel**.

 %% Cell type:markdown id: tags:

 * 1.1 - Download Minconda installer

 %% Cell type:code id: tags:

 ``` bash
 curl https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -o $HOME/Miniconda3.sh
 ```

 %% Cell type:markdown id: tags:

 * 1.2 - Create target directory

 %% Cell type:code id: tags:

 ``` bash
 mkdir -p ${CONDA_TARGET_DIR}
 ```

 %% Cell type:markdown id: tags:

 * 1.3 - Install Miniconda

 %% Cell type:code id: tags:

 ``` bash
 bash $HOME/Miniconda3.sh -b -u -p ${CONDA_TARGET_DIR}
 ```

 %% Cell type:code id: tags:

 ``` bash
 ${CONDA_TARGET_DIR}/bin/conda init bash
 ```

 %% Cell type:markdown id: tags:

 * 1.4 - Disable automatic activation

 %% Cell type:code id: tags:

 ``` bash
 ${CONDA_TARGET_DIR}/bin/conda config --set auto_activate_base false
 ```

 %% Cell type:markdown id: tags:

 ---
 ## 2. Create conda environment

 %% Cell type:markdown id: tags:

 Create new conda environment. The following steps can be repeated if multiple environments should be created. If the Python version differ towards the external Python version, a mix of Conda modules and external modules will not be possible

 %% Cell type:code id: tags:

 ``` bash
 ${CONDA_TARGET_DIR}/bin/conda create -n ${CONDA_ENV} -y python=3.6.8 ipykernel
 ```

 %% Cell type:markdown id: tags:

 ---
 ## 3. Create/Edit launch script for the Jupyter kernel

 %% Cell type:markdown id: tags:

 * 3.1 - Create kernel to allow access to the conda environment. Adapte `module purge` and `PYTHONPATH` according to the comments.

 %% Cell type:code id: tags:

 ``` bash
 echo '#!/bin/bash

 # module purge # optional to disable the external environment, necessary, if python version is different

 # Activate your Python virtual environment
 source '"${CONDA_TARGET_DIR}"'/bin/activate '"${CONDA_ENV}"'

 # Ensure python packages installed in conda are always prefered, not necessary if module purge is used
 export PYTHONPATH=${CONDA_PREFIX}/lib/python3.6/site-packages:${PYTHONPATH}

 exec python -m ipykernel $@' > ${CONDA_TARGET_DIR}/envs/${CONDA_ENV}/kernel.sh
 ```

 %% Cell type:code id: tags:

 ``` bash
 chmod +x ${CONDA_TARGET_DIR}/envs/${CONDA_ENV}/kernel.sh
 ```

 %% Cell type:markdown id: tags:

 ---
 ## 4. Create/Edit Jupyter kernel configuration

 %% Cell type:markdown id: tags:

 * 4.1 - Create user kernel, if you want to access the conda environment of a colleague, only these steps are necessary

 %% Cell type:code id: tags:

 ``` bash
 mkdir -p $HOME/.local/share/jupyter/kernels/conda_${CONDA_ENV}
 ```

 %% Cell type:markdown id: tags:

 * 4.2 - Adjust kernel.json file

 %% Cell type:code id: tags:

 ``` bash
 echo '{
 "argv": [
  "'"${CONDA_TARGET_DIR}"'/envs/'"${CONDA_ENV}"'/kernel.sh",
  "-f",
  "{connection_file}"
 ],
 "display_name": "conda_'"${CONDA_ENV}"'",
 "language": "python"
 }' > $HOME/.local/share/jupyter/kernels/conda_${CONDA_ENV}/kernel.json
 ```

 %% Cell type:markdown id: tags:

 Restart of JupyterLab might be necessary to see the kernel in the kernel selection overview.

--- a/001-Jupyter/Create_JupyterKernel_general.ipynb
+++ b/001-Jupyter/Create_JupyterKernel_general.ipynb
@@ -33,18 +33,28 @@
   "metadata": {},
   "source": [
    "* Set kernel name\n",
+    "  - must be lower case\n",
    "  - change if you like"
   ]
  },
  {
   "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
   "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "goebbert1_kernel\n"
+     ]
+    }
+   ],
   "source": [
    "# INPUT NEEDED:\n",
-    "export KERNEL_NAME=${USER}_kernel\n",
+    "KERNEL_NAME=${USER}_kernel\n",
    "\n",
+    "export KERNEL_NAME=$(echo \"${KERNEL_NAME}\" | awk '{print tolower($0)}')\n",
    "echo ${KERNEL_NAME} # double check"
   ]
  },

 %% Cell type:markdown id: tags:

 # Create your own Jupyter Kernel

 %% Cell type:markdown id: tags:

 ---
 ## Building your own Jupyter kernel is a three step process
 1. Create/Pimp new virtual Python environment
   * venv
 2. Create/Edit launch script for the Jupyter kernel
   * kernel.sh
 3. Create/Edit Jupyter kernel configuration
   * kernel.json

 %% Cell type:markdown id: tags:

 ### Settings

 %% Cell type:markdown id: tags:

 * Set kernel name
+  - must be lower case
  - change if you like

 %% Cell type:code id: tags:

 ``` bash
 # INPUT NEEDED:
-export KERNEL_NAME=${USER}_kernel
+KERNEL_NAME=${USER}_kernel

+export KERNEL_NAME=$(echo "${KERNEL_NAME}" | awk '{print tolower($0)}')
 echo ${KERNEL_NAME} # double check
 ```

+%% Output
+
+    goebbert1_kernel
+
 %% Cell type:markdown id: tags:

 * Set kernel type
    - private kernel = "\${HOME}/.local/"
    - project kernel  = "\${PROJECT}/.local/"

 %% Cell type:code id: tags:

 ``` bash
 # INPUT NEEDED:
 export KERNEL_TYPE=private # private or project

 ###################
 # private kernel
 if [ "${KERNEL_TYPE}" == "private" ]; then
  export KERNEL_SPECS_PREFIX=${HOME}/.local
  echo "private kernel"
 # project kernel
 else
  export KERNEL_SPECS_PREFIX=${PROJECT}/.local
  echo "project kernel"
 fi
 export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels

 # check if kernel name is unique
 if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then
  echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}"
  echo "       Rename kernel name or remove directory."
 fi

 echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME} # double check
 ```

 %% Cell type:markdown id: tags:

 * Set directory for kernels virtual environment
  - change if you like

 %% Cell type:code id: tags:

 ``` bash
 # INPUT NEEDED:
 export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels

 ###################
 mkdir -p ${KERNEL_VENVS_DIR}
 if [ "${KERNEL_TYPE}" != "private" ]; then
  echo "Please check the permissions and ensure your project partners have read/execute permissions:"
  namei -l ${KERNEL_VENVS_DIR}
 fi

 echo ${KERNEL_VENVS_DIR} # double check
 ```

 %% Cell type:markdown id: tags:

 ---

 %% Cell type:markdown id: tags:

 ## 1. Create/Pimp new virual Python environment

 %% Cell type:markdown id: tags:

 * 1.1 - Load required modules

 %% Cell type:code id: tags:

 ``` bash
 module -q purge
 module -q use $OTHERSTAGES
 module -q load Stages/Devel-2019a 2> /dev/null # any stage can be used
 module -q load GCCcore/.8.3.0     2> /dev/null
 module -q load Python/3.6.8                    # only Python is required
 module list # double check
 ```

 %% Cell type:markdown id: tags:

 * 1.2 - Load extra modules you need for your kernel

 %% Cell type:code id: tags:

 ``` bash
 # module load <module you need>
 ```

 %% Cell type:markdown id: tags:

 * 1.3 - Create and activate a virtual environment for the kernel
 and ensure python packages installed in the virtual environment are always prefered

 %% Cell type:code id: tags:

 ``` bash
 python -m venv --system-site-packages ${KERNEL_VENVS_DIR}/${KERNEL_NAME}
 source ${KERNEL_VENVS_DIR}/${KERNEL_NAME}/bin/activate
 export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.6/site-packages:${PYTHONPATH}
 echo ${VIRTUAL_ENV} # double check
 ```

 %% Cell type:markdown id: tags:

 * 1.4 - Install Python libraries required for communication with Jupyter

 %% Cell type:code id: tags:

 ``` bash
 which pip
 pip install --ignore-installed ipykernel
 ls ${VIRTUAL_ENV}/lib/python3.6/site-packages/ # double check
 ```

 %% Cell type:markdown id: tags:

 * 1.5 - Install whatever else you need in your Python virtual environment (using pip)

 %% Cell type:code id: tags:

 ``` bash
 #pip install <python-package you need>
 ```

 %% Cell type:markdown id: tags:

 ---

 %% Cell type:markdown id: tags:

 ## 2. Create/Edit launch script for the Jupyter kernel

 %% Cell type:markdown id: tags:

 * 2.1 - Create launch script, which loads your Python virtual environment and starts the ipykernel process inside:

 %% Cell type:code id: tags:

 ``` bash
 echo "#!/bin/bash

 # Load required modules
 module purge
 module use "'$OTHERSTAGES'"
 module load Stages/Devel-2019a
 module load GCCcore/.8.3.0
 module load Python/3.6.8

 # Load extra modules you need for your kernel (as you did in step 1.2)
 #module load <module you need>

 # Activate your Python virtual environment
 source ${KERNEL_VENVS_DIR}/${KERNEL_NAME}/bin/activate

 # Ensure python packages installed in the virtual environment are always prefered
 export PYTHONPATH=${VIRTUAL_ENV}/lib/python3.6/site-packages:"'${PYTHONPATH}'"

 exec python -m ipykernel "'$@' > ${VIRTUAL_ENV}/kernel.sh
 chmod +x ${VIRTUAL_ENV}/kernel.sh
 ```

 %% Cell type:markdown id: tags:

 ---

 %% Cell type:markdown id: tags:

 ## 3. Create/Edit Jupyter kernel configuration

 %% Cell type:markdown id: tags:

 * 3.1 - Create Jupyter kernel configuration directory and files

 %% Cell type:code id: tags:

 ``` bash
 python -m ipykernel install --name=${KERNEL_NAME} --prefix ${VIRTUAL_ENV}
 export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels
 ```

 %% Cell type:markdown id: tags:

 * 3.2 - Adjust kernel.json file

 %% Cell type:code id: tags:

 ``` bash
 mv ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json.orig

 echo '{
  "argv": [
    "'${KERNEL_VENVS_DIR}/${KERNEL_NAME}/kernel.sh'",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ],
  "display_name": "'${KERNEL_NAME}'",
  "language": "python"
 }' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json
 ```

 %% Cell type:markdown id: tags:

 * 3.3 - Create link to kernel specs

 %% Cell type:code id: tags:

 ``` bash
 cd ${KERNEL_SPECS_DIR}
 ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .
 ls ${KERNEL_SPECS_DIR} # double check
 ```

 %% Cell type:markdown id: tags:

 ---

 %% Cell type:markdown id: tags:

 ## 4. Cleanup

 %% Cell type:code id: tags:

 ``` bash
 deactivate
 ```