diff --git a/readme.md b/readme.md index 138ef2979fd6d319c01c7d18f83a48b9a0e74b2a..d94767ebeea86f26b7f5ca97ce6f69ec7715b71b 100644 --- a/readme.md +++ b/readme.md @@ -1,16 +1,47 @@ Supercomputing Environment Template using Python Virtual Environments ================= -# Idea +# TLDR +This repo contains an example on how to easily create a consistent working environment that can be used +in the "normal" supercomputer workflow as well as in Jupyter-JSC. + +There are two flavors of this type of repo: +1. The "normal" version based on environment modules and a python virtual environment +2. The singularity version where a Singularity container replaces the module environment + +The user creates a working environment as a fork of this repo, developes it along with the requirement of +their project and finally archives or publishes it jointly with the results of the research project. + +## HowTo Environment Modules +1. Clone the repo into a folder with the name of the environment module `git clone /path/to/repo ./my_favourite_environment`. +2. Edit `modules.sh` to select the modules to load prior to creating a virtual environment +3. Edit `requirements.txt` to select the packages to be installed via pip +4. Create the environment + * execute `setup.sh` to create the virtual environment + * execute `create_kernel.sh` to create a kernel for Jupyer-JSC +5. `source activate.sh` to enter the environment + +## HowTo Singularity +1. Clone the repo into a folder with the name of the environment module `git clone /path/to/repo ./my_favourite_environment`. +2. Edit `config.sh` to select the singularity container. +3. If required, edit `setup.sh` to make changes to the container. This is possible as we support a persistent overlay file system +3. Edit `requirements.txt` to select the packages to be installed via pip +4. Create the environment + * execute `setup.sh` to create the virtual environment + * execute `create_kernel.sh` to create a kernel for Jupyer-JSC +5. Make sure you execute `source activate.sh` each time you execute something within the container. + +# Description This project contains a lightweight set of scripts to easily create Python working environments on typical supercomputer setups, including creating Jupyter Kernels. +## Environment Modules On Supercomputers, typically a basic environment based on **Environment Modules**. This setup is carefully curated and optimized, including compilers, MPI version etc. Extra Python packages can be installed with pip into user space. This, however, does not create a reproducible environment that can be used by other users as well. -Conceptuall, with Virtual Environments, it is easily possible to create project-based virtual environments. +Conceptuall, with virtual environments, it is easily possible to create project-based virtual environments. These scripts streamline the creation und usage of such environments and make it easy for a users to share a setup and to put it under version control with the main code. @@ -19,6 +50,9 @@ developement. In the context of these setups, it is intended to include them as them into the workflow. This can e.g. mean that a compilation step is added in the setup step and setting appropriate environment variables is included in the activation step. +## Singularity +A useful alternative to environment modules are container-based environments. + # Details The setup is configured in the bash script `config.sh`. The user can define a name for the venv and directory where the venv files are stored. This defaults to the directory name of the containing folder and the "." folder