Skip to content
Snippets Groups Projects
Commit e8cab00f authored by Stefan Kesselheim's avatar Stefan Kesselheim
Browse files

update of docs

parent 9bc5523e
No related branches found
No related tags found
No related merge requests found
Supercomputing Environment Template using Python Virtual Environments
=================
# Idea
# TLDR
This repo contains an example on how to easily create a consistent working environment that can be used
in the "normal" supercomputer workflow as well as in Jupyter-JSC.
There are two flavors of this type of repo:
1. The "normal" version based on environment modules and a python virtual environment
2. The singularity version where a Singularity container replaces the module environment
The user creates a working environment as a fork of this repo, developes it along with the requirement of
their project and finally archives or publishes it jointly with the results of the research project.
## HowTo Environment Modules
1. Clone the repo into a folder with the name of the environment module `git clone /path/to/repo ./my_favourite_environment`.
2. Edit `modules.sh` to select the modules to load prior to creating a virtual environment
3. Edit `requirements.txt` to select the packages to be installed via pip
4. Create the environment
* execute `setup.sh` to create the virtual environment
* execute `create_kernel.sh` to create a kernel for Jupyer-JSC
5. `source activate.sh` to enter the environment
## HowTo Singularity
1. Clone the repo into a folder with the name of the environment module `git clone /path/to/repo ./my_favourite_environment`.
2. Edit `config.sh` to select the singularity container.
3. If required, edit `setup.sh` to make changes to the container. This is possible as we support a persistent overlay file system
3. Edit `requirements.txt` to select the packages to be installed via pip
4. Create the environment
* execute `setup.sh` to create the virtual environment
* execute `create_kernel.sh` to create a kernel for Jupyer-JSC
5. Make sure you execute `source activate.sh` each time you execute something within the container.
# Description
This project contains a lightweight set of scripts to easily create Python working environments on
typical supercomputer setups, including creating Jupyter Kernels.
## Environment Modules
On Supercomputers, typically a basic environment based on **Environment Modules**. This setup is carefully
curated and optimized, including compilers, MPI version etc. Extra Python packages can be installed
with pip into user space. This, however, does not create a reproducible environment that can be used
by other users as well.
Conceptuall, with Virtual Environments, it is easily possible to create project-based virtual environments.
Conceptuall, with virtual environments, it is easily possible to create project-based virtual environments.
These scripts streamline the creation und usage of such environments and make it easy for a users to share a setup
and to put it under version control with the main code.
......@@ -19,6 +50,9 @@ developement. In the context of these setups, it is intended to include them as
them into the workflow. This can e.g. mean that a compilation step is added in the setup step and
setting appropriate environment variables is included in the activation step.
## Singularity
A useful alternative to environment modules are container-based environments.
# Details
The setup is configured in the bash script `config.sh`. The user can define a name for the venv and directory
where the venv files are stored. This defaults to the directory name of the containing folder and the "." folder
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment