Supercomputing Environment Template using Python Virtual Environments
Idea
This project contains a lightweight set of scripts to easily create Python working environments on typical supercomputer setups, including creating Jupyter Kernels.
On Supercomputers, typically a basic environment based on Environment Modules. This setup is carefully curated and optimized, including compilers, MPI version etc. Extra Python packages can be installed with pip into user space. This, however, does not create a reproducible environment that can be used by other users as well.
Conceptuall, with Virtual Environments, it is easily possible to create project-based virtual environments. These scripts streamline the creation und usage of such environments and make it easy for a users to share a setup and to put it under version control with the main code.
Furthermore, in typical compute setup of scientific projects, one or more packages possibly are in active developement. In the context of these setups, it is intended to include them as submodules and add integrate them into the workflow. This can e.g. mean that a compilation step is added in the setup step and setting appropriate environment variables is included in the activation step.
Details
The setup is configured in the bash script config.sh
. The user can define a name for the venv and directory
where the venv files are stored. This defaults to the directory name of the containing folder and the "." folder
of the scripts. Please edit this file if you want a custom name and location for the venv.
The modules ontop of which the the venv should be built are defined in modules.sh
. Please edit the file
to your needs.
The file requirements.txt
contains a list of packages to be installed during the setup process. Add required
packages to this file to reproducibly add them to the venv.
The script setup.sh
creates the venv according to the config given in config.sh
. Please edit this
file to add a setup step for submodules (e.g. compilation of libraries). If only plain venvs are used, this file
can remain unchanged.
The script activate.sh
sets the environment variables such that the venv can be used. Please edit this file
to add environment variables for submodules. Note that it the script must be sourced to take effect. Example:
source <path_to_venv>/activate.sh
The script create_kernel.sh
will create a kernel json file in the user's home directory that can be found
by jupyter and a helper script in the virtual environment folder.
Intended Workflow
- Edit
config.sh
to change name an location of the venv if required. - Edit
modules.sh
to change the modules loaded prior to the creation of the venv. - Edit
requirements.txt
to change the packages to be installed during setup. - Edit
setup.sh
andactivate.sh
to add extra steps for custom modules. - Create a kernel with
create_kernel.sh