@@ -7,7 +7,7 @@ in the "normal" supercomputer workflow as well as in Jupyter-JSC.
There are two flavors of this type of repo:
1. The "normal" version based on environment modules and a python virtual environment
2. The singularity version where a Singularity container replaces the module environment
2. The singularity version where a Singularity container replaces the module environment. Check out the branch `singularity`.
The user creates a working environment as a fork of this repo, developes it along with the requirement of
their project and finally archives or publishes it jointly with the results of the research project.
...
...
@@ -22,14 +22,15 @@ their project and finally archives or publishes it jointly with the results of t
5.`source activate.sh` to enter the environment
## HowTo Singularity
1. Clone the repo into a folder with the name of the environment module `git clone /path/to/repo ./my_favourite_environment`.
2. Edit `config.sh` to select the singularity container.
3. If required, edit `setup.sh` to make changes to the container. This is possible as we support a persistent overlay file system
3. Edit `requirements.txt` to select the packages to be installed via pip
4. Create the environment
1. Get you favourite base image to use as starting point. Example: `singularity pull docker://nvcr.io/nvidia/pytorch:22.10-py3`
2. Clone the repo into a folder with the name of the environment module `git clone /path/to/repo ./my_favourite_environment`.
3. Edit `config.sh` to select the singularity container.
4. If required, edit `setup.sh` to make changes to the container. This is possible as we support a persistent overlay file system
5. Edit `requirements.txt` to select the packages to be installed via pip
6. Create the environment
* execute `setup.sh` to create the virtual environment
* execute `create_kernel.sh` to create a kernel for Jupyer-JSC
5. Make sure you execute `source activate.sh` each time you execute something within the container.
7. Make sure you execute `source activate.sh` each time you execute something within the container.
# Description
This project contains a lightweight set of scripts to easily create Python working environments on
...
...
@@ -51,7 +52,7 @@ them into the workflow. This can e.g. mean that a compilation step is added in t
setting appropriate environment variables is included in the activation step.
## Singularity
A useful alternative to environment modules are container-based environments. ToDo: More Details
A useful alternative to environment modules are container-based environments. On JUWELS Booster and JURECA DC, we highly recommend using NVIDIA's pre-built docker containers in the NGC https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch . Singularity can import docker images from public container registries with a command link `singularity pull docker://nvcr.io/nvidia/pytorch:22.10-py3`. This will automatically create a `.sif` file that you can use as starting point.
# Details
The setup is configured in the bash script `config.sh`. The user can define a name for the venv and directory