Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
J
jax_kernel
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Container registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
GitLab community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Stefan Kesselheim
jax_kernel
Commits
e8cab00f
Commit
e8cab00f
authored
4 years ago
by
Stefan Kesselheim
Browse files
Options
Downloads
Patches
Plain Diff
update of docs
parent
9bc5523e
No related branches found
No related tags found
No related merge requests found
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
readme.md
+36
-2
36 additions, 2 deletions
readme.md
with
36 additions
and
2 deletions
readme.md
+
36
−
2
View file @
e8cab00f
Supercomputing Environment Template using Python Virtual Environments
=================
# Idea
# TLDR
This repo contains an example on how to easily create a consistent working environment that can be used
in the "normal" supercomputer workflow as well as in Jupyter-JSC.
There are two flavors of this type of repo:
1.
The "normal" version based on environment modules and a python virtual environment
2.
The singularity version where a Singularity container replaces the module environment
The user creates a working environment as a fork of this repo, developes it along with the requirement of
their project and finally archives or publishes it jointly with the results of the research project.
## HowTo Environment Modules
1.
Clone the repo into a folder with the name of the environment module
`git clone /path/to/repo ./my_favourite_environment`
.
2.
Edit
`modules.sh`
to select the modules to load prior to creating a virtual environment
3.
Edit
`requirements.txt`
to select the packages to be installed via pip
4.
Create the environment
*
execute
`setup.sh`
to create the virtual environment
*
execute
`create_kernel.sh`
to create a kernel for Jupyer-JSC
5.
`source activate.sh`
to enter the environment
## HowTo Singularity
1.
Clone the repo into a folder with the name of the environment module
`git clone /path/to/repo ./my_favourite_environment`
.
2.
Edit
`config.sh`
to select the singularity container.
3.
If required, edit
`setup.sh`
to make changes to the container. This is possible as we support a persistent overlay file system
3.
Edit
`requirements.txt`
to select the packages to be installed via pip
4.
Create the environment
*
execute
`setup.sh`
to create the virtual environment
*
execute
`create_kernel.sh`
to create a kernel for Jupyer-JSC
5.
Make sure you execute
`source activate.sh`
each time you execute something within the container.
# Description
This project contains a lightweight set of scripts to easily create Python working environments on
typical supercomputer setups, including creating Jupyter Kernels.
## Environment Modules
On Supercomputers, typically a basic environment based on
**Environment Modules**
. This setup is carefully
curated and optimized, including compilers, MPI version etc. Extra Python packages can be installed
with pip into user space. This, however, does not create a reproducible environment that can be used
by other users as well.
Conceptuall, with
V
irtual
E
nvironments, it is easily possible to create project-based virtual environments.
Conceptuall, with
v
irtual
e
nvironments, it is easily possible to create project-based virtual environments.
These scripts streamline the creation und usage of such environments and make it easy for a users to share a setup
and to put it under version control with the main code.
...
...
@@ -19,6 +50,9 @@ developement. In the context of these setups, it is intended to include them as
them into the workflow. This can e.g. mean that a compilation step is added in the setup step and
setting appropriate environment variables is included in the activation step.
## Singularity
A useful alternative to environment modules are container-based environments.
# Details
The setup is configured in the bash script
`config.sh`
. The user can define a name for the venv and directory
where the venv files are stored. This defaults to the directory name of the containing folder and the "." folder
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment