Skip to content
Snippets Groups Projects
Commit 00ad3434 authored by Stefan Kesselheim's avatar Stefan Kesselheim
Browse files

Merge branch 'improvements' into 'master'

Improve everything

See merge request !1
parents f53eafb4 52fbeb24
No related branches found
No related tags found
No related merge requests found
#!/bin/bash #!/bin/bash
RELATIVE_PATH=`dirname ${BASH_SOURCE}`
ABSOLUTE_PATH=`realpath ${RELATIVE_PATH}`
[[ $0 != $BASH_SOURCE ]] && echo "The activation script must be sourced, otherwise the virtual environment will not work." || ( echo "Vars script must be sourced." && exit 1) ; # See https://stackoverflow.com/a/28336473
SOURCE_PATH="${BASH_SOURCE[0]:-${(%):-%x}}"
source ${ABSOLUTE_PATH}/config.sh RELATIVE_PATH="$(dirname "$SOURCE_PATH")"
source ${ABSOLUTE_PATH}/modules.sh ABSOLUTE_PATH="$(realpath "${RELATIVE_PATH}")"
export PYTHONPATH=`echo ${ENV_DIR}/lib/python*/site-packages`:${PYTHONPATH}
source ${ENV_DIR}/bin/activate
[[ "$0" != "${SOURCE_PATH}" ]] && echo "The activation script must be sourced, otherwise the virtual environment will not work." || ( echo "Vars script must be sourced." && exit 1) ;
source "${ABSOLUTE_PATH}"/config.sh
source "${ABSOLUTE_PATH}"/modules.sh
export PYTHONPATH="$(echo "${ENV_DIR}"/lib/python*/site-packages):${PYTHONPATH}"
source "${ENV_DIR}"/bin/activate
SOURCE_PATH="${BASH_SOURCE[0]:-${(%):-%x}}"
## Check if this script is sourced ## Check if this script is sourced
[[ $0 != $BASH_SOURCE ]] && echo "Setting vars" || ( echo "Vars script must be sourced." && exit 1) ; [[ "$0" != "${SOURCE_PATH}" ]] && echo "Setting vars" || ( echo "Vars script must be sourced." && exit 1) ;
## Determine location of this file ## Determine location of this file
RELATIVE_PATH=`dirname ${BASH_SOURCE}` RELATIVE_PATH="$(dirname "$SOURCE_PATH")"
ABSOLUTE_PATH=`realpath ${RELATIVE_PATH}` ABSOLUTE_PATH="$(realpath "${RELATIVE_PATH}")"
#################################### ####################################
### User Configuration ### User Configuration
export ENV_NAME=`basename $ABSOLUTE_PATH` # Default Name of the venv is the directory that contains this file export ENV_NAME="$(basename "$ABSOLUTE_PATH")" # Default Name of the venv is the directory that contains this file
export ENV_DIR=${ABSOLUTE_PATH}/venv # Default location of this VENV is "./venv" export ENV_DIR="${ABSOLUTE_PATH}"/venv # Default location of this VENV is "./venv"
#!/bin/bash #!/bin/bash
RELATIVE_PATH=`dirname ${BASH_SOURCE}` SOURCE_PATH="${BASH_SOURCE[0]:-${(%):-%x}}"
ABSOLUTE_PATH=`realpath ${RELATIVE_PATH}`
source ${ABSOLUTE_PATH}/config.sh
KERNELFILE=${ENV_DIR}/kernel.sh RELATIVE_PATH="$(dirname "$SOURCE_PATH")"
ABSOLUTE_PATH="$(realpath "${RELATIVE_PATH}")"
source "${ABSOLUTE_PATH}"/config.sh
echo the name is $ENV_NAME KERNELFILE="${ENV_DIR}"/kernel.sh
echo "Setting up the kernel script in the following dir: " ${KERNELFILE} echo the name is "$ENV_NAME"
echo "Setting up the kernel script in the following dir: " "${KERNELFILE}"
echo '#!/bin/bash echo '#!/bin/bash
source '"${ABSOLUTE_PATH}"'/activate.sh source "'"${ABSOLUTE_PATH}"'"/activate.sh
exec python -m ipykernel $@' > ${KERNELFILE} exec python -m ipykernel "$@"' > "${KERNELFILE}"
chmod a+x ${KERNELFILE} chmod a+x "${KERNELFILE}"
mkdir -p ~/.local/share/jupyter/kernels/${ENV_NAME} mkdir -p ~/.local/share/jupyter/kernels/"${ENV_NAME}"
echo '{ echo '{
"argv": [ "argv": [
"'"${KERNELFILE}"'", "'"${KERNELFILE}"'",
"-f", "-f",
"{connection_file}" "{connection_file}"
], ],
"display_name": "'${ENV_NAME}'", "display_name": "'"${ENV_NAME}"'",
"language": "python" "language": "python"
}' > ~/.local/share/jupyter/kernels/${ENV_NAME}/kernel.json }' > ~/.local/share/jupyter/kernels/"${ENV_NAME}"/kernel.json
ml GCC/9.3.0 ParaStationMPI/5.4.7-1 Python/3.8.5 module purge
module load GCC/10.3.0 ParaStationMPI/5.4.10-1 Python/3.8.5
...@@ -10,12 +10,12 @@ curated and optimized, including compilers, MPI version etc. Extra Python packag ...@@ -10,12 +10,12 @@ curated and optimized, including compilers, MPI version etc. Extra Python packag
with pip into user space. This, however, does not create a reproducible environment that can be used with pip into user space. This, however, does not create a reproducible environment that can be used
by other users as well. by other users as well.
Conceptuall, with Virtual Environments, it is easily possible to create project-based virtual environments. Conceptually, with Virtual Environments, it is easily possible to create project-based virtual environments.
These scripts streamline the creation und usage of such environments and make it easy for a users to share a setup These scripts streamline the creation and usage of such environments and make it easy for a users to share a setup
and to put it under version control with the main code. and to put it under version control with the main code.
Furthermore, in typical compute setup of scientific projects, one or more packages possibly are in active Furthermore, in typical compute setup of scientific projects, one or more packages possibly are in active
developement. In the context of these setups, it is intended to include them as submodules and add integrate development. In the context of these setups, it is intended to include them as submodules and add integrate
them into the workflow. This can e.g. mean that a compilation step is added in the setup step and them into the workflow. This can e.g. mean that a compilation step is added in the setup step and
setting appropriate environment variables is included in the activation step. setting appropriate environment variables is included in the activation step.
...@@ -32,16 +32,16 @@ packages to this file to reproducibly add them to the venv. ...@@ -32,16 +32,16 @@ packages to this file to reproducibly add them to the venv.
The script `setup.sh` creates the venv according to the config given in `config.sh`. Please **edit** this The script `setup.sh` creates the venv according to the config given in `config.sh`. Please **edit** this
file to add a setup step for submodules (e.g. compilation of libraries). If only plain venvs are used, this file file to add a setup step for submodules (e.g. compilation of libraries). If only plain venvs are used, this file
can remain unchanged. can remain unchanged. Note that the script *must* be ran at least once after the above configurations to actually create the environment.
The script `activate.sh` sets the environment variables such that the venv can be used. Please **edit** this file The script `activate.sh` sets the environment variables such that the venv can be used. Please **edit** this file
to add environment variables for submodules. Note that it the script must be *sourced* to take effect. Example: to add environment variables for submodules. Note that the script must be *sourced* to take effect. Example:
```bash ```bash
source <path_to_venv>/activate.sh source <path_to_venv>/activate.sh
``` ```
The script `create_kernel.sh` will create a kernel json file in the user's home directory that can be found The script `create_kernel.sh` will create a kernel json file in the user's home directory that can be found
by jupyter and a helper script in the virtual environment folder. by Jupyter and a helper script in the virtual environment folder.
...@@ -50,4 +50,5 @@ by jupyter and a helper script in the virtual environment folder. ...@@ -50,4 +50,5 @@ by jupyter and a helper script in the virtual environment folder.
2. Edit `modules.sh` to change the modules loaded prior to the creation of the venv. 2. Edit `modules.sh` to change the modules loaded prior to the creation of the venv.
3. Edit `requirements.txt` to change the packages to be installed during setup. 3. Edit `requirements.txt` to change the packages to be installed during setup.
4. Edit `setup.sh` and `activate.sh` to add extra steps for custom modules. 4. Edit `setup.sh` and `activate.sh` to add extra steps for custom modules.
5. Create a kernel with `create_kernel.sh` 5. Create the environment with `bash setup.sh`.
6. Create a kernel with `bash create_kernel.sh`.
#!/bin/bash #!/bin/bash
RELATIVE_PATH=`dirname ${BASH_SOURCE}`
ABSOLUTE_PATH=`realpath ${RELATIVE_PATH}`
source ${ABSOLUTE_PATH}/config.sh SOURCE_PATH="${BASH_SOURCE[0]:-${(%):-%x}}"
source ${ABSOLUTE_PATH}/modules.sh
python -m venv --prompt $ENV_NAME --system-site-packages ${ENV_DIR} RELATIVE_PATH="$(dirname "$SOURCE_PATH")"
ABSOLUTE_PATH="$(realpath "${RELATIVE_PATH}")"
source ${ABSOLUTE_PATH}/activate.sh source "${ABSOLUTE_PATH}"/config.sh
source "${ABSOLUTE_PATH}"/modules.sh
pip install -r ${ABSOLUTE_PATH}/requirements.txt
python -m venv --prompt "$ENV_NAME" --system-site-packages "${ENV_DIR}"
source "${ABSOLUTE_PATH}"/activate.sh
python -m pip install -r "${ABSOLUTE_PATH}"/requirements.txt
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment