... | ... | @@ -3,6 +3,7 @@ SERGHEI (Simulation Environment for Geomorphology, Hydrodynamics and Ecohydrolog |
|
|
This is a basic tutorial to build and run SERGHEI on JSC systems.
|
|
|
|
|
|
The overall procedure is as follows:
|
|
|
|
|
|
1. [Log into JSC systems](https://gitlab.jsc.fz-juelich.de/caviedesvoullieme1/serghei_tutorial/-/wikis/SERGHEI-tutorial#log-into-jsc-systems)
|
|
|
2. [Fetch the code from a repository](https://gitlab.jsc.fz-juelich.de/caviedesvoullieme1/serghei_tutorial/-/wikis/SERGHEI-tutorial#get-serghei-from-a-repository)
|
|
|
3. [Prepare the environment for compiling and building](https://gitlab.jsc.fz-juelich.de/caviedesvoullieme1/serghei_tutorial/-/wikis/SERGHEI-tutorial#preparing-the-environment)
|
... | ... | @@ -10,62 +11,65 @@ The overall procedure is as follows: |
|
|
5. [Run a test case](https://gitlab.jsc.fz-juelich.de/caviedesvoullieme1/serghei_tutorial/-/wikis/SERGHEI-tutorial#running-a-test-cases)
|
|
|
|
|
|
# Log into JSC systems
|
|
|
|
|
|
In this tutorial we will use JSC's [JURECA-DC](https://www.fz-juelich.de/en/news/archive/press-release/2021/2021-06-23-jureca-dc) machine. You can find JURECA-DC's hardware configuration [here](https://apps.fz-juelich.de/jsc/hps/jureca/configuration.html), which is part of the [full user documentation](https://apps.fz-juelich.de/jsc/hps/jureca/index.html). General rules and procedures for gaining [access](https://apps.fz-juelich.de/jsc/hps/jureca/access.html) to the machine are part of this documentation.
|
|
|
|
|
|
For this tutorial, you already have a JSC account and ssh keys ready, so we will skip these steps.
|
|
|
Therefore, to access JURECA-DC we need to `ssh` into it.
|
|
|
For this tutorial, you already have a JSC account and ssh keys ready, so we will skip these steps. Therefore, to access JURECA-DC we need to `ssh` into it.
|
|
|
|
|
|
***TO-DO: SHALL WE CREATE SOME PERSISTING SSH CONFIG TO USE?***
|
|
|
|
|
|
```ssh jureca```
|
|
|
|
|
|
**TO-DO: SHALL WE CREATE SOME PERSISTING SSH CONFIG TO USE?**
|
|
|
`ssh jureca`
|
|
|
{: .alert .alert-gitlab-orange}
|
|
|
|
|
|
# Get SERGHEI from a repository
|
|
|
SERGHEI is available at [https://gitlab.com/serghei-model/serghei](https://gitlab.com/serghei-model/serghei). You can simply click on the link to see the project GitLab repository.
|
|
|
|
|
|
SERGHEI is available at https://gitlab.com/serghei-model/serghei. You can simply click on the link to see the project GitLab repository.
|
|
|
|
|
|
## Get source code
|
|
|
|
|
|
To get the source code, open a terminal, and while in your account's home directory, simply **clone** the repository
|
|
|
|
|
|
```git clone https://gitlab.com/serghei-model/serghei.git```
|
|
|
`git clone https://gitlab.com/serghei-model/serghei.git`
|
|
|
|
|
|
(Note that the git repository itself is simply the repository URL with a `.git` extension. You can get this address in the GitLab web interface by pressing on the "Clone" button and copying the content under "Clone with HTTPS")
|
|
|
|
|
|
## SERGHEI path
|
|
|
You can now navigate to the SERGHEI root directory
|
|
|
```cd serghei```
|
|
|
|
|
|
You can now navigate to the SERGHEI root directory `cd serghei`
|
|
|
|
|
|
Check the path to the current directory using the `pwd` command. This should return be the absolute path to the local working copy of the repository (i.e., the `serghei` directory).
|
|
|
|
|
|
# Preparing the environment
|
|
|
|
|
|
## Environmental variable
|
|
|
|
|
|
Create an environmental variable for the path to the local working copy of the repository
|
|
|
|
|
|
```export SERGHEIPATH=$(pwd)```
|
|
|
`export SERGHEIPATH=$(pwd)`
|
|
|
|
|
|
This command exports the current path (which is returned by the `pwd` command) into the `SERGHEIPATH` variable. Bare in mind that this is not persisting once you log out or close the terminal. This means you need to set this again if you do.
|
|
|
|
|
|
You can check if this variable is set by running
|
|
|
|
|
|
```echo $SERGHEIPATH```
|
|
|
`echo $SERGHEIPATH`
|
|
|
|
|
|
**Optional but convenient**
|
|
|
|
|
|
A way setting this in a persisting way is to define it in your local `.bashrc` file. Edit the `.bashrc` file
|
|
|
|
|
|
```vim .bashrc```
|
|
|
`vim .bashrc`
|
|
|
|
|
|
and include a new line which includes the path to `serghei` (which you can copy from the `pwd` output).
|
|
|
|
|
|
```export SERGHEIPATH=/path/to/serghei```
|
|
|
`export SERGHEIPATH=/path/to/serghei`
|
|
|
|
|
|
## Dependencies
|
|
|
|
|
|
SERGHEI has a few dependencies (software on which it relies).
|
|
|
|
|
|
1. software to compile the code. We rely on the GCC compiler for this.
|
|
|
|
|
|
2. software to manage communications and parallelism. For this we require some flavour of the Message Passing Interface [MPI](https://en.wikipedia.org/wiki/Message_Passing_Interface). For JSC systems we can use [OpenMPI](https://www.open-mpi.org/), but we prefer [Parastation MPI](https://github.com/ParaStation/psmpi/branches).
|
|
|
|
|
|
3. software to manage parallel output to disk. We rely on [Parallel NetCDF](https://parallel-netcdf.github.io/).
|
|
|
|
|
|
4. In order to program, compile and run SERGHEI on different hardware architectures (namely CPUs and GPUs) we rely on the performance-portability library [Kokkos](https://github.com/kokkos/kokkos.git).
|
|
|
|
|
|
Typically, in your own computer you would have to download and install all this software. In HPC systems, (1) you don't have administrator rights to do so, (2) it can be complicated to build complex layers of software dependencies, and (3) typically used software is already installed and ready to use in the system.
|
... | ... | @@ -75,53 +79,58 @@ The software is usually not visible (in fact it is not loaded into the system) b |
|
|
With the exception of Kokkos, all other dependencies can be found in JSC's systems.
|
|
|
|
|
|
### Loading modules
|
|
|
|
|
|
To find a module try
|
|
|
|
|
|
```module spider gcc```
|
|
|
`module spider gcc`
|
|
|
|
|
|
In this way you can find the formal name of the module and the versions available in the system.
|
|
|
|
|
|
To load the GCC module try
|
|
|
|
|
|
```module load GCC```
|
|
|
`module load GCC`
|
|
|
|
|
|
Try to do the same for the Parastation MPI and the Parallel NetCDF modules. Use `module spider` to find the modules, and `module load` to load them.
|
|
|
|
|
|
Like the environmental variable before, the *environment* you have prepared by loading these modules will not survive when you log out or close your terminal, in which case you will have to repeat this procedure.
|
|
|
Like the environmental variable before, the _environment_ you have prepared by loading these modules will not survive when you log out or close your terminal, in which case you will have to repeat this procedure.
|
|
|
|
|
|
It is therefore practical to write a script that will make this easier. This of already exists for SERGHEI, and you can find it in the `serghei/machines` folder. Here you will find a number of scripts which load the modules required by SERGHEI for several machines.
|
|
|
|
|
|
Since we are using JURECA-DC, we should use the script for that machine. To load the environment defined in this script run
|
|
|
|
|
|
```source jurecaDCModules```
|
|
|
`source jurecaDCModules`
|
|
|
|
|
|
If all goes well, we now should have the environment ready. You can check which module are loaded with
|
|
|
|
|
|
``` module list```
|
|
|
`module list`
|
|
|
|
|
|
### Getting Kokkos
|
|
|
Kokkos is available in GitHub at [https://github.com/kokkos/kokkos.git](https://github.com/kokkos/kokkos.git). To clone the Kokkos repository, follow a analogous procedure to what was done to clone SERGHEI into your home directory.
|
|
|
|
|
|
Kokkos is available in GitHub at https://github.com/kokkos/kokkos.git. To clone the Kokkos repository, follow a analogous procedure to what was done to clone SERGHEI into your home directory.
|
|
|
|
|
|
Usually dependencies need to be built in the system (so that they are available locally as a binary, or through a module). For SERGHEI, Kokkos does not need to be built beforehand. It will be built when we build SERGHEI, if the path to Kokkos is properly set in the next step.
|
|
|
|
|
|
# Compiling and building
|
|
|
|
|
|
1. navigate to `serghei/src/`.
|
|
|
2. Open the `Makefile`. This is a script which controls how SERGHEI is compiled. Note that there are some definitions early on. In particular `KOKKOS_PATH` and `KOKKOS_SRC_PATH` are defined as a function of `HOME`. Note that the `Makefile` assumes that `KOKKOS_PATH` will be in `HOME` path. `HOME` is an environmental variable which contain the path to your home directory. Exit the file and check what your HOME environmental variable contains.
|
|
|
2. No we will use `make` to compile and build. Running `make` attempts to read the `Makefile`, which you can see exists in this directory. If yout ry to run `make` in a different directory which does not contain a `Makefile` you will get an error. `make` support parallel threads, so that the compilation is faster. For example
|
|
|
3. No we will use `make` to compile and build. Running `make` attempts to read the `Makefile`, which you can see exists in this directory. If yout ry to run `make` in a different directory which does not contain a `Makefile` you will get an error. `make` support parallel threads, so that the compilation is faster. For example
|
|
|
|
|
|
```make -j 8```
|
|
|
`make -j 8`
|
|
|
|
|
|
would use 8 threads to run the compilation. Give this a try. If everything is correctly set, you should see a message indicating a successful build.
|
|
|
|
|
|
3- check that the `serghei` binary exists at `serghei/bin`.
|
|
|
|
|
|
# Running a test case
|
|
|
|
|
|
Let's run a simple test to check if everything is working fine. We will use an analytical dam break case, included with SERGHEI as a unit test.
|
|
|
|
|
|
1. Navigate to `serghei/unitTests/ShallowWater/Analytical/dambreakX_4_1`
|
|
|
2. You will see that this here contains an `input` directory, which contains a number of input files to define the simulation. At this stage, we will not be concerned with how this works. We will focus on that later.
|
|
|
3. To run this case with SERGHEI, run
|
|
|
|
|
|
```$SERGHEIPATH/bin/serghei ./input/ ./output/ 1```
|
|
|
`$SERGHEIPATH/bin/serghei ./input/ ./output/ 1`
|
|
|
|
|
|
The first part of the command above (`$SERGHEIPATH/bin/serghei`) invokes the `serghei` binary, which is located in that path. If `$SERGHEIPATH` is not properly set, the computer will not find the `serghei` binary.
|
|
|
|
... | ... | |