... | ... | @@ -6,7 +6,7 @@ |
|
|
* The system, compiler and implementations must support silent NaN floating-point values.
|
|
|
* An MPI implementation (ideally CUDA aware)
|
|
|
* SIONlib: http://fz-juelich.de/ias/jsc/sionlib
|
|
|
* If using NVLink (cuda), Omni-Path (psm2), UCX (ucp), or IBVerbs (ibverbs)
|
|
|
* If using NVLink (cuda), Omni-Path (psm2), UCX (ucp), TCP(tcp), or IBVerbs (ibverbs)
|
|
|
* `minipmi`: https://github.com/kraused/minipmi
|
|
|
* If using NVLink (cuda):
|
|
|
* A standard compliant CUDA compiler
|
... | ... | @@ -19,49 +19,45 @@ |
|
|
In general you have to do the following steps
|
|
|
1. Install Dependencies
|
|
|
2. Set-Up Environment, Pahs etc.
|
|
|
3. Install `linktest` with `make`, disable unavailable transport layers
|
|
|
3. Install `linktest` with `make`, enable transport layers other than mpi
|
|
|
4. [Optional] Install `linktest-report` via pip (Update PYTHONPATH or use virtual environments)
|
|
|
5. [Optional] Install `pingponganalysis` with `make`
|
|
|
|
|
|
Quick-Start-Example installing Linktest with only MPI and TCP support. (Might not work on your system)
|
|
|
Quick-Start-Example installing Linktest with only MPI support. (Might not work on your system)
|
|
|
```bash
|
|
|
mkdir -p install
|
|
|
cd benchmark
|
|
|
ml GCC ParaStationMPI SIONlib SciPy-Stack # 1 + 2
|
|
|
make HAVE_MINIPMI=0 PREFIX=../install clean install # 3
|
|
|
make PREFIX=../install clean install # 3
|
|
|
python -m pip install ../python/setup.py --prefix=../install # 4
|
|
|
```
|
|
|
|
|
|
# Transport Layers
|
|
|
Transport Layers can be (de-)activated with `make HAVE_<LAYER>=(0)1`
|
|
|
# Build Variables / Transport Layers
|
|
|
Transport Layers can be (de-)activated with `make HAVE_<LAYER>=(0)1`
|
|
|
All Layers other than mpi require the tcp Layer and MiniPMI to be available.
|
|
|
The following transport layers and other variables may be used:
|
|
|
|
|
|
| Layer | Variable | Default | Description |
|
|
|
| ----- | -------- | ------- | ----------- |
|
|
|
| `mpi` | `HAVE_MPI` | Enabled | MPI |
|
|
|
| `tcp` | `HAVE_TCP` | Enabled | TCP sockets, if deactivated, layers below will crash at runtime (see chapter [Execution](Execution)) |
|
|
|
| | `HAVE_MINIPMI` | Enabled | If deactivated, deactivates all layers below |
|
|
|
| `ibverbs` | `HAVE_IBVERBS` | Enabled | Verbs-based implementation |
|
|
|
| `psm2` | `HAVE_PSM2` | Enabled | PSM2 (Omni-Path) |
|
|
|
| `ucp` | `HAVE_UCP` | Enabled | UCX (using high level api ucp) |
|
|
|
| `cuda` | `HAVE_CUDA` | Enabled | NVLink (node-internal) |
|
|
|
| `tcp` | `HAVE_TCP` | Disabled | Automatically enabled if any Layer below is enabled |
|
|
|
| `ibverbs` | `HAVE_IBVERBS` | Disabled| Verbs-based implementation |
|
|
|
| `psm2` | `HAVE_PSM2` | Disabled| PSM2 (Omni-Path) |
|
|
|
| `ucp` | `HAVE_UCP` | Disabled| UCX (using high level api ucp) |
|
|
|
| `cuda` | `HAVE_CUDA` | Disabled| NVLink (node-internal) |
|
|
|
| | `CUARCH` | | Needs to be set if `cuda` is enable |
|
|
|
| | `V` | | If defined prints detailed steps and additional variables |
|
|
|
| | `SYSTEM` | `generic` | Use predefined custom affinities for HCAs and GPUs |
|
|
|
| | `PREFIX` | `/usr/local/bin` | Install directory for binaries |
|
|
|
|
|
|
For `CUARCH` values see https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#virtual-architecture-feature-list
|
|
|
|
|
|
# JSC Build Examples
|
|
|
**JUWELS:**
|
|
|
# JSC Build Examples (TODO Update)
|
|
|
|
|
|
| Partitition | Command |
|
|
|
| ----------- | ------- |
|
|
|
| Cluster | `xenv -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>>` (uses the `generic` system settings) |
|
|
|
| Booster | `xenv -L CUDA -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>> SYSTEM="JUWELS Booster"` |
|
|
|
|
|
|
**JURECA:**
|
|
|
|
|
|
| Partitition | Command |
|
|
|
| ----------- | ------- |
|
|
|
| JURECA-DC CPU | `xenv -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 HAVE_CUDA=0 PREFIX_DIR=<<<PREFIX_DIR>>>` (uses the `generic` system settings) |
|
|
|
| Juwels Cluster and Juwels Booster | `xenv -L GCC -L ParaStationMPI -L SIONlib make install HAVE_TCP=1 HAVE_IBVERBS=1 HAVE_UCP=1 PREFIX=<<<DIR>>>`|
|
|
|
| JURECA-DC CPU | `xenv -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 HAVE_CUDA=0 PREFIX_DIR=<<<PREFIX_DIR>>>` (uses the `generic` system settings) |
|
|
|
| JURECA-DC CPU | `xenv -L CUDA -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>> SYSTEM="JURECA-DC GPU"` |
|
|
|
| Booster | `xenv -L Architecture/KNL -L Intel -L IntelMPI -L SIONlib make install HAVE_CUDA=0 PREFIX_DIR=<<<PREFIX_DIR>>>` |
|
|
|
|
... | ... | |