Changes

Page history
Update Build authored by Yannik Müller's avatar Yannik Müller
Show whitespace changes
Inline Side-by-side
Build.md
View page @ 6d57e893
# Dependencies
\* A standard compliant c++ compiller
```plaintext
\* The system, compiler and implementations must support silent NaN floating-point values.
```
\* An MPI implementation (ideally CUDA aware)
\* SIONlib: <http://fz-juelich.de/ias/jsc/sionlib>
\* If using NVLink (cuda), Omni-Path (psm2), UCX (ucp), or IBVerbs (ibverbs)
```plaintext
\* \`minipmi\`: <https://github.com/kraused/minipmi>
```
\* If using NVLink (cuda):
```plaintext
\* A standard compliant CUDA compiller
\* Python version 3.0.0 or a derivative
```
\* If using linktest-report tool
```plaintext
\* Python 3.8.5 or above
\* numpy and matplotlib
```
\# Build procedure
* A standard compliant c++ compiller
* The system, compiler and implementations must support silent NaN floating-point values.
* An MPI implementation (ideally CUDA aware)
* SIONlib: http://fz-juelich.de/ias/jsc/sionlib
* If using NVLink (cuda), Omni-Path (psm2), UCX (ucp), or IBVerbs (ibverbs)
* `minipmi`: https://github.com/kraused/minipmi
* If using NVLink (cuda):
* A standard compliant CUDA compiller
* Python version 3.0.0 or a derivative
* If using linktest-report tool
* Python 3.8.5 or above
* numpy and matplotlib
# Build procedure
In general you have to do the following steps
1\. Install Dependencies
2\. Set-Up Environment, Pahs etc.
3\. Install \`linktest\` with \`make\`, disable unavailable transportlayers
4\. Install \`linktest-report\` via pip (Update PYTHONPATH or use virtual environments)
5\. \[Optional\] Install \`pingponganalysis\` with `make`
1. Install Dependencies
2. Set-Up Environment, Pahs etc.
3. Install `linktest` with `make`, disable unavailable transportlayers
4. Install `linktest-report` via pip (Update PYTHONPATH or use virtual environments)
5. [Optional] Install `pingponganalysis` with `make`
Quick-Start Example installing only MPI and TCP support. (Might not work on your system)
\`\`\`bash
```bash
mkdir -p install
ml GCC ParaStationMPI SIONlib SciPy-Stack # 1 + 2
cd benchmark; #
make HAVE_MINIPMI=0 PREFIX=../install clean install # 3
python -m pip install ../python/setup.py --prefix=../install # 4
\`\`\`
```
exampleBuild.sh showcases a more complicated build procedure. You can adapt it to your needs.
A file named \`pingpong_results_bin.sion\` should be produced.
A file named `pingpong_results_bin.sion` should be produced.
Some examples for JSC systems can also be found in chapter [JSC Build Examples](#JSC-Build-Examples)
\## Transport Layers
Transport Layers can be (de-)activated with \`make HAVE_=(0)1\`
# Transport Layers
Transport Layers can be (de-)activated with `make HAVE_<LAYER>=(0)1`
The following transport layers and other variables may be used:
| Layer | Variable | Default | Description |
| ----- | -------- | ------- | ----------- |
| \`mpi\` | \`HAVE_MPI\` | Enabled | MPI |
| \`tcp\` | \`HAVE_TCP\` | Enabled | TCP sockets, if deactivated, layers below will crash at runtime (see chapter [Execution](/cstao-public/linktest/-/wikis/Execution)) |
| | \`HAVE_MINIPMI\` | Enabled | If deactivated, deactivates all layers below |
| \`ibverbs\` | \`HAVE_IBVERBS\` | Enabled | Verbs-based implementation |
| \`psm2\` | \`HAVE_PSM2\` | Enabled | PSM2 (Omni-Path) |
| \`ucp\` | \`HAVE_UCP\` | Enabled | UCX (using high level api ucp) |
| \`cuda\` | \`HAVE_CUDA\` | Enabled | NVLink (node-internal) |
| | \`CUARCH\` | | Needs to be set if \`cuda\` is enable |
For \`CUARCH\` values see <https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#virtual-architecture-feature-list>
\# JSC Build Examples
| `mpi` | `HAVE_MPI` | Enabled | MPI |
| `tcp` | `HAVE_TCP` | Enabled | TCP sockets, if deactivated, layers below will crash at runtime (see chapter [Execution](Execution)) |
| | `HAVE_MINIPMI` | Enabled | If deactivated, deactivates all layers below |
| `ibverbs` | `HAVE_IBVERBS` | Enabled | Verbs-based implementation |
| `psm2` | `HAVE_PSM2` | Enabled | PSM2 (Omni-Path) |
| `ucp` | `HAVE_UCP` | Enabled | UCX (using high level api ucp) |
| `cuda` | `HAVE_CUDA` | Enabled | NVLink (node-internal) |
| | `CUARCH` | | Needs to be set if `cuda` is enable |
For `CUARCH` values see https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#virtual-architecture-feature-list
# JSC Build Examples
**JUWELS:**
| Partitition | Command |
| ----------- | ------- |
| Cluster | \`xenv -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>>\` (uses the \`generic\` system settings) |
| Booster | \`xenv -L CUDA -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>> SYSTEM="JUWELS Booster"\` |
| Cluster | `xenv -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>>` (uses the `generic` system settings) |
| Booster | `xenv -L CUDA -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>> SYSTEM="JUWELS Booster"` |
**JURECA:**
| Partitition | Command |
| ----------- | ------- |
| JURECA-DC CPU | \`xenv -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 HAVE_CUDA=0 PREFIX_DIR=<<<PREFIX_DIR>>>\` (uses the \`generic\` system settings) |
| JURECA-DC CPU | \`xenv -L CUDA -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>> SYSTEM="JURECA-DC GPU"\` |
| Booster | \`xenv -L Architecture/KNL -L Intel -L IntelMPI -L SIONlib make install HAVE_CUDA=0 PREFIX_DIR=<<<PREFIX_DIR>>>\` |
| JURECA-DC CPU | `xenv -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 HAVE_CUDA=0 PREFIX_DIR=<<<PREFIX_DIR>>>` (uses the `generic` system settings) |
| JURECA-DC CPU | `xenv -L CUDA -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>> SYSTEM="JURECA-DC GPU"` |
| Booster | `xenv -L Architecture/KNL -L Intel -L IntelMPI -L SIONlib make install HAVE_CUDA=0 PREFIX_DIR=<<<PREFIX_DIR>>>` |
**JUSUF:**
- `xenv -L CUDA -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>> CUARCH=sm_70`
\- `xenv -L CUDA -L GCC -L ParaStationMPI -L SIONlib make install HAVE_PSM2=0 PREFIX_DIR=<<<PREFIX_DIR>>> CUARCH=sm_70`
# System customizations [Advanced]
\# System customizations \[Advanced\]
Some systems require modifications on a per-node basis that are currently programmatically included in the code. To enable them please compile with `SYSTEM="The systen name"`. Example: `SYSTEM="JUWELS Booster"`.
Some systems require modifications on a per-node basis that are currently programmatically included in the code. To enable them please compile with \`SYSTEM="The systen name"\`. Example: \`SYSTEM="JUWELS Booster"\`.
The \`GenericSystem\` instance can be customized via the environment, via a wrapper script such as:
\`\`\`bash
The `GenericSystem` instance can be customized via the environment, via a wrapper script such as:
```bash
export LINKTEST_SYSTEM_NODENAME_SUFFIX="i"
["0" == "$MPI_LOCALRANKID"](/cstao-public/linktest/-/wikis/%220%22%20==%20%22%24MPI_LOCALRANKID%22)
if [[ "0" == "$MPI_LOCALRANKID" ]]
then
```plaintext
export LINKTEST_SYSTEM_HCA_NAME="mlx5_0"
export LINKTEST_SYSTEM_HCA_PORT="1"
export LINKTEST_SYSTEM_GPU="0"
```
fi
["1" == "$MPI_LOCALRANKID"](/cstao-public/linktest/-/wikis/%221%22%20==%20%22%24MPI_LOCALRANKID%22)
if [[ "1" == "$MPI_LOCALRANKID" ]]
then
```plaintext
export LINKTEST_SYSTEM_HCA_NAME="mlx5_1"
export LINKTEST_SYSTEM_HCA_PORT="1"
export LINKTEST_SYSTEM_GPU="1"
```
fi
["2" == "$MPI_LOCALRANKID"](/cstao-public/linktest/-/wikis/%222%22%20==%20%22%24MPI_LOCALRANKID%22)
if [[ "2" == "$MPI_LOCALRANKID" ]]
then
```plaintext
export LINKTEST_SYSTEM_HCA_NAME="mlx5_2"
export LINKTEST_SYSTEM_HCA_PORT="1"
export LINKTEST_SYSTEM_GPU="2"
```
fi
["3" == "$MPI_LOCALRANKID"](/cstao-public/linktest/-/wikis/%223%22%20==%20%22%24MPI_LOCALRANKID%22)
if [[ "3" == "$MPI_LOCALRANKID" ]]
then
```plaintext
export LINKTEST_SYSTEM_HCA_NAME="mlx5_3"
export LINKTEST_SYSTEM_HCA_PORT="1"
export LINKTEST_SYSTEM_GPU="3"
```
fi
exec linktest.ibverbs $\*
\`\`\`
\ No newline at end of file
exec linktest.ibverbs $*
```