UCP timesout on JUSUF
UCP consistently fails on JUSUF. It is not that it is not supported, it just eternally hangs.
Here is the error:
timings[000] [getpart] t= 132.39775 us
srun: Job step aborted: Waiting up to 6 seconds for job step to finish.
error: *** job 80733 CANCELLED DUE TO TIME LIMIT ***
error: *** step 80733:0 CANCELLED DUE TO TIME LIMIT ***
srun: error: jsfc069: tasks 4-7: Terminated
srun: error: jsfc068: tasks 0-3: Terminated
Here is the command:
srun --ntasks=8 --distribution=block:block:block:pack Compile/benchmark/linktest --mode ucp --num-warmup-messages 2 --num-messages 4 --size-messages 1024 --num-slowest 2 --output linktest_ucp_2nx4c.sion --no-sion-file
This module combination (among all other possible non-OpenMPI) results in this error:
module load Stages/2020
module load GCC/9.3.0
module load ParaStationMPI/5.4.7-1
module load CUDA/11.0
module load SIONlib/1.7.6
OpenMPI has other issues on JUSUF.
Edited by Max Holicki