Why do bisection runs repeat measurements but do not do other measurements?
Linktest in bisection mode repeats the last set of measurements twice. This should not be the case, also it does not fully do a bisection run. It also repeats 2->9 in step 0,2,4. This is definitely bugged.
In the following serial log file the last set of measurements are down twice:
Executing following linktest:
with command:
srun --ntasks=10 --distribution=block:block:block:pack linktest --mode mpi --num-warmup-messages 3 --num-messages 10 --size-messages 16777216 --num-slowest 0 --output linktest_jureca_mpi_1nx10c_16777216B_bisect_serial.sion --bisect --serial-tests
------------------- Linktest Args ------------------------
Virtual-Cluster Implementation: mpi
Message length: 16777216 B
Number of Messages: 10
Number of Messages. (Warmup): 3
Communication Pattern: Bisection
use gpus: No
mixing pe order: No
serial test only: Yes
max serial retest: 0
write protocol (SION): Yes, funneled
output file: "linktest_jureca_mpi_1nx10c_16777216B_bisect_serial.sion"
----------------------------------------------------------
Starting Test of all connections:
---------------------------------
Serial PingPong for step 0:
0-> 6: 2.2227537 ms ( 7.0296 GiB/s) (l=0)
1-> 5: 2.2069611 ms ( 7.0799 GiB/s) (l=1)
2-> 9: 3.6102303 ms ( 4.3280 GiB/s) (l=2)
3-> 8: 3.8812451 ms ( 4.0258 GiB/s) (l=3)
4-> 7: 2.2088442 ms ( 7.0738 GiB/s) (l=4)
5-> 1: 2.1785425 ms ( 7.1722 GiB/s) (l=5)
6-> 0: 2.3167234 ms ( 6.7444 GiB/s) (l=6)
7-> 4: 2.4123942 ms ( 6.4770 GiB/s) (l=7)
8-> 3: 4.1455479 ms ( 3.7691 GiB/s) (l=8)
9-> 2: 3.6153788 ms ( 4.3218 GiB/s) (l=9)
avg: 2.87986 ms ( 5.4256 GiB/s) min: 2.17854 ms ( 7.1722 GiB/s) max: 4.14555 ms ( 3.7691 GiB/s) sum bw= 29.0108 GiB/s
Analyse Summary: min. 2.1785 ms ( 7.172 GiB/s) max. 4.1455 ms ( 3.769 GiB/s) avg. infe-2147483646 s ( 0.000 B/s)
Timing Summary: 0 step(s) required 1.31522 s ( infe-2147483646 s/step). 9 step(s) remain. Estimated time remaining: infe-2147483646 s
Serial PingPong for step 1:
0-> 9: 3.6299179 ms ( 4.3045 GiB/s) (l=0)
1-> 8: 3.6031393 ms ( 4.3365 GiB/s) (l=1)
2-> 7: 2.2306107 ms ( 7.0048 GiB/s) (l=2)
3-> 6: 2.2527603 ms ( 6.9359 GiB/s) (l=3)
4-> 5: 2.2818903 ms ( 6.8474 GiB/s) (l=4)
5-> 4: 2.2772388 ms ( 6.8614 GiB/s) (l=5)
6-> 3: 2.2015891 ms ( 7.0971 GiB/s) (l=6)
7-> 2: 2.2284041 ms ( 7.0117 GiB/s) (l=7)
8-> 1: 3.5516211 ms ( 4.3994 GiB/s) (l=8)
9-> 0: 3.6267739 ms ( 4.3082 GiB/s) (l=9)
avg: 2.78839 ms ( 5.6036 GiB/s) min: 2.20159 ms ( 7.0971 GiB/s) max: 3.62992 ms ( 4.3045 GiB/s) sum bw= 29.5535 GiB/s
Analyse Summary: min. 2.1785 ms ( 7.172 GiB/s) max. 4.1455 ms ( 3.769 GiB/s) avg. 5.6683 ms ( 2.757 GiB/s)
Timing Summary: 1 step(s) required 2.63097 s ( 2.63097 s/step). 8 step(s) remain. Estimated time remaining: 21.04772 s
Serial PingPong for step 2:
0-> 8: 3.6502759 ms ( 4.2805 GiB/s) (l=0)
1-> 6: 2.3381675 ms ( 6.6826 GiB/s) (l=1)
2-> 9: 3.6226743 ms ( 4.3131 GiB/s) (l=2)
3-> 5: 2.3899596 ms ( 6.5378 GiB/s) (l=3)
4-> 7: 2.2107821 ms ( 7.0676 GiB/s) (l=4)
5-> 3: 2.3414595 ms ( 6.6732 GiB/s) (l=5)
6-> 1: 2.3402370 ms ( 6.6767 GiB/s) (l=6)
7-> 4: 2.6078368 ms ( 5.9916 GiB/s) (l=7)
8-> 0: 3.6567684 ms ( 4.2729 GiB/s) (l=8)
9-> 2: 3.6189208 ms ( 4.3176 GiB/s) (l=9)
avg: 2.87771 ms ( 5.4297 GiB/s) min: 2.21078 ms ( 7.0676 GiB/s) max: 3.65677 ms ( 4.2729 GiB/s) sum bw= 28.4067 GiB/s
Analyse Summary: min. 2.1785 ms ( 7.172 GiB/s) max. 4.1455 ms ( 3.769 GiB/s) avg. 4.2730 ms ( 3.657 GiB/s)
Timing Summary: 2 step(s) required 3.93725 s ( 1.96862 s/step). 7 step(s) remain. Estimated time remaining: 13.78036 s
Serial PingPong for step 3:
0-> 7: 2.2098016 ms ( 7.0708 GiB/s) (l=0)
1-> 8: 3.5538111 ms ( 4.3967 GiB/s) (l=1)
2-> 5: 2.3487245 ms ( 6.6525 GiB/s) (l=2)
3-> 6: 2.2021736 ms ( 7.0953 GiB/s) (l=3)
4-> 9: 3.5345285 ms ( 4.4207 GiB/s) (l=4)
5-> 2: 2.3216334 ms ( 6.7302 GiB/s) (l=5)
6-> 3: 2.1950826 ms ( 7.1182 GiB/s) (l=6)
7-> 0: 2.2150667 ms ( 7.0540 GiB/s) (l=7)
8-> 1: 3.5824277 ms ( 4.3616 GiB/s) (l=8)
9-> 4: 3.7496877 ms ( 4.1670 GiB/s) (l=9)
avg: 2.79129 ms ( 5.5978 GiB/s) min: 2.19508 ms ( 7.1182 GiB/s) max: 3.74969 ms ( 4.1670 GiB/s) sum bw= 29.5334 GiB/s
Analyse Summary: min. 2.1785 ms ( 7.172 GiB/s) max. 4.1455 ms ( 3.769 GiB/s) avg. 3.7791 ms ( 4.135 GiB/s)
Timing Summary: 3 step(s) required 5.23049 s ( 1.74350 s/step). 6 step(s) remain. Estimated time remaining: 10.46099 s
Serial PingPong for step 4:
0-> 6: 2.2092771 ms ( 7.0724 GiB/s) (l=0)
1-> 5: 2.2092316 ms ( 7.0726 GiB/s) (l=1)
2-> 9: 3.6150933 ms ( 4.3222 GiB/s) (l=2)
3-> 8: 3.6002438 ms ( 4.3400 GiB/s) (l=3)
4-> 7: 2.2125806 ms ( 7.0619 GiB/s) (l=4)
5-> 1: 2.2542573 ms ( 6.9313 GiB/s) (l=5)
6-> 0: 2.2103431 ms ( 7.0690 GiB/s) (l=6)
7-> 4: 2.2326407 ms ( 6.9984 GiB/s) (l=7)
8-> 3: 3.6009623 ms ( 4.3391 GiB/s) (l=8)
9-> 2: 3.6217768 ms ( 4.3142 GiB/s) (l=9)
avg: 2.77664 ms ( 5.6273 GiB/s) min: 2.20923 ms ( 7.0726 GiB/s) max: 3.62178 ms ( 4.3142 GiB/s) sum bw= 29.7606 GiB/s
Analyse Summary: min. 2.1785 ms ( 7.172 GiB/s) max. 4.1455 ms ( 3.769 GiB/s) avg. 3.5285 ms ( 4.428 GiB/s)
Timing Summary: 4 step(s) required 6.53861 s ( 1.63465 s/step). 5 step(s) remain. Estimated time remaining: 8.17326 s
Linktest Timing Results - Iteration 1:
RESULT: Min Time: 2.17854248 ms ( 7.172 GiB/s)
RESULT: Max Time: 4.14554789 ms ( 3.769 GiB/s)
RESULT: Avg Time: 1.56821103 ms ( 9.964 GiB/s)