Skip to content
Snippets Groups Projects

Changes

Page history
  • New page
  • Templates
    • Update LinkTest SIONlib File Format authored by Max Holicki's avatar Max Holicki
    Show whitespace changes
    Inline Side-by-side
    LinkTest-SIONlib-File-Format.md
    View page @ 510a4934
    ...@@ -22,16 +22,16 @@ In the following we will describe the binary structure of each of the different ...@@ -22,16 +22,16 @@ In the following we will describe the binary structure of each of the different
    # File Header # File Header
    | Name | ID |Start [B] | End [B] | Size [B] | Fixed Value | Description | | Name | ID |Start [B] | End [B] | Size [B] | Fixed Value | Description |
    |-------------------------|----|----------|---------|----------|---------------|---------------------| |-------------------------|----|----------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
    | SIONlib Header | | 0 | 0+a | a | | SIONlib file header | | SIONlib Header | | 0 | 0+a | a | | SIONlib file header |
    | LinkTest ID String | | 0+a | 5+a | 5 | "LKTST" | LinkTest ASCII-character tag identifying the start of the LinkTest portion of the SION file. | | LinkTest ID String | | 0+a | 5+a | 5 | "LKTST" | LinkTest ASCII-character tag identifying the start of the LinkTest portion of the SION file. |
    | LinkTest Major Version | | 5+a | 9+a | 4 | | 32-bit unsigned integer identifying the LinkTest major version number. | | LinkTest Major Version | | 5+a | 9+a | 4 | | 32-bit unsigned integer identifying the LinkTest major version number. |
    | LinkTest Minor Version | | 9+a | 13+a | 4 | | 32-bit unsigned integer identifying the LinkTest minor version number. | | LinkTest Minor Version | | 9+a | 13+a | 4 | | 32-bit unsigned integer identifying the LinkTest minor version number. |
    | LinkTest Patchlevel | | 13+a | 17+a | 4 | | 32-bit unsigned integer identifying the LinkTest patchlevel number. | | LinkTest Patchlevel | | 13+a | 17+a | 4 | | 32-bit unsigned integer identifying the LinkTest patchlevel number. |
    | LinkTest GitHash | | 17+a | 58+a | 41 | | 41-byte null-terminated ASCII-character git hash identifying the commit used to generated the data. | | LinkTest GitHash | | 17+a | 58+a | 41 | | 41-byte null-terminated ASCII-character git hash identifying the commit used to generated the data. |
    | LinkTest Mode Length | b | 58+a | 62+a | 4 | | 32-bit unsigned interger specifying the length of the following string indicating the transport layer/mode/virtual-cluster implementation used for the LinkTest run. | | LinkTest Mode Length | b | 58+a | 62+a | 4 | | 32-bit unsigned integer specifying the length of the following string indicating the transport layer/mode/virtual-cluster implementation used for the LinkTest run. |
    | LinkTest Mode | | 62+a | 62+a+b | b | | Null-terminated ASCII character array identifying the transport layer/mode/virtual-cluster implementation used for the LinkTest run. | | LinkTest Mode | | 62+a | 62+a+b | b | | Null-terminated ASCII character array identifying the transport layer/mode/virtual-cluster implementation used for the LinkTest run. |
    | All-To-All Flag | A | 62+a+b | 63+a+b | 1 | | 8-bit integer that if non-zero indicates that all-to-all testing was performed. | | All-To-All Flag | c | 62+a+b | 63+a+b | 1 | | 8-bit integer that if non-zero indicates that all-to-all testing was performed. |
    | Bidirectional Flag | | 63+a+b | 64+a+b | 1 | | 8-bit integer that if non-zero indicates that bidirectional testing was performed. | | Bidirectional Flag | | 63+a+b | 64+a+b | 1 | | 8-bit integer that if non-zero indicates that bidirectional testing was performed. |
    | Unidirectional Flag | | 64+a+b | 65+a+b | 1 | | 8-bit integer that if non-zero indicates that unidirectional testing was performed. | | Unidirectional Flag | | 64+a+b | 65+a+b | 1 | | 8-bit integer that if non-zero indicates that unidirectional testing was performed. |
    | Bisection Flag | | 65+a+b | 66+a+b | 1 | | 8-bit integer that if non-zero indicates that bisection testing was performed. | | Bisection Flag | | 65+a+b | 66+a+b | 1 | | 8-bit integer that if non-zero indicates that bisection testing was performed. |
    ...@@ -48,11 +48,11 @@ In the following we will describe the binary structure of each of the different ...@@ -48,11 +48,11 @@ In the following we will describe the binary structure of each of the different
    | Message Size | | 83+a+b | 91+a+b | 8 | | 64-bit unsigned integer indicating the size of the messages passed between partners during measurements and warm up. | | Message Size | | 83+a+b | 91+a+b | 8 | | 64-bit unsigned integer indicating the size of the messages passed between partners during measurements and warm up. |
    | Num. Warm-up Messages | | 91+a+b | 99+a+b | 8 | | 64-bit unsigned integer indicating the number of warm-up messages passed between partners during measurements. | | Num. Warm-up Messages | | 91+a+b | 99+a+b | 8 | | 64-bit unsigned integer indicating the number of warm-up messages passed between partners during measurements. |
    | Collect P-Num | | 99+a+b |107+a+b | 8 | | Depreciated! | | Collect P-Num | | 99+a+b |107+a+b | 8 | | Depreciated! |
    | Num. Serial Retests | c | 107+a+b |115+a+b | 8 | | 64-bit unsigned integer indicating the number of serial retests of the worst connections. | | Num. Serial Retests | d | 107+a+b |115+a+b | 8 | | 64-bit unsigned integer indicating the number of serial retests of the worst connections. |
    | Num. Multiple Buffers | | 115+a+b |123+a+b | 8 | | 64-bit unsigned integer indicating the number of buffers used to store messages in a rolling fashion. | | Num. Multiple Buffers | | 115+a+b |123+a+b | 8 | | 64-bit unsigned integer indicating the number of buffers used to store messages in a rolling fashion. |
    | Buf. Radomization Seed | | 123+a+b |131+a+b | 8 | | The 64-bit seed used for buffer randomization. | | Buf. Randomization Seed | | 123+a+b |131+a+b | 8 | | The 64-bit seed used for buffer randomization. |
    | Num. Randomized Tasks | | 131+a+b |139+a+b | 8 | | The number of test iterations with randomized tasks. | | Num. Randomized Tasks | | 131+a+b |139+a+b | 8 | | The number of test iterations with randomized tasks. |
    | Task Radomization Seed | | 139+a+b |147+a+b | 8 | | The 64-bit seed used for task randomization. | | Task Randomization Seed | | 139+a+b |147+a+b | 8 | | The 64-bit seed used for task randomization. |
    # Data Chunk # Data Chunk
    ...@@ -62,10 +62,10 @@ If multiple permutations of randomized ranks were tests this data is stored back ...@@ -62,10 +62,10 @@ If multiple permutations of randomized ranks were tests this data is stored back
    ``` ```
    Data Chunk Header Data Chunk Header
    Data Chunk Data - Permuation 1 Data Chunk Data - Permutation 1
    Data Chunk Data - Permuation 2 Data Chunk Data - Permutation 2
    ... ...
    Data Chunk Data - Permuation M Data Chunk Data - Permutation M
    Data Chunk Footer Data Chunk Footer
    ``` ```
    ...@@ -76,52 +76,52 @@ Each Data Chunk start with a small header indicate where the rank ran. For the n ...@@ -76,52 +76,52 @@ Each Data Chunk start with a small header indicate where the rank ran. For the n
    | Name | ID |Start [B] | End [B] | Size [B] | Description | | Name | ID |Start [B] | End [B] | Size [B] | Description |
    |-------------------------|----|----------|---------|----------|-----------------------------------------------------------| |-------------------------|----|----------|---------|----------|-----------------------------------------------------------|
    | Hostname Length | d | 0 | 4 | 4 | Length of the hostname on which the rank ran. | | Hostname Length | d | 0 | 4 | 4 | Length of the hostname on which the rank ran. |
    | Hostname | | 4 | 4+d | d | The null-terminated ASCII hostname on which the rank ran. | | Hostname | | 4 | 4+e | d | The null-terminated ASCII hostname on which the rank ran. |
    | Core ID | | 4+d | 8+d | 4 | 32-bit integer indicating the core on which the rank ran. | | Core ID | | 4+e | 8+e | 4 | 32-bit integer indicating the core on which the rank ran. |
    ### Non-Zero Rank ### Non-Zero Rank
    | Name | ID |Start [B] | End [B] | Size [B] | Description | | Name | ID |Start [B] | End [B] | Size [B] | Description |
    |-------------------------|----|----------|---------|----------|-----------------------------------------------------------| |-------------------------|----|----------|---------|----------|-----------------------------------------------------------|
    | LinkTest ID String | | 0 | 5 | 5 | LinkTest "LKTST" ASCII-character tag. | | LinkTest ID String | | 0 | 5 | 5 | LinkTest "LKTST" ASCII-character tag. |
    | Hostname Length | d | 5 | 9 | 4 | Length of the hostname on which the rank ran. | | Hostname Length | d | 5 | 9 | 4 | Length of the hostname on which the rank ran. |
    | Hostname | | 9 | 9+d | d | The null-terminated ASCII hostname on which the rank ran. | | Hostname | | 9 | 9+e | d | The null-terminated ASCII hostname on which the rank ran. |
    | Core ID | | 9+d | 13+d | 4 | 32-bit integer indicating the core on which the rank ran. | | Core ID | | 9+e | 13+e | 4 | 32-bit integer indicating the core on which the rank ran. |
    ## Data Chunk Data ## Data Chunk Data
    After the Data Chunk header comes the recorded timing data, with rank 0 including additional aggregated data. If multiple randomized permutations of the ranks are used then the recorded timing data of each permuation follow directly after each other. Let us first present the table for rank 0 and then the table for the other ranks. After the Data Chunk header comes the recorded timing data, with rank 0 including additional aggregated data. If multiple randomized permutations of the ranks are used then the recorded timing data of each permutation follow directly after each other. Let us first present the table for rank 0 and then the table for the other ranks.
    ### Rank 0 ### Rank 0
    | Name | Start [B] | End [B] | Size [B] | Logical Expression | Description | | Name | Start [B] | End [B] | Size [B] | Logical Expression | Description |
    |-------------------------|-----------------------|-----------------------|----------|--------------------|---------------------| |-------------------------|-------------------|-------------------|----------|--------------------|-------------------------------------------------------------------------------------------------------|
    | Start Time | 8+d | 40+d | 32 | | Null-terminated ASCII character array stating the time when this test started. | | Start Time | 0 | 32 | 32 | | Null-terminated ASCII character array stating the time when this test started. |
    | Minimum Time | 40+d | 48+d | 8 | | Minimum recorded time for the test as a double-precision number. | | Minimum Time | 32 | 40 | 8 | | Minimum recorded time for the test as a double-precision number. |
    | Average Time | 48+d | 56+d | 8 | | Average recorded time for the test as a double-precision number. | | Average Time | 40 | 48 | 8 | | Average recorded time for the test as a double-precision number. |
    | Maximum Time | 56+d | 64+d | 8 | | Maximum recorded time for the test as a double-precision number. | | Maximum Time | 48 | 56 | 8 | | Maximum recorded time for the test as a double-precision number. |
    | All-To-All Minimum Time | 64+d | 64+d+1e | e | If A: e=8 else e=0 | Minimum recorded all-to-all time for the test as a double-precision number. | | All-To-All Minimum Time | 56 | 56+1f | f | If c: f=8 else f=0 | Minimum recorded all-to-all time for the test as a double-precision number. |
    | All-To-All Average Time | 64+d+1e | 64+d+2e | e | | Average recorded all-to-all time for the test as a double-precision number. | | All-To-All Average Time | 56+1f | 56+2f | f | | Average recorded all-to-all time for the test as a double-precision number. |
    | All-To-All Maximum Time | 64+d+2e | 64+d+3e | e | | Maximum recorded all-to-all time for the test as a double-precision number. | | All-To-All Maximum Time | 56+2f | 56+3f | f | | Maximum recorded all-to-all time for the test as a double-precision number. |
    | Timing Data | 64+d+3e | 64+d+3e+8(N-1) | 8(N-1) | | Recorded timing data for the rank as an array of double-precision numbers. | | Timing Data | 56+3f | 56+3f+8(N-1) | 8(N-1) | | Recorded timing data for the rank as an array of double-precision numbers. |
    | Access Pattern | 64+d+3e+8(N-1) | 64+d+3e+16(N-1) | 8(N-1) | | Access pattern for generating the recorded data for the rank as an array of 64-bit unsigned integers. | | Access Pattern | 56+3f+8(N-1) | 56+3f+16(N-1) | 8(N-1) | | Access pattern for generating the recorded data for the rank as an array of 64-bit unsigned integers. |
    | All-To-All Timing | 64+d+3e+16(N-1) | 64+d+3e+16(N-1)+f | f | If A: f=8 else f=0 | Recorded all-to-all time for the rank as a double-precision number. | | All-To-All Timing | 56+3f+16(N-1) | 56+4f+16(N-1) | f | | Recorded all-to-all time for the rank as a double-precision number. |
    | Retested Slow Timings | 64+d+3e+16(N-1)+f | 64+d+3e+16(N-1)+f+8c | 8c | | Timings for the retested slowest connections as an array of double precision numbers. | | Retested Slow Timings | 56+4f+16(N-1) | 56+4f+16(N-1)+8d | 8d | | Timings for the retested slowest connections as an array of double precision numbers. |
    | Slowest Timings | 64+d+3e+16(N-1)+f+8c | 64+d+3e+16(N-1)+f+16c | 8c | | Timings of the slowest connections as an array of double precision numbers. | | Slowest Timings | 56+4f+16(N-1)+8d | 56+4f+16(N-1)+16d | 8d | | Timings of the slowest connections as an array of double precision numbers. |
    | Slow Sending Partners | 64+d+3e+16(N-1)+f+16c | 64+d+3e+16(N-1)+f+24c | 8c | | Rank of the partner that initiated the connections for the slow timings. | | Slow Sending Partners | 56+4f+16(N-1)+16d | 56+4f+16(N-1)+24d | 8d | | Rank of the partner that initiated the connections for the slow timings. |
    | Slow Receiving Partners | 64+d+3e+16(N-1)+f+24c | 64+d+3e+16(N-1)+f+32c | 8c | | Rank of the partner that was on the receiving end of the connections for the slow timings. | | Slow Receiving Partners | 56+4f+16(N-1)+24d | 56+4f+16(N-1)+32d | 8d | | Rank of the partner that was on the receiving end of the connections for the slow timings. |
    | End Time | 64+d+3e+16(N-1)+f+32c | 96+d+3e+16(N-1)+f+32c | 32 | | Null-terminated ASCII character array stating the time when this test finished. | | End Time | 56+4f+16(N-1)+32d | 88+4f+16(N-1)+32d | 32 | | Null-terminated ASCII character array stating the time when this test finished. |
    ### Non-Zero Rank ### Non-Zero Rank
    | Name | Start [B] | End [B] | Size [B] | Logical Expression | Description | | Name | Start [B] | End [B] | Size [B] | Logical Expression | Description |
    |-------------------|-----------|-----------|----------|--------------------|---------------------| |-------------------|-----------|-----------|----------|--------------------|-------------------------------------------------------------------------------------------------------|
    | Timing Data | 0 | 8(N-1) | 8(N-1) | | Recorded timing data for the rank as an array of double-precision numbers. | | Timing Data | 0 | 8(N-1) | 8(N-1) | | Recorded timing data for the rank as an array of double-precision numbers. |
    | Access Pattern | 8(n-1) | 16(N-1) | 8(N-1) | | Access pattern for generating the recorded data for the rank as an array of 64-bit unsigned integers. | | Access Pattern | 8(n-1) | 16(N-1) | 8(N-1) | | Access pattern for generating the recorded data for the rank as an array of 64-bit unsigned integers. |
    | All-To-All Timing | 16(N-1) | 16(N-1)+f | f | If A: f=8 else f=0 | Recorded all-to-all time for the rank as a double-precision number. | | All-To-All Timing | 16(N-1) | 16(N-1)+g | g | If A: g=8 else g=0 | Recorded all-to-all time for the rank as a double-precision number. |
    ## Data Chunk Footer ## Data Chunk Footer
    Each Data Chunk end with an "END_BLOCK" for alignment purposes. Each Data Chunk end with an "END_BLOCK" for alignment purposes.
    | Name | Start [B] | End [B] | Size [B] | Description | | Name | Start [B] | End [B] | Size [B] | Description |
    |-------------------|-----------|-----------|----------|-----------------------------------------------------------| |-------------------|-----------|-----------|----------|-----------------------------------------------------------|
    | End Block | 0 | 10 | 10 | "END_BLOCK" statement indicating the end of a Data Chunk. | | End Block | 0 | 9 | 9 | "END_BLOCK" statement indicating the end of a Data Chunk. |
    After the Data Chunk Footer there maybe unused data till the end of a SIONlib Chunk. Please see [SIONlib](https://www.fz-juelich.de/en/ias/jsc/services/user-support/jsc-software-tools/sionlib) on how to identify the SIONlib chunk size. After the Data Chunk Footer there maybe unused data till the end of a SIONlib Chunk. Please see [SIONlib](https://www.fz-juelich.de/en/ias/jsc/services/user-support/jsc-software-tools/sionlib) on how to identify the SIONlib chunk size.
    ......