SIONlib issueshttps://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues2022-08-02T09:59:01+02:00https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/233Version 1.7.7 still reports as 1.7.6 & the SIONlib 32-bit Endianness identifi...2022-08-02T09:59:01+02:00Max HolickiVersion 1.7.7 still reports as 1.7.6 & the SIONlib 32-bit Endianness identification flag is 0Morning, could someone try generating a 1.7.7 SIONlib file please and have a look at the binary header. It looks like version 1.7.7:
1.) Does not set the endianness identification bytes.
2.) Still has the version set as 1.7.6.
I kind of...Morning, could someone try generating a 1.7.7 SIONlib file please and have a look at the binary header. It looks like version 1.7.7:
1.) Does not set the endianness identification bytes.
2.) Still has the version set as 1.7.6.
I kind of need those 4 bytes to reliably figure out if the program is little, big or mixed endian.
When I do module load SIONlib/1.7.7 I get this incorrect behavior in produced files.https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/232Memory leak when opening files in read only2022-03-26T12:10:51+01:00Max HolickiMemory leak when opening files in read onlyWhen sanitizing memory access for Linktest we noticed that reading from a SION file with SIONlib 1.7.7 causes a memory leak.
```
Direct leak of 1024 byte(s) in 1 object(s) allocated from:
#0 0x563dbe582378 in __interceptor_malloc (/h...When sanitizing memory access for Linktest we noticed that reading from a SION file with SIONlib 1.7.7 causes a memory leak.
```
Direct leak of 1024 byte(s) in 1 object(s) allocated from:
#0 0x563dbe582378 in __interceptor_malloc (/home/holicki/linktest/SIONtest.exe+0xf5378)
#1 0x563dbe5e0b33 in _sion_get_multi_filename /home/holicki/linktest/SIONlib/build-linux-gomp-openmpi/build/lib/sion_internal.c:915
#2 0x563dbe5c7314 in sion_generic_paropen /home/holicki/linktest/SIONlib/build-linux-gomp-openmpi/build/parlib/sion_generic.c:529
#3 0x563dbe5c40a7 in sion_paropen_mpi /home/holicki/linktest/SIONlib/build-linux-gomp-openmpi/build/parlib/sion_mpi_gen.c:205
```
It has to do with the last parameter of `sion_generic_paropen()`, which is a `char` pointer. The memory that this pointer points to is not deleted. If this is the users responsibility then this should be stated explicitly in the documentation. Otherwise SIONlib should clean this up when closing the file.
Note this problem does not always occur when writing.
This is still present in the newer SIONlib versions.Tom RidleyTom Ridleyhttps://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/228Integrate gcov(r)2021-05-26T10:11:05+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deIntegrate gcov(r)Specify `CTEST_COVERAGE_COMMAND`? Bonus points for making GitLab pick up the coverage metric.Specify `CTEST_COVERAGE_COMMAND`? Bonus points for making GitLab pick up the coverage metric.https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/226Build documentation through CMake2021-05-06T21:17:53+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deBuild documentation through CMakeCMake includes a `FindDoxygen` module that provides a convenience function `doxygen_add_docs()`. Use it!CMake includes a `FindDoxygen` module that provides a convenience function `doxygen_add_docs()`. Use it!https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/224OMP and OMPI use OpenMP locks incorrectly2021-02-01T19:13:30+01:00Benedikt Steinbuschb.steinbusch@fz-juelich.deOMP and OMPI use OpenMP locks incorrectlyIn `sion_paropen_omp` and `sion_paropen_ompi`, user defined locks based on OpenMP's `omp_lock_t` are registered via `sion_lock_register_lock_callbacks`. The lock instance is initialized in a global variable and while the access is "prote...In `sion_paropen_omp` and `sion_paropen_ompi`, user defined locks based on OpenMP's `omp_lock_t` are registered via `sion_lock_register_lock_callbacks`. The lock instance is initialized in a global variable and while the access is "protected" against data races by a `master` construct, no effort is made to protect against repeated initialization. (And also `omp_destroy_lock` is never called.) As a consequence, any program that calls `paropen_omp` or `paropen_ompi` more than once does not conform to the OpenMP specification.https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/223Tool version symbol renaming mechanism vs. different name mangling schemes2021-02-01T07:57:04+01:00Benedikt Steinbuschb.steinbusch@fz-juelich.deTool version symbol renaming mechanism vs. different name mangling schemesThe symbol renaming mechanism for the "tool" version relies on the fact that function names become symbol names, e.g. the function `sion_close` becomes a symbol `sion_close`. This does not work on platforms where this is not the case, e....The symbol renaming mechanism for the "tool" version relies on the fact that function names become symbol names, e.g. the function `sion_close` becomes a symbol `sion_close`. This does not work on platforms where this is not the case, e.g. Darwin, where an underscore is prepended, `_sion_close`.https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/222Tool version symbol renaming mechanism vs. ifdef2021-02-01T07:46:43+01:00Benedikt Steinbuschb.steinbusch@fz-juelich.deTool version symbol renaming mechanism vs. ifdefThe symbol renaming mechanism used to construct the "tool" version of SIONlib builds a regular version of the library and then works on a list of symbols produced via `nm`. This can miss symbols which are only available when a certain op...The symbol renaming mechanism used to construct the "tool" version of SIONlib builds a regular version of the library and then works on a list of symbols produced via `nm`. This can miss symbols which are only available when a certain option is turned on, e.g. IME Native integration.
Possible workaround: always include all functions, only `ifdef` function bodies?https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/220Tests buddy__{1,2} and fitest_{1,2} slow on CI2021-01-31T16:52:16+01:00Benedikt Steinbuschb.steinbusch@fz-juelich.deTests buddy__{1,2} and fitest_{1,2} slow on CIAlso in my local docker, so in both cases tests are running on docker in virtualisation. Native runs on macOS seem to be OK.Also in my local docker, so in both cases tests are running on docker in virtualisation. Native runs on macOS seem to be OK.https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/187Code reuse and composability2021-01-25T11:36:46+01:00Benedikt Steinbuschb.steinbusch@fz-juelich.deCode reuse and composabilityThe implementation of SIONlib misses a lot of opportunities for making code reusable and composable. E.g. `paropen_mapped` (n processes all open an individual, arbitrary number of logical files) is a generalisation of `paropen` (n proces...The implementation of SIONlib misses a lot of opportunities for making code reusable and composable. E.g. `paropen_mapped` (n processes all open an individual, arbitrary number of logical files) is a generalisation of `paropen` (n processes all open one logical file), but `paropen` is not implemented in terms of `paropen_mapped`. As a result, many features are not integrated well with one another, for example:
- there is no collective version of `sion_fwrite_key` and `sion_fread_key` and thus also no integration with buddy checkpointing
- `paropen_mapped` does not support buddy checkpointing
- `sion_dup` is not implemented for all modes of opening files
- `sion_seek` is not implemented for all modes of opening files and behaves differently depending on how the file was opened (see #177)
- `sion_open` does not support multiple physical files
- ...https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/210Duplication of information in generic API2021-08-25T15:20:30+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deDuplication of information in generic API`sion_generic_paropen` and `sion_generic_paropen_mapped` both have a "commgroup" argument ("global communicator") as well as two arguments `grank` ("global rank of process / calling task") and `gsize` ("size of global communicator"). The...`sion_generic_paropen` and `sion_generic_paropen_mapped` both have a "commgroup" argument ("global communicator") as well as two arguments `grank` ("global rank of process / calling task") and `gsize` ("size of global communicator"). The information contained in the latter two should be contained somehow in the first and indeed, every specific API in SIONlib (MPI, OpenMP, Hybrid) has fields in its "commgroup" structure that contain exactly the values passed in `grank` and `gsize`. Is it necessary to keep all three arguments or should there generic interface be extended to allow the user to register callbacks for inspecting the "commgroup" for rank and size on the generic level?2.0.0https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/212Make API creation foolproof2021-08-25T15:20:30+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deMake API creation foolproof`sion_generic_create_api` creates an API that is in a completely invalid state and the various `sion_generic_register...` functions all register a single callback function, leaving the API again in an invalid state until all necessary ca...`sion_generic_create_api` creates an API that is in a completely invalid state and the various `sion_generic_register...` functions all register a single callback function, leaving the API again in an invalid state until all necessary callbacks have been registered. Why not have `sion_generic_create_api` take all necessary callbacks as an argument and create a valid API descriptor in a single step?
As there seem to be different levels of capability that require fewer or more callbacks to be defined, there could be several `create` functions, one per level or, a single `create` function that accepts `NULL` for optional callbacks and sets the correct capability level.2.0.0https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/211sion_open_rank does not support key-value files, user-specified endianness(?)2019-12-02T16:03:44+01:00Benedikt Steinbuschb.steinbusch@fz-juelich.desion_open_rank does not support key-value files, user-specified endianness(?)https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/213Deprecate or remove sion_get_current_position etc.2021-08-25T15:20:29+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deDeprecate or remove sion_get_current_position etc.These functions expose internal fields, remove in favor of `sion_tell` (and possibly more functions to be implemented).These functions expose internal fields, remove in favor of `sion_tell` (and possibly more functions to be implemented).2.0.0-rc.4https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/203Adapt python bindings to 2.0 interface2019-12-02T16:03:44+01:00Benedikt Steinbuschb.steinbusch@fz-juelich.deAdapt python bindings to 2.0 interfaceOtherwise disable for now and adapt after 2.0.0 release.Otherwise disable for now and adapt after 2.0.0 release.2.0.0https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/204Document all new open functions and the accompanying option constructors and ...2021-08-25T15:20:29+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deDocument all new open functions and the accompanying option constructors and setters2.0.0-rc.4https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/205Write a guide for the 1.0 to 2.0 transition2021-08-25T15:20:30+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deWrite a guide for the 1.0 to 2.0 transitionDocument the following:
- changes in open functions
- option structs for advanced use cases
- split up seek function
- reading and writing "without" chunksDocument the following:
- changes in open functions
- option structs for advanced use cases
- split up seek function
- reading and writing "without" chunks2.0.0-rc.4https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/201Delete 1.X open functions, remove _with_options suffix from 2.x open functions2021-08-25T15:20:29+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deDelete 1.X open functions, remove _with_options suffix from 2.x open functions2.0.0-rc.4https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/202Re-design and implement Fortran bindings2021-08-25T15:20:30+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deRe-design and implement Fortran bindingsOtherwise disable for now and design and implement after 2.0.0 release.Otherwise disable for now and design and implement after 2.0.0 release.2.0.0https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/207Adapt all examples to new interface2021-08-25T15:20:30+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deAdapt all examples to new interfaceOr move examples to separate repository and adapt post 2.0.0 releaseOr move examples to separate repository and adapt post 2.0.0 release2.0.0https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues/206Design and implement forward-compatibility CLI utility2021-08-25T15:20:30+02:00Benedikt Steinbuschb.steinbusch@fz-juelich.deDesign and implement forward-compatibility CLI utilityReads a SIONlib container with file format 6 and writes a file format 5 containerReads a SIONlib container with file format 6 and writes a file format 5 container2.0.0