Skip to content

ThreadSanitizer: data race maestro-core/maestro/event.c:162 in mstro_event_domain_destroy

In Maestro+MPI benchmark with one producer and one consumer, where the producer produces 20 CDOs and are consumed by the consumer.

Execution hangs after the consumer receives the 2nd CDO and declares the 3rd CDO and sets its size. Running Maestro from branch 103-implement-ofi-transport

gdb backtrace when the consumer hangs

#0 0x00002ac5377aa54d in __lll_lock_wait () from /usr/lib64/libpthread.so.0
#1 0x00002ac5377a71b2 in pthread_rwlock_rdlock () from /usr/lib64/libpthread.so.0
#2 0x00002ac5374a8e79 in mstro_attribute_dict_set ()
  from /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/install//lib/libmaestro.so.0
#3 0x0000000000402f5a in require_CDOs (cdos=0x7ffcd17b1510, num_CDOs=20, injector_ids=0x1cbeb10, num_injectors=1, num_attributes=0, 
  size_attributes=0, cdo_data_size=100) at omp_consumer.c:196
#4 0x0000000000401d18 in main (argc=6, argv=0x7ffcd17b1848) at core_benchmark.c:244

Thread sanitizer output

WARNING: ThreadSanitizer: data race (pid=2770)
  Write of size 8 at 0x7d100002a6d8 by main thread:
    #0 free ../../../../libsanitizer/tsan/tsan_interceptors.cc:581 (libtsan.so.0+0x000000028092)
    #1 mstro_event_domain_destroy /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/maestro/event.c:162 (libmaestro.so.0+0x000000069364)
    #2 __libc_start_main <null> (libc.so.6+0x000000022554)

  Previous atomic read of size 1 at 0x7d100002a6d8 by thread T3 (mutexes: write M94150270153547656, write M94853957595335912, write M95557645037100760):
    #0 pthread_mutex_lock ../../../../libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:3309 (libtsan.so.0+0x0000000378fe)
    #1 mstro_event_destroy /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/maestro/event.c:312 (libmaestro.so.0+0x000000069a88)

  Mutex M94150270153547656 is already destroyed.

  Mutex M94853957595335912 is already destroyed.

  Mutex M95557645037100760 is already destroyed.

  Thread T3 (tid=2924, running) created by main thread at:
    #0 pthread_create ../../../../libsanitizer/tsan/tsan_interceptors.cc:876 (libtsan.so.0+0x00000002929d)
    #1 mstro_pm_attach /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/maestro/ofi.c:2941 (libmaestro.so.0+0x00000007cb0e)
    #2 __libc_start_main <null> (libc.so.6+0x000000022554)

SUMMARY: ThreadSanitizer: data race /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/maestro/event.c:162 in mstro_event_domain_destroy

Address sanitizer output

ERROR: AddressSanitizer: SEGV on unknown address 0x60e07480007e (pc 0x2ad8b263814b bp 0x2ad8b2346260 sp 0x7ffc40c820f8 T0)
  #0 0x2ad8b263814a in __pthread_rwlock_rdlock (/usr/lib64/libpthread.so.0+0xb14a)
  #1 0x2ad8b233a568 in mstro_schema_lookup_symbol /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/attributes/maestro-schema.c:1534
  #2 0x2ad8b233a568 in mstro_schema_lookup_symbol /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/attributes/maestro-schema.c:1522
  #3 0x2ad8b233a568 in mstro_attribute_dict_set /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/attributes/maestro-schema.c:2384
  #4 0x403f57 in require_CDOs /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/examples/omp_consumer.c:196
  #5 0x402497 in main /users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/examples/core_benchmark.c:244
  #6 0x2ad8b286b554 in __libc_start_main (/usr/lib64/libc.so.6+0x22554)
  #7 0x401a88 (/users/staff/dmi-dmi/abdelazi/workspace/Maestro/ofi_transport/maestro-core/examples/core_bench.o+0x401a88)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/lib64/libpthread.so.0+0xb14a) in __pthread_rwlock_rdlock