Skip to content
Snippets Groups Projects
Select Git revision
  • main
  • devel default
  • 107-compilation-error-when-building-maestro-core-on-m1-apple-processors
  • 108-implement-cpu-id-query-for-apple-m1-hardware
  • 58-scripting-interface-to-maestro-core
  • 101-need-ci-test-using-installed-maestro
  • 57-sphinx-documentation
  • 105-memory-leak-in-pm-message-envelope-handling
  • 104-permit-disabling-memory-pool
  • 103-liberl-installation-issue-on-devel
  • 94-maestro-rdma-transport-ignores-max_msg_size-2
  • 102-possible-race-in-check_pm_redundant_interlock-test
  • 97-check-if-shm-provider-can-be-enabled-after-libfabric-1-14-is-in-our-tree-2
  • 100-include-maestro-attributes-h-cannot-include-mamba-header-from-deps-path
  • 97-check-if-shm-provider-can-be-enabled-after-libfabric-1-14-is-in-our-tree
  • 17-job-failed-282354-needs-update-of-mio-interface-and-build-rules
  • 96-test-libfabric-update-to-1-13-or-1-14
  • feature/stop-telemetry-after-all-left
  • 94-maestro-rdma-transport-ignores-max_msg_size
  • 93-improve-performance-of-mstro_attribute_val_cmp_str
  • v0.3_rc1
  • maestro_d65
  • d65_experiments_20211113
  • v0.2
  • v0.2_rc1
  • d3.3
  • d3.3-review
  • d5.5
  • d5.5-review
  • v0.1
  • d3.2
  • d3.2-draft
  • v0.0
33 results

showroom.rst

Blame
    • showroom.rst 4.21 KiB

    Showroom

    "What can I use Maestro Core for?"

    Overview

    Workflow applications using Maestro Core offer and demand CDOs to the :ref:`Maestro Pool<cdo_management>`, a conceptual entity that represents the set of resources contributed to Maestro, in particular the set of offered CDOs. One typical use of Maestro Core is bypassing persistent storage for application coupling, by way of the Maestro Pool.

    :ref:`CDOs<cdo>` are basically Maestro Core currency, they contain data and metadata, including user-defined metadata plus data semantics such as data layout information and other information on the data usage the transfer scheduler may take advantage of.

    A minimal Maestro-enabled multi-app setup would consist in one producer application, one consumer application and one Pool Manager application, the latter being shipped with Maestro Core and alternatively invokable via the Maestro Core API. The :ref:`Pool Manager<pm>` is a Maestro Core-produced application that handles networking, transport scheduling and propagates pool events for inspection.

    Maestro core components overview

    :ref:`Events<events>` allow for higher-level control of the Maestro-enabled workflow, and interfacing with many useful components. In particular, they allow the implementation of data-driven Workflow Managers or more generally Execution Frameworks, meaning they can schedule jobs based on CDO availability and location which is indeed notified by events. :ref:`Data management<cdo_management>` (and :ref:`memory management<mamba>` as well) is then effectively delegated by Execution Frameworks and applications to Maestro Core.

    As an example of useful workflow components, staging and saving CDOs of choice may be implemented by users via Librarian and Archiver components, the latter relying especially on events to be notified of the availability of relevant CDOs for archiving purposes. A keep-alive proxy may allow to retain CDOs from being withdrawn from the pool, which would be before they have been properly handled by all the components that might have needed them. Maestro-enabled components natively produce logs as well as a telemetry report at finalize time, nonetheless Events allow to implement more specific telemetry/logger components, with :ref:`selectors<events>` allowing :ref:`cherry-picking<story_cherry>` of events and CDOs.

    CDOs convey a great deal of data and memory semantics. Data usage information such as CDO dependences or an eager transfer policy for instance (both yet to be implemented) helps the scheduler and thus general performance. Data layout information lets Maestro make the transformations between a given offered CDO in a given data layout L1 (say row-major) to the demanded CDO on the consumer side that may have a layout L2 (say column-major). This allows user to abstract away data layout, especially if tools / source-to-source edition fills the data layout attribute (semi) automatically. Data layouts may be distributed (WIP) and Maestro Core will soon handle the redistribution complying with producer and consumer layout and dsitribution scheme requirements.

    Anatomy of the Core Data Object

    CDO location is expressed as a so-called Memory Object, which of course can mean a certain number of layers (DRAM, GDRAM, Parallel Filesystem, Object Store, etc) and associated access methods (pointer, path, object ID, etc.), which is then used by Maestro Core to handle the transport between layers and nodes.

    Demos

    A few demonstrators are shipped with Maestro Core, which are compiled and run through

    make check
    • Local multi-threaded setup demo_mvp_d3_2: with a few parameters such as number of producer, consumer and archiver threads.
    • Multi-application setup check_pm_interlock: The interlock demo is a minimal workflow with Pool Manager and two applications, exchanging a CDO under the Pool Manager supervision, using filesystem and object store transport, RDMA is used if any transport unavailabilities.
    • Logger sample application simple_telemetry_listener
    • Archiver sample application simple_archiver