@@ -37,10 +37,10 @@ management<cdo_management>` (and :ref:`memory management<mamba>` as well) is
...
@@ -37,10 +37,10 @@ management<cdo_management>` (and :ref:`memory management<mamba>` as well) is
then effectively delegated by Execution Frameworks and applications to Maestro
then effectively delegated by Execution Frameworks and applications to Maestro
Core.
Core.
As an example of useful workflow components, staging and saving CDOs of choice
As an example of useful workflow components, staging and archiving CDOs of
may be implemented by users via Librarian and Archiver components, the latter
choice may be implemented by users via dedicated components, archiving relying
relying especially on events to be notified of the availability of relevant
especially on events to be notified of the availability of relevant CDOs for
CDOs for archiving purposes.
archiving purposes.
A keep-alive proxy may allow to retain CDOs from being withdrawn from the pool,
A keep-alive proxy may allow to retain CDOs from being withdrawn from the pool,
which would be before they have been properly handled by all the components
which would be before they have been properly handled by all the components
that might have needed them.
that might have needed them.
...
@@ -49,28 +49,18 @@ at finalize time, nonetheless Events allow to implement more specific
...
@@ -49,28 +49,18 @@ at finalize time, nonetheless Events allow to implement more specific
telemetry/logger components, with :ref:`selectors<events>` allowing
telemetry/logger components, with :ref:`selectors<events>` allowing
:ref:`cherry-picking<story_cherry>` of events and CDOs.
:ref:`cherry-picking<story_cherry>` of events and CDOs.
CDOs convey a great deal of data and memory semantics.
CDOs may convey a considerable range of data and memory semantics.
Data usage information such as CDO dependences or an eager transfer policy for
In particular, data layout information lets Maestro make the transformations
instance *(both yet to be implemented)* helps the scheduler and thus general
between a given offered CDO in a given data layout L1 (say row-major) to the
performance.
demanded CDO on the consumer side that may have a layout L2 (say column-major).
Data layout information lets Maestro make the transformations between a given
This allows user to abstract away data layout, especially if tools or
offered CDO in a given data layout L1 (say row-major) to the demanded CDO on
source-to-source substitution fills the data layout attribute (semi)
the consumer side that may have a layout L2 (say column-major). This allows
automatically. Data layouts may be distributed and Maestro Core handle the
user to abstract away data layout, especially if tools / source-to-source
redistribution complying with producer and consumer layout and distribution
edition fills the data layout attribute (semi) automatically. Data layouts may
scheme requirements. CDO location is expressed as a so-called Memory Object,
be distributed *(WIP)* and Maestro Core *will soon* handle the redistribution
which can mean a certain number of layers (DRAM, GDRAM, NVRAM, PFS, etc.) and
complying with producer and consumer layout and dsitribution scheme
associated access methods (pointer, path, object ID, etc.), which is then used
requirements.
by Maestro Core to handle the transport between layers and nodes.
.. image:: img/core_data_object.png
:alt: Anatomy of the Core Data Object
CDO location is expressed as a so-called Memory Object, which of course can mean a certain number of layers (DRAM, GDRAM, Parallel Filesystem, Object Store, etc) and associated access methods (pointer, path, object ID, etc.), which is then used by Maestro Core to handle the transport between layers and nodes.