Skip to content
Snippets Groups Projects
Commit 450283a2 authored by Christopher Haine's avatar Christopher Haine
Browse files

showroom + cosmetics

parent e018c5e8
No related branches found
No related tags found
No related merge requests found
Pipeline #111659 failed
......@@ -7,3 +7,8 @@
{% endblock %}
{%- block footer %}
<br>
{% endblock %}
......@@ -13,7 +13,7 @@
<div id="searchbox" style="display: none" role="search">
<div class="searchformwrapper">
<form class="search" action="{{ pathto('search') }}" method="get">
<input type="text" placeholder="Search the docs" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="text" placeholder="Search docs" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
</form>
</div>
</div>
......
......@@ -43,7 +43,8 @@ actual data to be transferred, we are going to need a pointer and a size:
&my_data_size, ...);
At this point of the CDO definition the upcoming transfer will indeed consist
of data and metadata.
of data and metadata. :ref:`User-defined<user-metadata>` and
:ref:`layout<layout>` metadata in particular may also be added.
.. _mamba:
......
......@@ -11,19 +11,14 @@ Workflow applications using Maestro Core offer and demand CDOs to the
:ref:`Maestro Pool<cdo_management>`, a conceptual entity that represents the
set of resources contributed to Maestro, in particular the set of offered CDOs.
One typical use of Maestro Core is bypassing persistent storage for application
coupling, by way of the Maestro Pool.
coupling, by way of the Maestro Pool. The :ref:`Pool Manager<pm>` is a Maestro
Core-produced application that handles networking, transport scheduling and
propagates pool events for inspection.
:ref:`CDOs<cdo>` are basically Maestro Core currency, they contain data and metadata,
including user-defined metadata plus data semantics such as data layout
information and other information on the data usage the transfer scheduler may
take advantage of.
A minimal Maestro-enabled multi-app setup would consist in one producer
application, one consumer application and one Pool Manager application, the
latter being shipped with Maestro Core and alternatively invokable via the
Maestro Core API. The :ref:`Pool Manager<pm>` is a Maestro Core-produced
application that handles networking, transport scheduling and propagates
pool events for inspection.
:ref:`CDOs<cdo>` are basically Maestro Core currency, they contain data and
metadata, including user-defined metadata plus data semantics such as data
layout information and other information on data usage the transfer scheduler
may take advantage of.
.. image:: img/maestro_core_components.png
:alt: Maestro core components overview
......@@ -32,35 +27,29 @@ pool events for inspection.
workflow, and interfacing with many useful components. In particular, they
allow the implementation of data-driven Workflow Managers or more generally
Execution Frameworks, meaning they can schedule jobs based on CDO availability
and location which is indeed notified by events. :ref:`Data
management<cdo_management>` (and :ref:`memory management<mamba>` as well) is
then effectively delegated by Execution Frameworks and applications to Maestro
Core.
As an example of useful workflow components, staging and archiving CDOs of
choice may be implemented by users via dedicated components, archiving relying
especially on events to be notified of the availability of relevant CDOs for
archiving purposes.
A keep-alive proxy may allow to retain CDOs from being withdrawn from the pool,
which would be before they have been properly handled by all the components
that might have needed them.
Maestro-enabled components natively produce logs as well as a telemetry report
at finalize time, nonetheless Events allow to implement more specific
telemetry/logger components, with :ref:`selectors<events>` allowing
:ref:`cherry-picking<story_cherry>` of events and CDOs.
CDOs may convey a considerable range of data and memory semantics.
In particular, data layout information lets Maestro make the transformations
and location which is indeed notified by events.
As an example of another useful workflow components, staging and archiving CDOs
of choice may be implemented by users via dedicated components, archiving
relying especially on events to be notified of the availability of
(possibly :ref:`cherry-picked<story_cherry>`) CDOs for archiving purposes.
In addition to :ref:`data management<cdo_management>`, Maestro Core facilitates
:ref:`memory management<mamba>`, allowing to abstract away memory layer
consideration as well as providing features such as tiling.
CDO location is expressed as a so-called Memory Object,
which can mean a certain number of layers (DRAM, GDRAM, NVRAM, PFS, etc.) and
associated access methods (pointer, path, object ID, etc.), which is then used
by Maestro Core to handle the transport between layers and nodes.
CDOs may convey a wide range of data and memory semantics. In particular,
:ref:`data layout<layout>` information lets Maestro make the transformations
between a given offered CDO in a given data layout L1 (say row-major) to the
demanded CDO on the consumer side that may have a layout L2 (say column-major).
This allows user to abstract away data layout, especially if tools or
source-to-source substitution fills the data layout attribute (semi)
automatically. Data layouts may be distributed and Maestro Core handle the
redistribution complying with producer and consumer layout and distribution
scheme requirements. CDO location is expressed as a so-called Memory Object,
which can mean a certain number of layers (DRAM, GDRAM, NVRAM, PFS, etc.) and
associated access methods (pointer, path, object ID, etc.), which is then used
by Maestro Core to handle the transport between layers and nodes.
scheme requirements.
Demos
......
Tuning
======
Make Maestro Core perform.
Have Maestro Core perform.
Architecture
------------
......@@ -20,11 +20,11 @@ Knobs
Maestro features a range of environment variables that can be played with.
* MSTRO_LOG_LEVEL
* ``MSTRO_LOG_LEVEL``
Log level (Error|Warning|Info|Debug). (Optional)
* MSTRO_LOG_MODULES
* ``MSTRO_LOG_MODULES``
Selection of modules that should log (Optional)
Typically
......@@ -38,27 +38,27 @@ Maestro features a range of environment variables that can be played with.
to record logs that would slow us down, but still want the stats report at
``mstro_finalize()`` time.
* MSTRO_LOG_COLOR
* ``MSTRO_LOG_COLOR``
Log color. (green|blue|brightblue|...) (Optional). Typically helps visually parse small logs between a couple apps.
* MSTRO_LOG_DST
* ``MSTRO_LOG_DST``
Select logging output channel. (stdout|stderr|syslog) (Optional)
* MSTRO_TRANSPORT_DEFAULT
* ``MSTRO_TRANSPORT_DEFAULT``
Which transport method to choose by default. (RDMA|GFS|MIO) (Optional)
* MSTRO_TRANSPORT_GFS_DIR
* ``MSTRO_TRANSPORT_GFS_DIR``
Directory for GFS transport. (Optional)
* MSTRO_OFI_NUM_RECV
* ``MSTRO_OFI_NUM_RECV``
Number of concurrent receive operations per endpoint. (Optional)
* MSTRO_OFI_CQ_NUM_THREADS
* ``MSTRO_OFI_CQ_NUM_THREADS``
Number of threads servicing OFI completion events. (Optional)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment