diff --git a/docs/sphinx/_templates/layout.html b/docs/sphinx/_templates/layout.html index f4a9393eacdb099cf616aa90e8e71440f42656d0..6de7344f30657d10d9846ee0dc377eecbab65fef 100644 --- a/docs/sphinx/_templates/layout.html +++ b/docs/sphinx/_templates/layout.html @@ -7,3 +7,8 @@ {% endblock %} +{%- block footer %} + <br> +{% endblock %} + + diff --git a/docs/sphinx/_templates/searchbox.html b/docs/sphinx/_templates/searchbox.html index cdb37031e27c09cbcbe8233493cac59657113a1d..2204daf20b0ac4b97516f81a574a03aa2033eae0 100644 --- a/docs/sphinx/_templates/searchbox.html +++ b/docs/sphinx/_templates/searchbox.html @@ -13,7 +13,7 @@ <div id="searchbox" style="display: none" role="search"> <div class="searchformwrapper"> <form class="search" action="{{ pathto('search') }}" method="get"> - <input type="text" placeholder="Search the docs" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/> + <input type="text" placeholder="Search docs" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/> </form> </div> </div> diff --git a/docs/sphinx/essentials.rst b/docs/sphinx/essentials.rst index 16232247a982882f570621f16acdb88c23a811d9..66519808d065cc620e45f837b3b9d3ade31f00ed 100644 --- a/docs/sphinx/essentials.rst +++ b/docs/sphinx/essentials.rst @@ -43,7 +43,8 @@ actual data to be transferred, we are going to need a pointer and a size: &my_data_size, ...); At this point of the CDO definition the upcoming transfer will indeed consist -of data and metadata. +of data and metadata. :ref:`User-defined<user-metadata>` and +:ref:`layout<layout>` metadata in particular may also be added. .. _mamba: diff --git a/docs/sphinx/showroom.rst b/docs/sphinx/showroom.rst index f2a091d9a8c5b679554fc5da378a31268c1852d8..9efc957896641882b58b86133a20159bc2b9396a 100644 --- a/docs/sphinx/showroom.rst +++ b/docs/sphinx/showroom.rst @@ -11,19 +11,14 @@ Workflow applications using Maestro Core offer and demand CDOs to the :ref:`Maestro Pool<cdo_management>`, a conceptual entity that represents the set of resources contributed to Maestro, in particular the set of offered CDOs. One typical use of Maestro Core is bypassing persistent storage for application -coupling, by way of the Maestro Pool. +coupling, by way of the Maestro Pool. The :ref:`Pool Manager<pm>` is a Maestro +Core-produced application that handles networking, transport scheduling and +propagates pool events for inspection. -:ref:`CDOs<cdo>` are basically Maestro Core currency, they contain data and metadata, -including user-defined metadata plus data semantics such as data layout -information and other information on the data usage the transfer scheduler may -take advantage of. - -A minimal Maestro-enabled multi-app setup would consist in one producer -application, one consumer application and one Pool Manager application, the -latter being shipped with Maestro Core and alternatively invokable via the -Maestro Core API. The :ref:`Pool Manager<pm>` is a Maestro Core-produced -application that handles networking, transport scheduling and propagates -pool events for inspection. +:ref:`CDOs<cdo>` are basically Maestro Core currency, they contain data and +metadata, including user-defined metadata plus data semantics such as data +layout information and other information on data usage the transfer scheduler +may take advantage of. .. image:: img/maestro_core_components.png :alt: Maestro core components overview @@ -32,35 +27,29 @@ pool events for inspection. workflow, and interfacing with many useful components. In particular, they allow the implementation of data-driven Workflow Managers or more generally Execution Frameworks, meaning they can schedule jobs based on CDO availability -and location which is indeed notified by events. :ref:`Data -management<cdo_management>` (and :ref:`memory management<mamba>` as well) is -then effectively delegated by Execution Frameworks and applications to Maestro -Core. - -As an example of useful workflow components, staging and archiving CDOs of -choice may be implemented by users via dedicated components, archiving relying -especially on events to be notified of the availability of relevant CDOs for -archiving purposes. -A keep-alive proxy may allow to retain CDOs from being withdrawn from the pool, -which would be before they have been properly handled by all the components -that might have needed them. -Maestro-enabled components natively produce logs as well as a telemetry report -at finalize time, nonetheless Events allow to implement more specific -telemetry/logger components, with :ref:`selectors<events>` allowing -:ref:`cherry-picking<story_cherry>` of events and CDOs. - -CDOs may convey a considerable range of data and memory semantics. -In particular, data layout information lets Maestro make the transformations +and location which is indeed notified by events. +As an example of another useful workflow components, staging and archiving CDOs +of choice may be implemented by users via dedicated components, archiving +relying especially on events to be notified of the availability of +(possibly :ref:`cherry-picked<story_cherry>`) CDOs for archiving purposes. + +In addition to :ref:`data management<cdo_management>`, Maestro Core facilitates +:ref:`memory management<mamba>`, allowing to abstract away memory layer +consideration as well as providing features such as tiling. +CDO location is expressed as a so-called Memory Object, +which can mean a certain number of layers (DRAM, GDRAM, NVRAM, PFS, etc.) and +associated access methods (pointer, path, object ID, etc.), which is then used +by Maestro Core to handle the transport between layers and nodes. + +CDOs may convey a wide range of data and memory semantics. In particular, +:ref:`data layout<layout>` information lets Maestro make the transformations between a given offered CDO in a given data layout L1 (say row-major) to the demanded CDO on the consumer side that may have a layout L2 (say column-major). This allows user to abstract away data layout, especially if tools or source-to-source substitution fills the data layout attribute (semi) automatically. Data layouts may be distributed and Maestro Core handle the redistribution complying with producer and consumer layout and distribution -scheme requirements. CDO location is expressed as a so-called Memory Object, -which can mean a certain number of layers (DRAM, GDRAM, NVRAM, PFS, etc.) and -associated access methods (pointer, path, object ID, etc.), which is then used -by Maestro Core to handle the transport between layers and nodes. +scheme requirements. Demos diff --git a/docs/sphinx/tuning.rst b/docs/sphinx/tuning.rst index 48247c449402bbc8bbb6aa509227ca0797b6e3f1..86d0a965c9189dbf49d67acef5d01167b78f9a6f 100644 --- a/docs/sphinx/tuning.rst +++ b/docs/sphinx/tuning.rst @@ -1,7 +1,7 @@ Tuning ====== -Make Maestro Core perform. +Have Maestro Core perform. Architecture ------------ @@ -20,11 +20,11 @@ Knobs Maestro features a range of environment variables that can be played with. -* MSTRO_LOG_LEVEL +* ``MSTRO_LOG_LEVEL`` Log level (Error|Warning|Info|Debug). (Optional) -* MSTRO_LOG_MODULES +* ``MSTRO_LOG_MODULES`` Selection of modules that should log (Optional) Typically @@ -38,27 +38,27 @@ Maestro features a range of environment variables that can be played with. to record logs that would slow us down, but still want the stats report at ``mstro_finalize()`` time. -* MSTRO_LOG_COLOR +* ``MSTRO_LOG_COLOR`` Log color. (green|blue|brightblue|...) (Optional). Typically helps visually parse small logs between a couple apps. -* MSTRO_LOG_DST +* ``MSTRO_LOG_DST`` Select logging output channel. (stdout|stderr|syslog) (Optional) -* MSTRO_TRANSPORT_DEFAULT +* ``MSTRO_TRANSPORT_DEFAULT`` Which transport method to choose by default. (RDMA|GFS|MIO) (Optional) -* MSTRO_TRANSPORT_GFS_DIR +* ``MSTRO_TRANSPORT_GFS_DIR`` Directory for GFS transport. (Optional) -* MSTRO_OFI_NUM_RECV +* ``MSTRO_OFI_NUM_RECV`` Number of concurrent receive operations per endpoint. (Optional) -* MSTRO_OFI_CQ_NUM_THREADS +* ``MSTRO_OFI_CQ_NUM_THREADS`` Number of threads servicing OFI completion events. (Optional)