Description
Maestro is a data- and memory-aware middleware framework that addresses the ubiquitous problems of data movement in complex memory hierarchies that exist at multiple levels of the HPC software stack.
This repository contains the Maestro Core Library.
It features the Maestro Core API, unit tests that can be used as examples, and some simple applications that can serve as reusable components in user workflows, like a telemetry listener and a pool manager application.
For full documentation check out the documentation at readsthedocs.io.
Installation
Please refer to INSTALL.md
Usage
Maestro can be executed on various sizes and types of machines from a simple laptop to large HPC clusters. On Cray systems, please build all binaries on the service nodes (login nodes) and execute on compute nodes.
Access an installed Maestro version
Please include the main Maestro header
file in your code
#include "maestro.h"
Please add the include path
and library path
of Maestro to the compilation/linking command
-I$(MAESTRO_PATH)/include/maestro -L$(MAESTRO_PATH)/lib -lmaestro
Please export
the path to Maestro library before running
export LD_LIBRARY_PATH=$(MAESTRO_PATH)/lib:$LD_LIBRARY_PATH
where $(MAESTRO_PATH)
is Maestro install path specified during configuration with ./configure --prefix=$(MAESTRO_PATH)
Unit tests
Build the unit tests only
make check TESTS=
Run the unit tests only
make check TESTS
Build and run the unit tests. This may take some time
make check
Limits
Maestro requires quite a few file descriptors and also locks pages into memory for RDMA purposes. We try to give a diagnostic message if errors are triggered that may be due to resource constraints. Still, we recommend
ulimit -n 1024
ulimit -l 256
to set at least 1024 file descriptors and 256k of RDMA space.
When using a system that uses PBSPro
or ALPS
workload manager, please export
export APRUN_XFER_LIMITS=1
before submiting your job to ensure enough limits for Maestro on the compute nodes.
Fabric provider choice/ High-Performance interconnect usage
Maestro isolates the user from the multitude of network provider choices by using
libfabric
, and transparently choosing 'the best' connectivity between components. Unfortunately this
functionality is not fully working, due to issues in the upstream libfabric
code, and in incomplete testing
of our usage of it.
Please use
export FI_PROVIDER=provider
to force a specific fabric provider.
List of supported providers
Provider | Support |
---|---|
sockets | TCP/IP networking |
verbs | Infiniband networks and slingshot |
gni | Cray Areis networks |
psm2 | Intel Omni-Path networks |
You can execute
./Maestro-source-dir/deps/libfabric/util/fi_info
to see all the discovered providers by libfabrics, where Maestro-source-dir
is the directory containing Maestro source.
The safest (and lowest performance) connectivity is provided by the sockets
provider. It should work on almost any network that can support TCP/IP networking, including ethernet, IB, and GNI (Aries).
Usage of the tcp
and tcp;ofi_rxm
provider is currently broken, an upstream
issue is open.
On Cray XC systems the GNI (Aries) provider is supported. If you compile with
the rdma-credentials
and gni-headers
modules loaded, the GNI provider should
be autoselected if a GNI NIC is found at runtime.
NOTE that GNI NICs on login nodes typically do not work, due to a limitation of
the libfabric/gni
driver, so you will have to run your application exclusively
on compute nodes, or manually switch the components running on login nodes to
the sockets provider.
If you are using GNI you will implicitly be using Cray libdrc
, a mechanism to obtain network
authentication tokens. Maestro core is requesting workflow-level tokens that even
support running multiple components of a workflow from different user IDs. In
some cases the system may run out of tokens, and there is no user-level token
inquiry tool available. If you see failure of GNI startup, try running your application with
DRC_DEBUG_LEVEL=DEBUG
and look for an error message like
LIBDRC:CORE:DEBUG rdmacred.c:658 - finished acquire request, rc=-28
If you see this, contact your system admin to clear cached DRC credentials.
Examples
Examples can be found in /examples
directory. It includes currently one simple example, single_node_pool_op.c
. It is a multi-threaded application (pthread) consists of a producer thread, and two consumer threads. single_node_pool_op.c
is based on d3.2 of Maestro project, more reading (d3.2) here.
To build the example, please use
make MAESTRO=$(MAESTRO_PATH)
Example paramters, such as num_producers
, num_consumers
, num_archivers
, cdo_size
, and cdo_count
, can be configured using single_node_pool_op_config.yaml
file.
Before executing the binary, please export LD_LIBRARY_PATH=$(MAESTRO_PATH)/lib:$LD_LIBRARY_PATH
and then
./single_node_pool_op.o
Demos
Local multithreaded demo (MVP1)
MVP1 consists in a local multithreaded demo application. More reading (d3.2) here
Reference version is tagged d3.2-draft
, on master branch.
make check
also builds the demo executable demo_mvp_d3_2
in addition to examples, and runs it.
./run_demo.sh
permits to run the demo alone.
Adaptive Transport demo
Pool manager interlock demo uses a three application setup, comprising one pool
manager process, and showing GFS and MIO transport. More reading (d5.5
to appear on BSCW) and
information on how to setup a VM to run Mero here
Reference version is tagged d5.5-review
, on master branch.
The pool manager interlock demonstration check_pm_interlock.sh is automatically launched with make check.
Documentation
Doxygen as well as Sphinx documentation is available and compiled in docs folder.
Common issues/FAQs
-
If you have many network interfaces/many addresses assigned to an interface (may happen with
IPv6
rather suddenly) thelibfabric
setup of the pool manager may hittoo many open files/errno=-24
issues. Checkulimit -n
, and increase the limit. -
If you see clients stuck at
JOIN
time while everything else looks good, there is a chance that your firewalling intercepts the packages. -
No, the allocation that was captured in the cdo handle must not be touched until after DISPOSE. Of course you can forget the pointer you have, but you must not re-use the allocation or free it.
-
A producer cannot directly figure that out (unless you do complicated event ops). The idea is: The consumer must submit the REQUIRE before the WITHDRAW occurs. This can be accomplished by 1. pre-posting the REQUIRE, or 2. by posting it after observing an OFFER:before or OFFER:after event (for safety with a 'require-ack' flag or any earlier event, like DECLARE or SEAL), 3. by posting it in a WITHDRAW:before with require-ack set. In all these cases Maestro will ensure that the REQUIRE can be satisfied, by taking a copy (more or less eagerly, this is to be tuned], or by blocking WITHDRAW.
-
Every OFFER must be followed by WITHDRAW (and DISPOSE); every REQUIRE must be followed by RETRACT or DEMAND (and DISPOSE). Remember that one OFFER can satisfy many REQUIRES for the same CDO; WITHDRAW indicates that you're no longer ready to do so (and maestro needs to ensure outstanding REQUIRES can still be satisfied if their DEMAND comes in)
-
WITHDRAW may block if maestro decides that it cannot or does not want to take a copy and there is at least one outstanding REQUIRE for the CDO, or a DEMAND is still in progress. DISPOSE should never block (but may take some time -- but not related to the pool protocol).
Authors and acknowledgment
The Maestro -- Data Orchestration in High Performance Computing project has received funding from the European Union’s Horizon 2020 research and innovation program through grant agreement 801101.