Maestro Core
------------

This repository contains the Maestro Core Library, as developed for D3.2.
It features the Maestro Core API, used by example code and a MVP demonstrator.

Installation
------------

Please refer to INSTALL.md

Examples
--------

Please use 
```
make check
```
to build and run the test examples.
This may take some time.

Limits
------

Maestro-core needs quite a few file descriptors and also wants to lock pages
into memory for RDMA purposes.  We try to give a diagnostic message if errors
are triggered that may be due to resource constraints. Still, we recommend

```
ulimit -n 1024
ulimit -l 256
```

to set at least 1024 file descriptors and 256k of RDMA space.


Local multithreaded demo (MVP1)
-------------------------------

MVP1 consists in a local multithreaded demo application. More reading (d3.2) here :
`https://bscw.zam.kfa-juelich.de/bscw/bscw.cgi/2995531`

Reference version is tagged d3.2-draft, on master branch.

`make check` also builds the demo executable `demo_mvp_d3_2` in addition to examples, and runs it.
`./run_demo.sh` permits to run the demo alone.


Adaptive Transport demo
-----------------------

Pool manager interlock demo uses a three application setup, comprising one pool
manager process, and showing GFS and MIO transport. More reading (d5.5 to appear on BSCW) and
information on how to setup a VM to run Mero here:
`https://gitlab.version.fz-juelich.de/maestro/maestro-mero-vm`

Reference version is tagged d5.5-review, on master branch.

The pool manager interlock demonstration `./tests/check_pm_interlock.sh` is
automatically launched with make check.


fabric provider choice/ High-Performance Interconnect usage
-----------------------------------------------------------

Maestro-core is trying hard to isolate the user from the multitude of network provider choices by using 
libfabric, and transparently choosing 'the best' connectivity between components. Unfortunately this
functionality is not fully working, due to issues in the upstream libfabric code, and in incomplete testing
of our usage of it.

The safest (and lowest performance) connectivity is provided by the `sockets` provider. You can force usage of 
that by setting

```
FI_PROVIDER=sockets
```

in your environment. It should work on most any network that can support TCP/IP
networking, including ethernet, IB, and GNI (Aries).

Usage of the `tcp` and `tcp;ofi_rxm` provider is currently broken, an upstream
issue is open.

On Cray XC systems the GNI (Aries) provider is supported. If you compile with
the `rdma-credentials` and `gni-headers` modules loaded the GNI provider should
be autoselected if a GNI NIC is found at runtime. 

NOTE that GNI NICs on login nodes typically do not work, due to a limitation of
the libfabric/gni driver, so you will have to run your application exclusively
on compute nodes, or manually switch the components running on login nodes to
the sockets provider.

The GNI driver can be forced by setting

```
FI_PROVIDER=gni
```


If you are using GNI you will implicitly be using Cray libdrc, a mechanism to obtain network
authentication tokens. Maestro-core is requesting workflow-level tokens that even
support running multiple components of a workflow from different user IDs. In
some cases the system may run out of tokens, and there is no user-level token
inquiry tool available. If you see failure of GNI startup, try running your application with

```
DRC_DEBUG_LEVEL=DEBUG
```

and look for an error message like

```
LIBDRC:CORE:DEBUG        rdmacred.c:658 - finished acquire request, rc=-28 
```

If you see this, contact your system admin to clear cached DRC credentials.


Documentation
-------------

Doxygen documentation is available and compiled in `./docs` folder.

Common issues/FAQs
------------------

* If you have many network interfaces/many addresses assigned to an interface
  (may happen with IPv6 rather suddenly) the libfabric setup of the pool
  manager may hit 'too many open files/errno=-24' issues. Check `ulimit -n`,
  and increase the limit.

* If you see clients stuck at JOIN time while everything else looks good, there
  is a chance that your firewalling intercepts the packages.