attributes.md

Maestro Attribute Specification

Introduction

Each CDO in Maestro has an attribute table (a key-value store). Abstractly, attributes are associated to the CDO, but practially they belong to the CDO instance accessible to a Maestro application through the CDO handle it hold. In many cases the application will not notice this difference. However, the pool manager keeps a table of all CDO instances (and their physical locations) along with the instance specific attributes, but serves queries by taking the union over some core attributes (in particular, for layouts), and checks consistency for others.

Attribute keys are strings, interpreteded in tree structure, with `.' (period) as separators. The root of the attribute key tree is the empty string, making '.' the starting character of any full-qualified key; that permits operations to use relative keys (which never start with a dot), that are interpreted relative to a (application-, not thread-local) default namespace.

Attribute values are either elementary types (that directly map to programming-language elementary types, including integer, double, and blob), or simple aggregate types (arrays, lists, dictionaries) over these. The permissible values are defined in the schema definition below.

We are using YAML for writing and printing attribute data. There is a built-in schema defining the core attributes, and users can add their own user-defined schemata. The union of all schemata defined at Maestro startup forms the basis of attribute handling (active attribute schema).¹ All components of a Maestro workflow need to share the same active attribute schema.

The legal syntax of Maestro schemata is a subset of the full YAML specification. Schemata are parsed at startup, but not validated. This is partially due to the lack of embeddable and performant YAML schema validators.

Similarly, Maestro core will only use as much of the schema information as is necessary to correctly handle attribute values: it will not perform validation of user-specified attributes, but will rely on correctness of their type when handling them.

YAML-Schema-Schema

Maestro permits the following elementary data types. Our choice is guided by the following principles:

• simplicity,

• direct mapping to a protobuf datatype for serialization,

• direct mapping to C and FORTRAN data types.

---

Schema type	protobuf type	C type	FORTRAN type

The current schema definition is given in maestro-schema-schema.yaml in the maestro source tree. Any modification to the core schema or a user-defined schema should be validated against this definition. We are currently using yamale, a python-based schema validator. It can be installed simply by

 $ pip3 install yamale

and used by

 $ cd attributes 
 
 $ yamale -s maestro-core-core.yaml --strict <your-schema>

Built-in Schema

The built-in schema is defined in maestro-core.yaml in the source tree. Developers should be aware that some of this schema may be built-in at compile time for efficiency, and specifying a modified schema at Maestro startup time may raise an error. If the core schema is changed, the Maestro core library should be rebuilt.

User-defined Schemata

Users can define their own schemata. A superficial example is given in user.yaml.

1. There are currently no provisions for dynamically modifying the active schemata at runtime. ↩