Submodule experimental.config.utils.schema

Since: 3.2.0

The experimental.config.utils.schema module is used to validate and process parts of cluster configurations that have arbitrary user-defined structures:

• app.cfg for applications loaded using the app option
• roles_cfg for custom roles developed as a part of a cluster application

The module provides an API to get and set configuration values, filter and transform configuration data, and so on.

As an example, consider an application role that has a single configuration option – an HTTP endpoint address.

 
roles: [ http_api ]

This is how you can use the experimental.config.utils.schema module to process the role configuration:

1. Load the module:

 
local schema = require('experimental.config.utils.schema')local http_api_schema = schema.new('http_api', schema.scalar({ type = 'string' }))local function validate(cfg)    http_api_schema:validate(cfg)endlocal function apply(cfg)    local http_api_cfg = http_api_schema:get(cfg)    log.info("HTTP API endpoint: %s", http_api_cfg)endlocal function stop()    log.info("The 'http_api' role is stopped")endreturn {    validate = validate,    apply = apply,    stop = stop,}

2. Define a schema – the root object that stores information about the role's configuration – using schema.new(). The example below shows a schema that includes a single string option:

 
local http_api_schema = schema.new('http_api', schema.scalar({ type = 'string' }))

 
Learn more in[config_utils_schema_definition](#config_utils_schema_definition).

3. Use the validate() method of the schema object to validate configuration values against the schema. In case of a role, call this method inside the role's validate() function:

 
local function validate(cfg)    http_api_schema:validate(cfg)end

 
Learn more in[config_utils_schema_validating_configuration](#config_utils_schema_validating_configuration).

4. Refer to values of configuration options using the get() method inside the role's apply() function. Learn more in config_utils_schema_get_configuration.

A configuration schema stores information about a user-defined configuration structure that can be passed inside an app.cfg or a roles_cfg section. It includes option names, types, hierarchy, and other aspects of a configuration.

To create a schema, use the schema.new() function. It has the following arguments:

• Schema name – an arbitrary string to use as an identifier.
• Root schema node – a table describing the hierarchical schema structure starting from the root.
• (Optional) methods – user-defined functions that can be called on this schema object.

Schema nodes describe the hierarchy of options within a schema. There are two types of schema nodes:

• Scalar nodes hold a single value of a supported primitive type. For example, a string configuration option of a role is a scalar node in its schema.
• Composite nodes include multiple values in different forms: records, arrays, or maps.

A node can have annotations – named attributes that enable customization of its behavior, for example, setting a default value.

Scalar nodes hold a single value of a primitive type, for example, a string or a number. For the full list of supported scalar types, see config_utils_schema_data_types.

This configuration has one scalar node of the string type:

 
roles: [ http_api ]

To define a scalar node in a schema, use schema.scalar(). The following schema can be used to process the configuration shown above:

 
local http_api_schema = schema.new('http_api', schema.scalar({ type = 'string' }))

If a scalar node has a limited set of allowed values, you can also define it with the schema.enum(). Pass the list of allowed values as its argument:

 
scheme = schema.enum({ 'http', 'https' }),

Scalar nodes can have the following data types:

Scalar type Lua type Comment

string string
number number
integer number Only integer numbers boolean boolean true or false string, number or number, string string or number
any Arbitrary Lua value May be used to declare an arbitrary value that doesn't need validation.

Record is a composite node that includes a predefined set of other nodes, scalar or composite. In YAML, a record is represented as a node with nested fields. For example, the following configuration has a record node http_api with three scalar fields:

 
roles: [ http_api ]

To define a record node in a schema, use schema.record(). The following schema describes the configuration above:

 
local listen_address_schema = schema.new('listen_address', schema.record({    scheme = schema.enum({ 'http', 'https' }),    host = schema.scalar({ type = 'string' }),    port = schema.scalar({ type = 'integer' })}))

Records are also used to define nested schema nodes of non-primitive types. In the example below, the http_api node includes another record listen_address.

 
roles: [ http_api ]

The following schema describes this configuration:

 
local listen_address_schema = schema.new('listen_address', schema.record({    listen_address = schema.record({        scheme = schema.enum({ 'http', 'https' }),        host = schema.scalar({ type = 'string' }),        port = schema.scalar({ type = 'integer' })    })}))

Array is a composite node type that includes a collection of items of the same type. The items can be either scalar or composite nodes.

In YAML, array items start with hyphens. For example, the following configuration includes an array named http_api. Each its item is a record with three fields: host, port, and scheme:

 
roles: [ http-api ]

To create an array node in a schema, use schema.array(). The following schema describes this configuration:

 
local listen_address_schema = schema.new('listen_address', schema.array({    items = schema.record({        scheme = schema.enum({ 'http', 'https' }),        host = schema.scalar({ type = 'string' }),        port = schema.scalar({ type = 'integer' })    })}))

There is also the schema.set() function that enables creating arrays with a limited set of allowed items.

Map is a composite node type that holds an arbitrary number of key-value pairs of predefined types.

In YAML, a map is represented as a node with nested fields. For example, the following configuration has the endpoints node:

 
roles: [ http_api ]

To create a map node in a schema, use schema.map(). If this node is declared as a map as shown below, the endpoints section can include any number of options with arbitrary names and boolean values.

 
local listen_address_schema = schema.new('listen_address', schema.record({    scheme = schema.enum({ 'http', 'https' }),    host = schema.scalar({ type = 'string' }),    port = schema.scalar({ type = 'integer' }),    endpoints = schema.map({ key = schema.scalar({ type = 'string' }),                             value = schema.scalar({ type = 'boolean' }) })}))

Node annotations are named attributes that define its various aspects. For example, scalar nodes have a required annotation type that defines the node value type. Other annotations can, for example, set a node's default value and a validation function, or store arbitrary user-provided data.

Annotations are passed in a table to the node creation function:

Node annotations fall into three groups:

• Built-in annotations are handled by the module. These are: type, validate, allowed_values, default and apply_default_if. Note that validate and allowed_values are used for validation only. default and apply_default_if can transform the configuration.
• User-defined annotations add named node attributes that can be used in the application or role code.
• Computed annotations allow access to annotations of other nodes throughout the schema.

Built-in annotations are interpreted by the module itself. There are the following built-in annotations:

• type – the node value type. The type must be explicitly specified for scalar nodes, except for those created with schema.enum(). For composite nodes and scalar enums, the corresponding constructors schema.record(), schema.map(), schema.array(), schema.set(), and schema.enum() set the type automatically.
• allowed_values – (optional) a list of possible node values.
• validate – (optional) a validation function for the provided node value.
• default – (optional) a value to use if the option is not specified in the configuration.
• apply_default_if – (optional) a function that defines when to apply the default value.

Consider the following role configuration:

 
roles: [ http_api ]

The following schema uses built-in annotations default, allowed_values, and validate to define default and allowed option values and validation functions:

 
undefined

Validation functions can look as follows:

 
local function validate_host(host, w)    local host_pattern = "^(%d+)%.(%d+)%.(%d+)%.(%d+)$"    if not host:match(host_pattern) then        w.error("'host' should be a string containing a valid IP address, got %q", host)    endendlocal function validate_port(port, w)    if port <= 1 or port >= 65535 then        w.error("'port' should be between 1 and 65535, got %d", port)    endend

A schema node can have user-defined annotations with arbitrary names. Such annotations are used to implement custom behavior. You can get their names and values from the schema and use in the role or application code.

Example: the env user-defined annotation is used to provide names of environment variables from which the configuration values can be taken.

 
local listen_address_schema = schema.new('listen_address', schema.record({    scheme = schema.enum({ 'http', 'https' }, { env = 'HTTP_SCHEME' }),    host = schema.scalar({ type = 'string', env = 'HTTP_HOST' }),    port = schema.scalar({ type = 'integer', env = 'HTTP_PORT' })}))

See the full sample here: config_utils_schema_env-vars.

Computed annotations enable access from a node to annotations of its ancestor nodes.

In the example below, the listen_address record validation function refers to the protocol annotation of its ancestor node:

 
local listen_address = schema.record({    scheme = schema.enum({ 'http', 'https' }),    host = schema.scalar({ type = 'string' }),    port = schema.scalar({ type = 'integer' })}, {    validate = function(data, w)        local protocol = w.schema.computed.annotations.protocol        if protocol == 'iproto' and data.scheme ~= nil then            w.error("iproto doesn't support 'scheme'")        end    end,})

The following schema with listen_address passes the validation:

 
local http_listen_address_schema = schema.new('http_listen_address', schema.record({    name = schema.scalar({ type = 'string' }),    listen_address = listen_address,}, {    protocol = 'http',}))

If this record is added to a schema with protocol = 'iproto', the listen_address validation fails with an error:

 
local iproto_listen_address_schema = schema.new('iproto_listen_address', schema.record({    name = schema.scalar({ type = 'string' }),    listen_address = listen_address,}, {    protocol = 'iproto',}))

A schema can implement custom logic with methods – user-defined functions that can be called on this schema.

For example, this schema has the format method that returns its fields merged in a URI string:

 
local listen_address_schema = schema.new(        "listen_address",        schema.record(                {                    scheme = schema.enum({ "http", "https" }),                    host = schema.scalar({ type = "string" }),                    port = schema.scalar({ type = "integer" })                }        ),        {            methods = {                format = function(_self, url)                    return string.format("%s://%s:%d", url.scheme, url.host, url.port)                end            }        })

The schema object's validate() method performs all the necessary checks on the provided configuration. It validates the configuration structure, node types, allowed values, and other aspects of the schema.

When writing roles, call this function inside the role validation function:

 
local function validate(cfg)    listen_address_schema:validate(cfg)end

To get configuration values, use the schema object's get() method. It takes the configuration and the full path to the node as arguments:

 
local function apply(cfg)    local scheme = listen_address_schema:get(cfg, 'listen_address.scheme')    local host = listen_address_schema:get(cfg, 'listen_address.host')    local port = listen_address_schema:get(cfg, 'listen_address.port')    log.info("HTTP API endpoint: %s://%s:%d", scheme, host, port)end

The schema object has methods that transform configuration data based on the schema, for example, apply_default(), merge(), set().

The following sample shows how to apply default values from the schema to fill missing configuration fields:

 
local function apply(cfg)    local cfg_with_defaults = listen_address_schema:apply_default(cfg)    local scheme = listen_address_schema:get(cfg_with_defaults, 'scheme')    local host = listen_address_schema:get(cfg_with_defaults, 'host')    local port = listen_address_schema:get(cfg_with_defaults, 'port')    log.info("HTTP API endpoint: %s://%s:%d", scheme, host, port)end

The schema.fromenv() function allows getting configuration values from environment variables. The example below shows how to do this by adding a user-defined annotation env:

 
local listen_address_schema = schema.new('listen_address', schema.record({    scheme = schema.enum({ 'http', 'https' }, { env = 'HTTP_SCHEME' }),    host = schema.scalar({ type = 'string', env = 'HTTP_HOST' }),    port = schema.scalar({ type = 'integer', env = 'HTTP_PORT' })}))local function collect_env_cfg()    local res = {}    for _, w in listen_address_schema:pairs() do        local env_var = w.schema.env        if env_var ~= nil then            local value = schema.fromenv(env_var, os.getenv(env_var), w.schema)            listen_address_schema:set(res, w.path, value)        end    end    return resend

The function also uses schema object methods:

• pairs() to iterate over the schema nodes.
• set() to assign configuration values.

schema.array(array_def)

Create an array node of a configuration schema.

Parameters:

• array_def (table) — a table in the following format:

 
{    items = <schema_node_object>,    <..annotations..>}

See also: schema_node_object, schema_node_annotation.

Returns

the created array node as a table with the following fields:

• type: array
• items: a table describing an array item as a schema node
• annotations, if provided in array_def

Return type

table

See also: config_utils_schema_nodes_array

schema.enum(allowed_values, annotations)

A shortcut for creating a string scalar node with a limited set of allowed values.

• allowed_values (table) — a list of enum members – values allowed for the node

• annotations (table) — annotations (see schema_node_annotation)

Returns

the created scalar node as a table with the following fields:

• type: string
• allowed_values: allowed node values
• annotations, if annotations is provided

Return type

table

See also: config_utils_schema_nodes_scalar

Parse an environment variable as a value of the given schema node. The env_var_name parameter is used only for error messages. The value (raw_value) should be received using os.getenv() or os.environ().

How the raw value is parsed depends on the schema_node type:

• Scalar:
  • string: return the value as is
  • number or integer: parse the value as a number or an integer
  • string, number: attempt to parse as a number; in case of a failure return the value as is
  • boolean: accept true and false (case-insensitively), or 1 and 0 for true and false values correspondingly
  • any: parse the value as a JSON
• Map: parse either as JSON (if the raw value starts with {) or as a comma-separated string of key=value pairs: key1=value1,key2=value2
• Array: parse either as JSON (if the raw value starts with [) or as a comma-separated string of items: item1,item2,item3

• env_var_name (string) — environment variable name to use for error messages

• raw_value (string) — environment variable value

• schema_node (schema_node_object) — a schema node (see schema_node_object)

Returns

the parsed value

Return type

table

See also: config_utils_schema_env-vars

schema.map(map_def)

Create a map node of a configuration schema.

• map_def (table) — a table in the following format:

 
{    key = <schema_node_object>,    value = <schema_node_object>,    <..annotations..>}

See also: schema_node_object, schema_node_annotation.

Returns

the created map node as a table with the following fields:

• type: map
• key: map key type
• value: map value type
• annotations, if provided in map_def

Return type

table

See also: config_utils_schema_nodes_map

schema.new(schema_name, schema_node[, { methods = <...> }])

Create a schema object.

• schema_name (string) — a name

• schema_node (table) — a root schema node

• methods (table) — methods

Returns

a new schema object (see schema_object) as a table with the following fields:

 
> - `name`: the schema name> - `schema`: a table with schema nodes> - `methods`: a table with user-provided methods

Return type

table

See also: config_utils_schema_getting_started

schema.record(fields[, annotations])

Create a record node of a configuration schema.

• fields (table) — a table of fields in the following format:

 
{    [<field_name>] = <schema_node_object>,    <...>}

See also: schema_node_object.

• annotations (table) — annotations (see config_utils_schema_annotation)

Returns

the created record node as a table with the following fields:

• type: record
• fields: a table describing the record fields
• annotations, if provided

Return type

table

See also: config_utils_schema_nodes_record

schema.scalar(scalar_def)

Create a scalar node of a configuration schema.

• scalar_def (table) — a table in the following format:

 
{    type = <scalar_type>,    <..annotations..>}

See also: schema_node_object, schema_node_annotation.

• type (string) — data type (see config_utils_schema_data_types)

• annotations (table) — annotations (see config_utils_schema_annotation)

Returns

the created scalar node as a table with the following fields:

• type: the node type (see config_utils_schema_data_types)
• annotations, if provided

Return type

table

See also: config_utils_schema_nodes_scalar

schema.set(allowed_values, annotations)

Shortcut for creating an array node of unique string values from the given list of allowed values.

• allowed_values (table) — allowed values of array items

• annotations (table) — annotations (see config_utils_schema_annotation)

Returns

the created array node as a table with the following fields:

• type: array
• items: a table describing an array item as a schema node
• validate: an auto-generated validation function that checks that the values don't repeat
• annotations, if provided

Return type

table

See also: config_utils_schema_nodes_array

method apply_default(data)

Apply default values to scalar nodes. The functions takes the default built-in annotation values of the scalar nodes and applies them based on the apply_default_if annotation. If there is no apply_default_if annotation on a node, the default value is also applied.

Parameters:

• data (any) — configuration data

Returns

configuration data with applied schema defaults

See also: default, apply_default_if

method filter(data, f)

Filter data based on the schema annotations. The method returns an iterator by configuration nodes for which the given filter function f returns true.

The filter function f receives the following table as the argument:

 
w = {    path = <array-like table>,    schema = <schema node>,    data = <data at the given path>,}

The filter function returns a boolean value that is interpreted as "accepted" or "not accepted".

Example:

Calling a function on all schema nodes that have the my_annotation annotation defined:

 
s:filter(function(w)    return w.schema.my_annotation ~= nilend):each(function(w)    do_something(w.data)end)

• data (any) — configuration data

• f (function) — filter function

Returns

a luafun iterator

method get(data, path)

Get nested configuration values at the given path. The path can be either a dot-separated string (http.scheme) or an array-like table ({ 'http', 'scheme'}).

Example:

 
local scheme = listen_address_schema:get(cfg, 'listen_address.scheme')

• data (any) — configuration data

• path (string/table) — path to the target node as:

• a string in the dot notation

• an array-like table

Returns

data at the given path

See also: see config_utils_schema_get_configuration

method map(data, f, f_ctx)

Transform data by the given function. The data fields are transformed by the function passed in the second argument (f), while its structure remains unchanged.

The transformation function takes three arguments:

• data – the configuration data
• w – walkthrough node with the following fields:
  • w.schema – schema node
  • w.path – the path to the schema node
  • w.error() – a function for printing human-readable error messages
• ctx – additional context for the transformation function. Can be used to provide values for a specific call.

An example of the transformation function:

 
local function f(data, w, ctx)    if w.schema.type == 'string' and data ~= nil then        return data:gsub('{{ *foo *}}', ctx.foo)    end    return dataend

The map() method traverses all fields of the schema records, even if they are nil or box.NULL in the provided configuration. This allows using this method to set computed default values for missing fields. Note that this is not the case for maps and arrays since the schema doesn't define their fields to traverse.

• data (any) — configuration data

• f (function) — transformation function

• f_ctx (any) — user-provided context for the transformation function

Returns

transformed configuration data

method merge(data_a, data_b)

Merge two configurations. The method merges configurations in a single node hierarchy, preferring the latter in case of a collision.

The following merge rules are used:

• any present value is preferred over nil and box.NULL

• box.NULL is preferred over nil

• for scalar and array nodes, the right-hand value is used

• records and maps are deeply merged, that is, the merge is performed recursively for their nested nodes

• data_a (any) — configuration data

• data_b (any) — configuration data

Returns

merged configuration data

method pairs()

Walk over the schema and return scalar, array, and map schema nodes

Returns

a luafun iterator

Example:

 
for _, w in schema:pairs() do    local path = w.path    local schema = w.schema    -- <...>end

method set(data, path, value)

Set a given value at the given path in a configuration. The path can be either a dot-separated string (http.scheme) or an array-like table ({ 'http', 'scheme'}).

• data (any) — configuration data

• path (string/table) — path to the target node as:

• a string in the dot notation

• an array-like table

• value (any) — new value

Returns

updated configuration data

Example: see config_utils_schema_env-vars

Validate data against the schema. If the data doesn't adhere to the schema, an error is raised.

The method performs the following checks:

• field type checks: field values are checked against the schema node types

• allowed values: if a node has the allowed_values annotations of schema nodes, the corresponding data field is checked against the allowed values list

• validation functions: if a validation function is defined for a node (the validate annotation), it is executed to check that the provided value is valid.

• data (any) — data

Example: see config_utils_schema_annotation and config_utils_schema_validating_configuration

See also: allowed_values, validate

User-defined methods in the schema.

See also: config_utils_schema_methods

Schema name.

Schema nodes hierarchy.

See also: config-utils-schema_node_object

The following elements of tables passed as node constructor arguments are parsed by the modules as built-in annotations:

• allowed_values

  A list of allowed values for a node.

  See also: schema_object:validate()

<!– –>

• apply_default_if

  A boolean function that defines whether to apply the default value specified using default. If this function returns true on a provided configuration data, the node receives the default value upon the schema_object.apply_default() method call.

  The function takes two arguments:

  • data – the configuration data
  • w – walkthrough node with the following fields:
    • w.schema – schema node
    • w.path – the path to the schema node
    • w.error() – a function for printing human-readable error messages

  See also: schema_object:apply_default()

<!– –>

• default

  A default value to use for a scalar node if it's not specified explicitly.

  Example: see config_utils_schema_transform_configuration

  See also: schema_object:apply_default()

<!– –>

• type

  A schema node type.

  See also: config_utils_schema_data_types

<!– –>

• validate

  A function used to validate node data. The function must raise an error to fail the check. The function is called upon the schema_object:validate() function calls.

  The function takes two arguments:

  • data – the configuration data
  • w – walkthrough node with the following fields:
    • w.schema – schema node
    • w.path – the path to the schema node
    • w.error() – a function for printing human-readable error messages

  Example:

  A function that checks that a string is a valid IP address:

 
local function validate_host(host, w)    local host_pattern = "^(%d+)%.(%d+)%.(%d+)%.(%d+)$"    if not host:match(host_pattern) then        w.error("'host' should be a string containing a valid IP address, got %q", host)    endend

See also: schema_object:validate()

: schema_node_object

A list of values allowed for the node. The values are taken from the allowed_values node annotation.

A function to define when to apply the default node value. The value is taken from the apply_default_if annotation.

computed.annotations stores the node's computed annotations.

Node's default value. The value is taken from the default annotation.

Child nodes for record nodes. See also config_utils_schema_nodes_record.

Node items for array nodes. See also config_utils_schema_nodes_array

Node type for scalar nodes. See config_utils_schema_data_types

Node value validation function. The value is taken from the validate annotation.