Parsing YAML

pdo debian sw eng

This is part of a series of posts on ideas for an Ansible-like provisioning system, implemented in Transilience.

The time has come for me to try and prototype if it's possible to load some Transilience roles from Ansible's YAML instead of Python.

The data models of Transilience and Ansible are not exactly the same. Some of the differences that come to mind:

Ansible has a big pot of global variables; Transilience has a well defined set of role-specific variables.
Roles in Ansible are little more than a chunk of playbook that one includes; Roles in Transilience are self-contained and isolated, support pipelined batches of tasks, and can use full Python logic.
Transilience does not have a template action: the equivalent is a copy action that uses the Role's rendering engine to render the template.
Handlers in Ansible are tasks identified by a name in a global namespace; handlers in Transilience are Roles, identified by their Python classes.

To simplify the work, I'll start from loading a single role out of Ansible, not an entire playbook.

TL;DR: scroll to the bottom of the post for the conclusion!

Loading tasks

The first problem of loading an Ansible task is to figure out which of the keys is the module name. I have so far failed to find precise reference documentation about what keyboards are used to define a task, so I'm going by guesswork, and if needed a look at Ansible's sources.

My first attempt goes by excluding all known non-module keywords:

        candidates = []
        for key in task_info.keys():
            if key in ("name", "args", "notify"):
                continue
            candidates.append(key)

        if len(candidates) != 1:
            raise RoleNotLoadedError(f"could not find a known module in task {task_info!r}")

        modname = candidates[0]
        if modname.startswith("ansible.builtin."):
            name = modname[16:]
        else:
            name = modname

This means that Ansible keywords like when or with will break the parsing, and it's fine since they are not supported yet.

args seems to carry arguments to the module, when the module main argument is not a dict, as may happen at least with the command module.

Task parameters

One can do all sorts of chaotic things to pass parameters to Ansible tasks: for example string lists can be lists of strings or strings with comma-separated lists, and they can be preprocesed via Jinja2 templating, and they can be complex data structures that might contain strings that need Jinja2 preprocessing.

I ended up mapping the behaviours I encountered in an AST-like class hierarchy which includes recursive complex structures.

Variables

Variables look hard: Ansible has a big free messy cauldron of global variables, and Transilience needs a predefined list of per-role variables.

However, variables are mainly used inside Jinja2 templates, and Jinja2 can parse to an Abstract Syntax Tree and has useful methods to examine its AST.

Using that, I managed with resonable effort to scan an Ansible role and generate a list of all the variables it uses! I can then use that list, filter out facts-specific names like ansible_domain, and use them to add variable definition to the Transilience roles. That is exciting!

Handlers

Before loading tasks, I load handlers as one-action roles, and index them by name. When an Ansible task notifies a handler, I can then loop up by name the roles I generated in the earlier pass, and I have all that I need.

Parsed Abstract Syntax Tree

Most of the results of all this parsing started looking like an AST, so I changed the rest of the prototype to generate an AST.

This means that, for a well defined subset of Ånsible's YAML, there exists now a tool that is able to parse it into an AST and raeson with it.

Transilience's playbooks gained a --ansible-to-ast option to parse an Ansible role and dump the resulting AST as JSON:

$ ./provision --help
usage: provision [-h] [-v] [--debug] [-C] [--ansible-to-python role]
                 [--ansible-to-ast role]

Provision my VPS

optional arguments:
[...]
  -C, --check           do not perform changes, but check if changes would be
                        needed
  --ansible-to-ast role
                        print the AST of the given Ansible role as understood
                        by Transilience

The result is extremely verbose, since every parameter is itself a node in the tree, but I find it interesting.

Here is, for example, a node for an Ansible task which has a templated parameter:

    {
      "node": "task",
      "action": "builtin.blockinfile",
      "parameters": {
        "path": {
          "node": "parameter",
          "type": "scalar",
          "value": "/etc/aliases"
        },
        "block": {
          "node": "parameter",
          "type": "template_string",
          "value": "root: {{postmaster}}\n{% for name, dest in aliases.items() %}\n{{name}}: {{dest}}\n{% endfor %}\n"
        }
      },
      "ansible_yaml": {
        "name": "configure /etc/aliases",
        "blockinfile": {},
        "notify": "reread /etc/aliases"
      },
      "notify": [
        "RereadEtcAliases"
      ]
    },

Here's a node for an Ansible template task converted to Transilience's model:

    {
      "node": "task",
      "action": "builtin.copy",
      "parameters": {
        "dest": {
          "node": "parameter",
          "type": "scalar",
          "value": "/etc/dovecot/local.conf"
        },
        "src": {
          "node": "parameter",
          "type": "template_path",
          "value": "dovecot.conf"
        }
      },
      "ansible_yaml": {
        "name": "configure dovecot",
        "template": {},
        "notify": "restart dovecot"
      },
      "notify": [
        "RestartDovecot"
      ]
    },

Executing

The first iteration of prototype code for executing parsed Ansible roles is a little execise in closures and dynamically generated types:

    def get_role_class(self) -> Type[Role]:
        # If we have handlers, instantiate role classes for them
        handler_classes = {}
        for name, ansible_role in self.handlers.items():
            handler_classes[name] = ansible_role.get_role_class()

        # Create all the functions to start actions in the role
        start_funcs = []
        for task in self.tasks:
            start_funcs.append(task.get_start_func(handlers=handler_classes))

        # Function that calls all the 'Action start' functions
        def role_main(self):
            for func in start_funcs:
                func(self)

        if self.uses_facts:
            role_cls = type(self.name, (Role,), {
                "start": lambda host: None,
                "all_facts_available": role_main
            })
            role_cls = dataclass(role_cls)
            role_cls = with_facts(facts.Platform)(role_cls)
        else:
            role_cls = type(self.name, (Role,), {
                "start": role_main
            })
            role_cls = dataclass(role_cls)

        return role_cls

Now that the parsed Ansible role is a proper AST, I'm considering redesigning that using a generic Role class that works as an AST interpreter.

Generating Python

I maintain a library that can turn an invoice into Python code, and I have a convenient AST. I can't not generate Python code out of an Ansible role!

$ ./provision --help
usage: provision [-h] [-v] [--debug] [-C] [--ansible-to-python role]
                 [--ansible-to-ast role]

Provision my VPS

optional arguments:
[...]
  --ansible-to-python role
                        print the given Ansible role as Transilience Python
                        code
  --ansible-to-ast role
                        print the AST of the given Ansible role as understood
                        by Transilience

And will you look at this annotated extract:

$ ./provision --ansible-to-python mailserver
from __future__ import annotations
from typing import Any
from transilience import role
from transilience.actions import builtin, facts

# Role classes generated from Ansible handlers!
class ReloadPostfix(role.Role):
    def start(self):
        self.add(
            builtin.systemd(unit='postfix', state='reloaded'),
            name='reload postfix')


class RestartDovecot(role.Role):
    def start(self):
        self.add(
            builtin.systemd(unit='dovecot', state='restarted'),
            name='restart dovecot')


# The role, including a standard set of facts
@role.with_facts([facts.Platform])
class Role(role.Role):
    # These are the variables used by Jinja2 template files and strings. I need
    # to use Any, since Ansible variables are not typed
    aliases: Any = None
    myhostname: Any = None
    postmaster: Any = None
    virtual_domains: Any = None

    def all_facts_available(self):
        ...
        # A Jinja2 string inside a string list!
        self.add(
            builtin.command(
                argv=[
                    'certbot', 'certonly', '-d',
                    self.render_string('mail.{{ansible_domain}}'), '-n',
                    '--apache'
                ],
                creates=self.render_string(
                    '/etc/letsencrypt/live/mail.{{ansible_domain}}/fullchain.pem'
                )),
            name='obtain mail.* letsencrypt certificate')

        # A converted template task!
        self.add(
            builtin.copy(
                dest='/etc/dovecot/local.conf',
                src=self.render_file('templates/dovecot.conf')),
            name='configure dovecot',
            # Notify referring to the corresponding Role class!
            notify=RestartDovecot)

        # Referencing a variable collected from a fact!
        self.add(
            builtin.copy(dest='/etc/mailname', content=self.ansible_domain),
            name='configure /etc/mailname',
            notify=ReloadPostfix)
        ...

Conclusion

Transilience can load a (growing) subset of Ansible syntax, one role at a time, which contains:

All actions defined in Transilience's builtin.* namespace
Ansible's template module (without block_start_string, block_end_string, lstrip_blocks, newline_sequence, output_encoding, trim_blocks, validate, variable_end_string, variable_start_string)
Jinja2 templates in string parameters, even when present inside lists and dicts and nested lists and dicts
Variables from facts provided by transilience.actions.facts.Platform
Variables used in jitsi templates, both in strings and in files, provided by host vars, group vars, role parameters, and facts
Notify using handlers defined within the role. Notifying handlers from other roles is not supported, since roles in Transilience are self-contained

The role loader in Transilience now looks for YAML when it does not find a Python module, and runs it pipelined and fast!

There is code to generate Python code from an Ansible module: you can take an Ansible role, convert it to Python, and then work on it to add more complex logic, or clean it up for adding it to a library of reusable roles!

Next: Ansible conditionals