Lustenberg.org :: Ansible Variable Gathering

Ansible is well known to have a less than simple precedence rule set, which is born of necessity.

For the majority of tasks, this complexity can be handled by having separate variable names when two ansible groups need to both set a variable, and them combined via a template or set_facts.

However, there are two cases routinely encountered: ‘Site’ / ‘Common’ Ansible groups defining packages and user accounts. The former being a simple list, and the latter typically being a complex nested dictionary. While they can be mostly worked around via clever templating, it is clunky at best.

Cue a lookup plugin!

$ cat lookup_plugins/gather_dict.py
from ansible.plugins.lookup import LookupBase

# gathers matching variables, returns array of values to be fed into `combine()`
class LookupModule(LookupBase):
    def run(self, terms, variables, **kwargs):

        found_vars = sorted(variables.keys())

        try:
            suffix = kwargs['suffix']
            found_vars = [ x for x in found_vars if x.endswith(suffix) ]
        except KeyError:
            pass

        try:
            prefix = kwargs['prefix']
            found_vars = [ x for x in found_vars if x.startsswith(prefix) ]
        except KeyError:
            pass

        try:
            exclude = kwargs['exclude']
            exc = { x: 1 for x in variables[exclude] }
            found_vars = [ x for x in found_vars if x not in exc ]
        except KeyError:
            pass

        return [ variables[x] for x in found_vars ]

$ cat lookup_plugins/gather_list.py
from ansible.plugins.lookup import LookupBase

# gather lists matching prefix/suffix, return combined sorted list
class LookupModule(LookupBase):
    def run(self, terms, variables, **kwargs):
[... same as gather_dict up to return]
        acc = {}
        for v in found_vars:
            for i in variables[v]:
                acc[i] = 1

        return [sorted(acc.keys())]

The specific case addressed here is having machines grouped into various high level roles, without rising to the level of needing a specific top level runbook:

all: bare minimum. Admin users are added to wheel, define common system packages such as openssh and shells.
desktop: machines using a GUI. Add packages for X, browsers, etc.
Other roles as needed; I have a radio group that added packages such as wstjx, and adds my user to the dialout group to access serial ports.

So how is this actually used? Starting with the all group, the basic user structure is defined:

all_users:
  alexl:
    password: '$6$rounds=10000$v...'
    uid: 1000
    groups:
      - sudo
      - wheel
    authorized_keys:
      - ssh-rsa 'AAAA...Personal 2024'

Groups then define additional structure to combine on top of the all include:

$ cat group_vars/desktop.yml
hardware_users:
  alexl:
    groups:
      - audio
      - bluetooth
      - video
...

$ cat group_vars/radio.yml
laptop_users:
  alexl:
    groups:
      - dialout

We then invoke the lookup, gathering all variables that end in _users. As to be flexible on how variables are combined, and to keep the implementation simple, gather_dict returns an array of hashes that can simply be fed into combine(), and it’s arguments are used to set behaviour. In this case, we want a deep merge of the hashes, and any lists encountered should be appended.

$ cat roles/common/tasks/main.yml
- name: set facts
  set_fact:
    users: "{{ lookup('gather_dict', suffix='_users') | ansible.builtin.combine( recursive=true, list_merge='append' ) }}"
    packages_remove: "{{ lookup('gather_list',suffix='_packages_remove') }}"
    packages_install: "{{ lookup('gather_list',suffix='_packages_install', exclude='packages_remove') }}"
  tags:
    - always

- debug:
    var: users
  tags:
    - test

Details omitted for brevity:

# run against local laptop and droplet
$ ansible-playbook -i inventory.yml site.yml -t test -l neptune,styx
debug...
  styx ok: {
    "changed": false,
    "users": {
        "alexl": {
            "groups": [
                "sudo"
            ],
        },
    }
}
  neptune ok: {
    "changed": false,
    "users": {
        "alexl": {
            "groups": [
                "sudo",
                "audio",
                "bluetooth",
                "dialout",
                "video"
            ],
        },
    }
}

Now that we have a nicely defined structure, it can be consumed in a single pass to create user accounts, group membership, and ssh keys.

No playbook changes needed if another group is added, as long as the variable naming scheme is followed.