# salt.pillar.makostack

Simple and flexible YAML ext_pillar which can read pillar from within pillar.

Added in version 2016.3.0.

This custom saltstack `ext_pillar` is a direct ripoff of the 'stack' ext_pillar, simply ported to use mako instead of jinja2 for templating.

It supports the following features:

- multiple config files that are mako templates with support for `pillar`, `__grains__`, `__salt__`, `__opts__` objects.

- a config file renders as an ordered list of files. Unless absolute, the paths of these files are relative to the current config file - if absolute, they will be treated literally.

- this list of files are read in order as mako templates with support for `stack`, `pillar`, `__grains__`, `__salt__`, `__opts__` objects.

- all these rendered files are then parsed as `yaml`.

- then all yaml dicts are merged in order, with support for the following. merging strategies: `merge-first`, `merge-last`, `remove`, and `overwrite`.

- stack config files can be matched based on `pillar`, `grains`, or `opts` values, which make it possible to support kind of self-contained environments.

## Configuration in Salt

Like any other external pillar, its configuration takes place through the `ext_pillar` key in the master config file.

However, you can configure MakoStack in 3 different ways:

### Single config file

This is the simplest option, you just need to set the path to your single MakoStack config file like below:

    ext_pillar:
      - makostack: /path/to/stack.cfg

### List of config files

You can also provide a list of config files:

    ext_pillar:
      - makostack:
          - /path/to/stack1.cfg
          - /path/to/stack2.cfg

### Select config files through grains|pillar|opts matching

You can also opt for a much more flexible configuration: MakoStack allows one to select the config files for the current minion based on matching values from either grains, or pillar, or opts objects.

Here is an example of such a configuration, which should speak by itself:

    ext_pillar:
      - makostack:
          pillar:environment:
            dev: /path/to/dev/stack.cfg
            prod: /path/to/prod/stack.cfg
          grains:custom:grain:
            value:
              - /path/to/stack1.cfg
              - /path/to/stack2.cfg
          opts:custom:opt:
            value: /path/to/stack0.cfg

### Grafting data from files to arbitrary namespaces

An extended syntax for config files permits defining "graft points" on a per-config-file basis. As an example, if the file foo.cfg would produce the following:

    foo:
      - bar
      - baz

and you specified the cfg file as /path/to/foo.cfg:yummy:fur, the following would actually end up in pillar after all merging was complete:

    yummy:
      fur:
        foo:
          - bar
          - baz

## MakoStack configuration files

The config files that are referenced in the above `ext_pillar` configuration are mako templates which must render as a simple ordered list of `yaml` files that will then be merged to build pillar data.

Unless an absolute path name is specified, the path of these `yaml` files is assumed to be relative to the directory containing the MakoStack config file. If a path begins with '/', however, it will be treated literally and can be anywhere on the filesystem.

The following variables are available in mako templating of makostack configuration files:

- `pillar`: the pillar data (as passed by Salt to our `ext_pillar` function)

- `minion_id`: the minion id ;-)

- `__opts__`: a dictionary of mostly Salt configuration options

- `__grains__`: a dictionary of the grains of the minion making this pillar call

- `__salt__`: a dictionary of Salt module functions, useful so you don't have to duplicate functions that already exist (note: runs on the master)

So you can use all the power of mako to build your list of `yaml` files that will be merged in pillar data.

For example, you could have a MakoStack config file which looks like:

    $ cat /path/to/stack/config.cfg
    core.yml
    osarchs/%{ __grains__['osarch'] }}.yml
    oscodenames/%{ __grains__['oscodename'] }.yml
    % for role in pillar.get('roles', []):
    roles/%{ role }.yml
    % endfor
    minions/%{ minion_id }.yml

And the whole directory structure could look like:

    $ tree /path/to/stack/
    /path/to/stack/
    ├── config.cfg
    ├── core.yml
    ├── osarchs/
    │   ├── amd64.yml
    │   └── armhf.yml
    ├── oscodenames/
    │   ├── wheezy.yml
    │   └── jessie.yml
    ├── roles/
    │   ├── web.yml
    │   └── db.yml
    └── minions/
        ├── test-1-dev.yml
        └── test-2-dev.yml

## Overall process

In the above MakoStack configuration, given that test-1-dev minion is an amd64 platform running Debian Jessie, and which pillar `roles` is `["db"]`, the following `yaml` files would be merged in order:

- `core.yml`

- `osarchs/amd64.yml`

- `oscodenames/jessie.yml`

- `roles/db.yml`

- `minions/test-1-dev.yml`

Before merging, every files above will be preprocessed as mako templates. The following variables are available in mako templating of `yaml` files:

- `stack`: the MakoStack pillar data object that has currently been merged (data from previous `yaml` files in MakoStack configuration)

- `pillar`: the pillar data (as passed by Salt to our `ext_pillar` function)

- `minion_id`: the minion id ;-)

- `__opts__`: a dictionary of mostly Salt configuration options

- `__grains__`: a dictionary of the grains of the minion making this pillar call

- `__salt__`: a dictionary of Salt module functions, useful so you don't have to duplicate functions that already exist (note: runs on the master)

So you can use all the power of mako to build your pillar data, and even use other pillar values that has already been merged by MakoStack (from previous `yaml` files in MakoStack configuration) through the `stack` variable.

Once a `yaml` file has been preprocessed by mako, we obtain a Python dict - let's call it `yml_data` - then, MakoStack will merge this `yml_data` dict in the main `stack` dict (which contains already merged MakoStack pillar data). By default, MakoStack will deeply merge `yml_data` in `stack` (similarly to the `recurse` salt `pillar_source_merging_strategy`), but 3 merging strategies are currently available for you to choose (see next section).

Once every `yaml` files have been processed, the `stack` dict will contain your whole own pillar data, merged in order by MakoStack. So MakoStack `ext_pillar` returns the `stack` dict, the contents of which Salt takes care to merge in with all of the other pillars and finally return the whole pillar to the minion.

## Merging strategies

The way the data from a new `yaml_data` dict is merged with the existing `stack` data can be controlled by specifying a merging strategy. Right now this strategy can either be `merge-last` (the default), `merge-first`, `remove`, or `overwrite`.

Note that scalar values like strings, integers, booleans, etc. are always evaluated using the `overwrite` strategy (other strategies don't make sense in that case).

The merging strategy can be set by including a dict in the form of:

    __: <merging strategy>

as the first item of the dict or list. This allows fine grained control over the merging process.

### `merge-last` (default) strategy

If the `merge-last` strategy is selected (the default), then content of dict or list variables is merged recursively with previous definitions of this variable (similarly to the `recurse` salt `pillar_source_merging_strategy`). This allows for extending previously defined data.

### `merge-first` strategy

If the `merge-first` strategy is selected, then the content of dict or list variables are swapped between the `yaml_data` and `stack` objects before being merged recursively with the `merge-last` previous strategy.

### `remove` strategy

If the `remove` strategy is selected, then content of dict or list variables in `stack` are removed only if the corresponding item is present in the `yaml_data` dict. This allows for removing items from previously defined data.

### `overwrite` strategy

If the `overwrite` strategy is selected, then the content of dict or list variables in `stack` is overwritten by the content of `yaml_data` dict. So this allows one to overwrite variables from previous definitions.

## Merging examples

Let's go through small examples that should clarify what's going on when a `yaml_data` dict is merged in the `stack` dict.

When you don't specify any strategy, the default `merge-last` strategy is selected:

`stack` | `yaml_data` | `stack` (after merge)
--- | --- | ---
    users:
      tom:
        uid: 500
        roles:
          - sysadmin
      root:
        uid: 0
    users:
      tom:
        uid: 1000
        roles:
          - developer
      mat:
        uid: 1001
    users:
      tom:
        uid: 1000
        roles:
          - sysadmin
          - developer
      mat:
        uid: 1001
      root:
        uid: 0

Then you can select a custom merging strategy using the `__` key in a dict:

`stack` | `yaml_data` | `stack` (after merge)
--- | --- | ---
    users:
      tom:
        uid: 500
        roles:
          - sysadmin
      root:
        uid: 0
    users:
      __: merge-last
      tom:
        uid: 1000
        roles:
          - developer
      mat:
        uid: 1001
    users:
      tom:
        uid: 1000
        roles:
          - sysadmin
          - developer
      mat:
        uid: 1001
      root:
        uid: 0
    users:
      tom:
        uid: 500
        roles:
          - sysadmin
      root:
        uid: 0
    users:
      __: merge-first
      tom:
        uid: 1000
        roles:
          - developer
      mat:
        uid: 1001
    users:
      tom:
        uid: 500
        roles:
          - developer
          - sysadmin
      mat:
        uid: 1001
      root:
        uid: 0
    users:
      tom:
        uid: 500
        roles:
          - sysadmin
      root:
        uid: 0
    users:
      __: remove
      tom:
      mat:
    users:
      root:
        uid: 0
    users:
      tom:
        uid: 500
        roles:
          - sysadmin
      root:
        uid: 0
    users:
      __: overwrite
      tom:
        uid: 1000
        roles:
          - developer
      mat:
        uid: 1001
    users:
      tom:
        uid: 1000
        roles:
          - developer
      mat:
        uid: 1001

You can also select a custom merging strategy using a `__` object in a list:

`stack` | `yaml_data` | `stack` (after merge)
--- | --- | ---
    users:
      - tom
      - root
    users:
      - __: merge-last
      - mat
    users:
      - tom
      - root
      - mat
    users:
      - tom
      - root
    users:
      - __: merge-first
      - mat
    users:
      - mat
      - tom
      - root
    users:
      - tom
      - root
    users:
      - __: remove
      - mat
      - tom
    users:
      - root
    users:
      - tom
      - root
    users:
      - __: overwrite
      - mat
    users:
      - mat

salt.pillar.makostack.ext_pillar(minion_id, pillar, *args, **kwargs)