Using knockout_prefix in puppet hiera hierarchy

Question

I'd like to use deep_merge option knockout_prefix to remove entries from a hiera array within puppet.

# upper hierarchy
---
  foo:
    - a
    - b
    - c

# lower hierarchy
---
  foo:
    - '--b'
    - y
    - z

# expected result
foo => [a,c,y,z]

I'm using current puppet 4.x PC1 installation. Hiera hierarchy configuration according to official documentation is

[...] my hierarchy stuff omitted
:merge_behavior: deeper
:deep_merge_options:
  :knockout_prefix: '--'

My system should be configured correctly to use this feature:

$ hiera -v
3.0.6
$ /opt/puppetlabs/puppet/bin/gem list --local
*** LOCAL GEMS ***

activemodel (4.2.5)
activesupport (4.2.5)
bigdecimal (1.2.4)
builder (3.2.2)
bundler (1.10.6)
deep_merge (1.0.1)
facter (3.1.4)
[...]

What's my mistake here?

Answer 1

knockout_prefix is used to knockout specific keys and not values in an array, as you have. using your code as an example this would look like

# upper hierarchy
---
lookup_options:
  foo:
    merge:
      strategy: deep
      knockout_prefix: '--'
foo:
  a: a
  b: b
  c: c
# lower hierarchy
---
foo:
  b: --
  y: y
  z: z
# expected result
foo = { 'a' => 'a', 'c' => 'c', 'y' => 'y', 'z' => 'z' }

Answer 2

Hiera 3 merge behavior applies to hash merge lookups. The values associated with key 'foo' in your data are arrays, not hashes, so hash-merging does not apply to them. If you attempt a hash merge lookup on them (i.e. $result = hiera_hash('foo')) then Hiera should throw an error.

If you instead perform an array merge lookup (i.e. $result = hiera_array('foo')) then your knockout prefix is irrelevant. In this case, hiera forms an array of the values of the specified key from every hierarchy level (these are expected to be arrays or strings), flattens it, and removes duplicates. On your data, the result of this should be a six-element array, ['a', 'b', 'c', '--b', 'y', 'z'].

And of course there is the ordinary priority lookup, which you get from automated data binding or from explicitly calling the hiera('foo'). Supposing that "upper hierarchy" means the one with higher priority, the result of that lookup would be ['a', 'b', 'c'].

Answer 3

I know I'm coming in a couple of years and puppet versions late, but I recently got it in my head to try exactly this and couldn't find a solution elsewhere. Note that it is not a direct equivalent of a knockout: it's a puppet runtime manipulation of the list, and it doesn't obey the hierarchical order of the entries. That is, if you "knockout" a value at the lowest-weighted hiera entry, it will still be "knocked out" further up the chain.

Here is the code in my puppet class, assuming the foo hiera from the question:

# Make an array of all of the items that start with `--`
#  (i.e., those that will be knocked out)
$_foo_knockout = $foo.filter |$item| { $item =~ /^--/ }

# Make a new array of items that:
#  * don't directly match an item in the knockout list
#  * don't match an item in the knockout list when prefixed with `--`
$_foo_filtered = $foo.filter |$item| {
  !($item in $_foo_knockout or "--${ex}" in $_foo_knockout)
}

With the example hiera, this would result in

$foo => ['a','b','c','--b','y','z']
$_foo_filtered => ['a','c','y','z']

Notes:

• Tested with hiera = 3.5.0 and puppet = 6.7.2
• Array is unique merged
• Reiterating that an item can be knocked out from any level of the hierarchy