I have the following YAML:
paths:
patha: /path/to/root/a
pathb: /path/to/root/b
pathc: /path/to/root/c
How can I "normalise" this, by removing /path/to/root/ from the three paths, and have it as its own setting, something like:
paths:
root: /path/to/root/
patha: *root* + a
pathb: *root* + b
pathc: *root* + c
Obviously that's invalid, I just made it up. What's the real syntax? Can it be done?
I don't think it is possible. You can reuse "node" but not part of it.
bill-to: &id001
given : Chris
family : Dumars
ship-to: *id001
This is perfectly valid YAML and fields given and family are reused in ship-to block. You can reuse a scalar node the same way but there's no way you can change what's inside and add that last part of a path to it from inside YAML.
If repetition bother you that much I suggest to make your application aware of root property and add it to every path that looks relative not absolute.
Yes, using custom tags. Example in Python, making the !join tag join strings in an array:
import yaml
## define custom tag handler
def join(loader, node):
seq = loader.construct_sequence(node)
return ''.join([str(i) for i in seq])
## register the tag handler
yaml.add_constructor('!join', join)
## using your sample data
yaml.load("""
paths:
root: &BASE /path/to/root/
patha: !join [*BASE, a]
pathb: !join [*BASE, b]
pathc: !join [*BASE, c]
""")
Which results in:
{
'paths': {
'patha': '/path/to/root/a',
'pathb': '/path/to/root/b',
'pathc': '/path/to/root/c',
'root': '/path/to/root/'
}
}
The array of arguments to !join can have any number of elements of any data type, as long as they can be converted to string, so !join [*a, "/", *b, "/", *c] does what you would expect.
I've create a library, available on Packagist, that performs this function: https://packagist.org/packages/grasmash/yaml-expander
Example YAML file:
type: book
book:
title: Dune
author: Frank Herbert
copyright: ${book.author} 1965
protaganist: ${characters.0.name}
media:
- hardcover
characters:
- name: Paul Atreides
occupation: Kwisatz Haderach
aliases:
- Usul
- Muad'Dib
- The Preacher
- name: Duncan Idaho
occupation: Swordmaster
summary: ${book.title} by ${book.author}
product-name: ${${type}.title}
Example logic:
// Parse a yaml string directly, expanding internal property references.
$yaml_string = file_get_contents("dune.yml");
$expanded = \Grasmash\YamlExpander\Expander::parse($yaml_string);
print_r($expanded);
Resultant array:
array (
'type' => 'book',
'book' =>
array (
'title' => 'Dune',
'author' => 'Frank Herbert',
'copyright' => 'Frank Herbert 1965',
'protaganist' => 'Paul Atreides',
'media' =>
array (
0 => 'hardcover',
),
),
'characters' =>
array (
0 =>
array (
'name' => 'Paul Atreides',
'occupation' => 'Kwisatz Haderach',
'aliases' =>
array (
0 => 'Usul',
1 => 'Muad\'Dib',
2 => 'The Preacher',
),
),
1 =>
array (
'name' => 'Duncan Idaho',
'occupation' => 'Swordmaster',
),
),
'summary' => 'Dune by Frank Herbert',
);
In some languages, you can use an alternative library, For example, tampax is an implementation of YAML handling variables:
const tampax = require('tampax');
const yamlString = `
dude:
name: Arthur
weapon:
favorite: Excalibur
useless: knife
sentence: "{{dude.name}} use {{weapon.favorite}}. The goal is {{goal}}."`;
const r = tampax.yamlParseString(yamlString, { goal: 'to kill Mordred' });
console.log(r.sentence);
// output : "Arthur use Excalibur. The goal is to kill Mordred."
Editor's Note: poster is also the author of this package.
That your example is invalid is only because you chose a reserved character to start your scalars with. If you replace the * with some other non-reserved character (I tend to use non-ASCII characters for that as they are seldom used as part of some specification), you end up with perfectly legal YAML:
paths:
root: /path/to/root/
patha: ♦root♦ + a
pathb: ♦root♦ + b
pathc: ♦root♦ + c
This will load into the standard representation for mappings in the language your parser uses and does not magically expand anything.
To do that use a locally default object type as in the following Python program:
# coding: utf-8
from __future__ import print_function
import ruamel.yaml as yaml
class Paths:
def __init__(self):
self.d = {}
def __repr__(self):
return repr(self.d).replace('ordereddict', 'Paths')
@staticmethod
def __yaml_in__(loader, data):
result = Paths()
loader.construct_mapping(data, result.d)
return result
@staticmethod
def __yaml_out__(dumper, self):
return dumper.represent_mapping('!Paths', self.d)
def __getitem__(self, key):
res = self.d[key]
return self.expand(res)
def expand(self, res):
try:
before, rest = res.split(u'♦', 1)
kw, rest = rest.split(u'♦ +', 1)
rest = rest.lstrip() # strip any spaces after "+"
# the lookup will throw the correct keyerror if kw is not found
# recursive call expand() on the tail if there are multiple
# parts to replace
return before + self.d[kw] + self.expand(rest)
except ValueError:
return res
yaml_str = """\
paths: !Paths
root: /path/to/root/
patha: ♦root♦ + a
pathb: ♦root♦ + b
pathc: ♦root♦ + c
"""
loader = yaml.RoundTripLoader
loader.add_constructor('!Paths', Paths.__yaml_in__)
paths = yaml.load(yaml_str, Loader=yaml.RoundTripLoader)['paths']
for k in ['root', 'pathc']:
print(u'{} -> {}'.format(k, paths[k]))
which will print:
root -> /path/to/root/
pathc -> /path/to/root/c
The expanding is done on the fly and handles nested definitions, but you have to be careful about not invoking infinite recursion.
By specifying the dumper, you can dump the original YAML from the data loaded in, because of the on-the-fly expansion:
dumper = yaml.RoundTripDumper
dumper.add_representer(Paths, Paths.__yaml_out__)
print(yaml.dump(paths, Dumper=dumper, allow_unicode=True))
this will change the mapping key ordering. If that is a problem you have
to make self.d a CommentedMap (imported from ruamel.yaml.comments.py)
With Yglu, you can write your example as:
paths:
root: /path/to/root/
patha: !? .paths.root + a
pathb: !? .paths.root + b
pathc: !? .paths.root + c
Disclaimer: I am the author of Yglu.
I have wrote my own library on Python to expand variables being loaded from directories with a hierarchy like:
/root
|
+- /proj1
|
+- config.yaml
|
+- /proj2
|
+- config.yaml
|
... and so on ...
The key difference here is that the expansion must be applied only after all the config.yaml files is loaded, where the variables from the next file can override the variables from the previous, so the pseudocode should look like this:
env = YamlEnv()
env.load('/root/proj1/config.yaml')
env.load('/root/proj1/proj2/config.yaml')
...
env.expand()
As an additional option the xonsh script can export the resulting variables into environment variables (see the yaml_update_global_vars function).
The scripts:
https://sourceforge.net/p/contools/contools/HEAD/tree/trunk/Scripts/Tools/cmdoplib.yaml.py https://sourceforge.net/p/contools/contools/HEAD/tree/trunk/Scripts/Tools/cmdoplib.yaml.xsh
Pros:
${MYUNDEFINEDVAR} -> *$/{MYUNDEFINEDVAR})${env:MYVAR})\\ to / in a path variable (${env:MYVAR:path})Cons:
${MYSCOPE.MYVAR} is not implemented)