I would like to merge arrays in YAML, and load them via ruby -
some_stuff: &some_stuff
- a
- b
- c
combined_stuff:
<<: *some_stuff
- d
- e
- f
I'd like to have the combined array as [a,b,c,d,e,f]
I receive the error: did not find expected key while parsing a block mapping
How do I merge arrays in YAML?
If the aim is to run a sequence of shell commands, you may be able to achieve this as follows:
# note: no dash before commands
some_stuff: &some_stuff |-
a
b
c
combined_stuff:
- *some_stuff
- d
- e
- f
This is equivalent to:
some_stuff: "a\nb\nc"
combined_stuff:
- "a\nb\nc"
- d
- e
- f
I have been using this on my gitlab-ci.yml (to answer @rink.attendant.6 comment on the question).
Working example that we use to support requirements.txt having private repos from gitlab:
.pip_git: &pip_git
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf "ssh://[email protected]"
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
test:
image: python:3.7.3
stage: test
script:
- *pip_git
- pip install -q -r requirements_test.txt
- python -m unittest discover tests
use the same `*pip_git` on e.g. build image...
where requirements_test.txt contains e.g.
-e git+ssh://[email protected]/example/[email protected]#egg=example
This post assumes the following context:
lfender6445 wishes to merge two or more lists within a YAML file, and have those merged lists appear as one singular list when parsed.
This may be obtained simply by assigning YAML anchors to mappings, where the desired lists appear as child elements of the mappings. There are caveats to this, however, (see "Pitfalls" infra).
In the example below we have three mappings (list_one, list_two, list_three) and three anchors
and aliases that refer to these mappings where appropriate.
When the YAML file is loaded in the program we get the list we want, but it may require a little modification after load (see pitfalls below).
list_one: &id001
- a
- b
- c
list_two: &id002
- e
- f
- g
list_three: &id003
- h
- i
- j
list_combined:
- *id001
- *id002
- *id003
## list_combined
[
[
"a",
"b",
"c"
],
[
"e",
"f",
"g"
],
[
"h",
"i",
"j"
]
]
This approach allows creation of merged lists by use of the alias and anchor feature of YAML.
Although the output result is a nested list of lists, this can be easily transformed using the flatten method.
flatten methodflatten ;; Merge/flatten an array of arraysflatten ;; http://ruby-doc.org/core-2.2.2/Array.html#method-i-flattenflatten ;; https://softwareengineering.stackexchange.com/a/254676/23884This is not going to work:
merge is only supported by the YAML specifications for mappings and not for sequences
you are completely mixing things by having a merge key <<
followed by the key/value separator : and a value that is a
reference and then continue with a list at the same indentation
level
This is not correct YAML:
combine_stuff:
x: 1
- a
- b
So your example syntax would not even make sense as a YAML extension proposal.
If you want to do something like merging multiple arrays you might want to consider a syntax like:
combined_stuff:
- <<: *s1, *s2
- <<: *s3
- d
- e
- f
where s1, s2, s3 are anchors on sequences (not shown) that you
want to merge into a new sequence and then have the d, e and f
appended to that. But YAML is resolving these kind of structures depth
first, so there is no real context available during the processing
of the merge key. There is no array/list available to you where you
could attach the processed value (the anchored sequence) to.
You can take the approach as proposed by @dreftymac, but this has the huge disadvantage that you somehow need to know which nested sequences to flatten (i.e. by knowing the "path" from the root of the loaded data structure to the parent sequence), or that you recursively walk the loaded data structure searching for nested arrays/lists and indiscriminately flatten all of them.
A better solution IMO would be to use tags to load data structures that do the flattening for you. This allows for clearly denoting what needs to be flattened and what not and gives you full control over whether this flattening is done during loading, or done during access. Which one to choose is a matter of ease of implementation and efficiency in time and storage space. This is the same trade-off that needs to be made for implementing the merge key feature and there is no single solution that is always the best.
E.g. my ruamel.yaml library uses the brute force merge-dicts during
loading when using its safe-loader, which results in merged
dictionaries that are normal Python dicts. This merging has to be done
up-front, and duplicates data (space inefficient) but is fast in value
lookup. When using the round-trip-loader, you want to be able to dump
the merges unmerged, so they need to be kept separate. The dict like
datastructure loaded as a result of round-trip-loading, is space
efficient but slower in access, as it needs to try and lookup a key
not found in the dict itself in the merges (and this is not cached, so
it needs to be done every time). Of course such considerations are
not very important for relatively small configuration files.
The following implements a merge like scheme for lists in python using objects with tag flatten
which on-the-fly recurses into items which are lists and tagged toflatten. Using these two tags
you can have YAML file:
l1: &x1 !toflatten
- 1
- 2
l2: &x2
- 3
- 4
m1: !flatten
- *x1
- *x2
- [5, 6]
- !toflatten [7, 8]
(the use of flow vs block style sequences is completely arbitrary and has no influence on the loaded result).
When iterating over the items that are the value for key m1 this
"recurses" into the sequences tagged with toflatten, but displays
other lists (aliased or not) as a single item.
One possible way with Python code to achieve that is:
import sys
from pathlib import Path
import ruamel.yaml
yaml = ruamel.yaml.YAML()
@yaml.register_class
class Flatten(list):
yaml_tag = u'!flatten'
def __init__(self, *args):
self.items = args
@classmethod
def from_yaml(cls, constructor, node):
x = cls(*constructor.construct_sequence(node, deep=True))
return x
def __iter__(self):
for item in self.items:
if isinstance(item, ToFlatten):
for nested_item in item:
yield nested_item
else:
yield item
@yaml.register_class
class ToFlatten(list):
yaml_tag = u'!toflatten'
@classmethod
def from_yaml(cls, constructor, node):
x = cls(constructor.construct_sequence(node, deep=True))
return x
data = yaml.load(Path('input.yaml'))
for item in data['m1']:
print(item)
which outputs:
1
2
[3, 4]
[5, 6]
7
8
As you can see you can see, in the sequence that needs flattening, you can either use an alias to a tagged sequence or you can use a tagged sequence. YAML doesn't allow you to do:
- !flatten *x2
, i.e. tag an anchored sequence, as this would essentially make it into a different datastructure.
Using explicit tags is IMO better than having some magic going on as
with YAML merge keys <<. If nothing else you now have to go through
hoops if you happen to have a YAML file with a mapping that has a key
<< that you don't want to act like a merge key, e.g. when you make a
mapping of C operators to their descriptions in English (or some other natural language).
Another way to enable merging arrays in python is by defining a !flatten tag.
(This uses PyYAML, unlike Anthon's answer above. This may be necessary in cases when you don't have control over which package is used in the backed, e.g., anyconfig).
import yaml
yaml.add_constructor("!flatten", construct_flat_list)
def flatten_sequence(sequence: yaml.Node) -> Iterator[str]:
"""Flatten a nested sequence to a list of strings
A nested structure is always a SequenceNode
"""
if isinstance(sequence, yaml.ScalarNode):
yield sequence.value
return
if not isinstance(sequence, yaml.SequenceNode):
raise TypeError(f"'!flatten' can only flatten sequence nodes, not {sequence}")
for el in sequence.value:
if isinstance(el, yaml.SequenceNode):
yield from flatten_sequence(el)
elif isinstance(el, yaml.ScalarNode):
yield el.value
else:
raise TypeError(f"'!flatten' can only take scalar nodes, not {el}")
def construct_flat_list(loader: yaml.Loader, node: yaml.Node) -> List[str]:
"""Make a flat list, should be used with '!flatten'
Args:
loader: Unused, but necessary to pass to `yaml.add_constructor`
node: The passed node to flatten
"""
return list(flatten_sequence(node))
This recursive flattening takes advantage of the PyYAML document structure, which parses all arrays as SequenceNodes, and all values as ScalarNodes.
The behavior can be tested (and modified) in the following test function.
import pytest
def test_flatten_yaml():
# single nest
param_string = """
bread: &bread
- toast
- loafs
chicken: &chicken
- *bread
midnight_meal: !flatten
- *chicken
- *bread
"""
params = yaml.load(param_string)
assert sorted(params["midnight_meal"]) == sorted(
["toast", "loafs", "toast", "loafs"]
)
