Parse an AWS CloudFormation template with the PyYAML library

Question

I am writing a custom Python application using the PyYAML library that needs to read in AWS CloudFormation YAML templates.

I know the templates are valid CloudFormation templates, because I tested them using validate-template:

▶ aws cloudformation validate-template --template-body file://cloudformation.yml

When I try to read them using the PyYAML library, however, I get errors like:

yaml.scanner.ScannerError: mapping values are not allowed here

and

could not determine a constructor for the tag "!Sub"

and others.

By way of example, I try this AWS example template:

▶ curl -s \
    https://raw.githubusercontent.com/awslabs/aws-cloudformation-templates/master/aws/services/CloudFormation/FindInMap_Inside_Sub.yaml \
    -o FindInMap_Inside_Sub.yaml

And then:

▶ python
Python 2.7.15 (default, Nov 27 2018, 21:40:55) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.load(open('FindInMap_Inside_Sub.yaml'))

Which leads to:

yaml.constructor.ConstructorError: could not determine a constructor for the tag '!FindInMap'
  in "FindInMap_Inside_Sub.yaml", line 89, column 45

How can I parse a CloudFormation YAML file using a library like PyYAML or others?

Answer 1

It is possible to use the cfn_tools library that ships with the aws-cfn-template-flip project.

Install the library:

▶ pip install cfn_flip

Then the simplest Python to read in the template might be:

#!/usr/bin/env python
  
import yaml
from cfn_tools import load_yaml, dump_yaml

text = open('./FindInMap_Inside_Sub.yaml').read()
data = load_yaml(text)

print(dump_yaml(data))

This library is not really documented but there are also various methods in there for customising the formatting of the output worth exploring.

Answer 2

Their aws-cfn-template-flip project that converts cfn templates to/from json and yaml is a good starting point. Example check out the yaml_loader.py script. It shows how it's adding yaml constructors. At the bottom, you'll see:

CfnYamlLoader.add_constructor(TAG_MAP, construct_mapping)
CfnYamlLoader.add_multi_constructor("!", multi_constructor)

You'll probably be interested in the construct_mapping method there. From there, you can look how the code works.

Answer 3

I had some trouble with Alex's answer, because it was automatically converting my CF template into long form. So any !Ref Thing call was being converted into a dictionary mapping.

If you want to match the original input of your input.template file, use this:

from cfn_tools import load_yaml
import cfn_flip.yaml_dumper
import yaml


with open('input.template') as f:
    raw = f.read()
    data_dict = load_yaml(raw)

with open('output.template', 'w') as f:
    dumper = cfn_flip.yaml_dumper.get_dumper(clean_up=False, long_form=False)
    raw = yaml.dump(
        data_dict,
        Dumper=dumper,
        default_flow_style=False,
        allow_unicode=True
    )
    f.write(raw)

You can also change clean_up=False to True to perform some smart formatting, which worked well in my case.

I found this after running the cfn cli tool and seeing the correct short form output on my template. Then I used that main file as a reference and followed the code path.