How to parse JSON from Terraform null_resource into map using data external block

0
votes

I am trying to parse JSON key/value pairs into a map I can use in Terraform during a lookup.

I've created a null_resource with a local-exec provisioner to run my aws cli command and then parsed with jq to clean it up. The JSON looks good, the correct key/value pairs are displayed when run from the CLI. I created an external data block to convert the JSON into an TF map, but I'm getting an Inccorect attribute error from TF.

resource "null_resource" "windows_vars" {
  provisioner "local-exec" {
    command = "aws ssm --region ${var.region} --profile ${var.profile}  get-parameters-by-path --recursive --path ${var.path} --with-decryption | jq '.Parameters | map({'key': .Name, 'value': .Value}) | from_entries'"
  }
}

data "external" "json" {
  depends_on  = [null_resource.windows_vars]
  program     = ["echo", "${null_resource.windows_vars}"]
}

output "map" {
  value = ["${values(data.external.json.result)}"]
}

I expected the key/value pairs to be added to a TF map I could use elsewhere. I got the following error:

Error: Incorrect attribute value type

  on instances/variables.tf line 33, in data "external" "json":
  33:   program     = ["echo", "${null_resource.windows_vars}"]

Inappropriate value for attribute "program": element 1: string required.

JSON output looks like this:

{
  "/vars/windows/KEY_1": "VALUE_1",
  "/vars/windows/KEY_2": "VALUE_2",
  "/vars/windows/KEY_3": "VALUE_3",
  "/vars/windows/KEY_4": "VALUE_4"
}
2
jsonterraform
If you are on 0.12.x, have you looked into this: terraform.io/docs/configuration/functions/jsondecode.html – Matt Schuchard
and there is exist data source for ssm parameter store you can used directly, why create the null_resource – BMW
@BMW you should write that up as an answer as it's a much better approach than what the OP is trying to do. – ydaetskcoR
@BMW I didn't use the aws_ssm_parameter data block because I am accessing these parameters in a shared services account from every account in my organization. – The Bloody Nine

2 Answers

1
votes

I actually answered my own question. I am using a data external block to run my aws cli command and referencing the block in my module.

data "external" "json" {
  program = ["sh", "-c", "aws ssm --region ${var.region} --profile ${var.profile} get-parameters-by-path --recursive --path ${var.path} --with-decryption | jq '.Parameters | map({'key': .Name, 'value': .Value}) | from_entries'"]
}

The ${var.amis["win2k19_base"]} will do a lookup on a map of ami ids I use and I am using that as the key in the parameter store for the value I am looking for.

Inside my module I am using this:

instance_var = data.external.json.result["${var.path}${var.amis["win2k19_base"]}"]

Thank you for the great suggestions.

1
votes

An alternative way to address this would be to write a data-only module which encapsulates the data fetching and has its own configured aws provider to fetch from the right account.

Although it's usually not recommended for a child module to have its own provider blocks, that is allowed and can be okay if the provider in question is only being used to fetch data sources, because Terraform will never need to "destroy" those. The recommendation against nested module provider blocks is that it will cause trouble if you remove a module while the resource objects declared inside it still exist, and then there's no provider configuration left to use to destroy them.

With that said, here's an example of the above idea, intended to be used as a child module which can be imported by any configuration that needs access to this data:

variable "region" {}

variable "profile" {}

variable "path" {}

provider "aws" {
  region  = var.region
  profile = var.profile
}

data "aws_ssm_parameter" "foo" {
  name = var.path
}

output "result" {
  # For example we'll just return the entire thing, but in
  # practice it might be better to pre-process the output
  # into a well-defined shape for consumption by the calling
  # modules, so that they can rely on a particular structure.
  value = jsondecode(data.aws_ssm_parameter.foo)
}

I don't think the above is exactly equivalent to the original question since AFAIK aws_ssm_parameter does not do a recursive fetch at the time of writing, but I'm not totally sure. My main purpose here was to show the idea of using a nested module with its own provider configuration as an alternative way to fetch data from a specific account/region.

A more direct response to the original question is that provisioners are designed as one-shot actions and so it's not possible to access any data they might return. The external data source is one way to run an external program to gather some data, if a suitable data source is not already available.