How do I pick elements from a terraform list

Question

I am creating a series of resources in terraform (in this case, dynamo DB table). I want to apply IAM policies to subgroups of them. E.g.

resource "aws_dynamodb_table" "foo" {
  count = "${length(var.tables)}"
  name           = "foo-${element(var.tables,count.index)}"
  tags {
    Name = "foo-${element(var.tables,count.index)}"
    Environment = "<unsure how to get this>"
    Source = "<unsure how to get this>"
  }
}

All of these share some common element, e.g. var.sources is a list composed of the Cartesian product of var.environments and var.sources:

environments = ["dev","qa","prod"]
sources = ["a","b","c"]

So:

tables = ["a:dev","a:qa","a:prod","b:dev","b:qa","b:prod","c:dev","c:qa","c:prod"]

I want to get the arns of the created dynamo tables that have, e.g. c (i.e. those with the name ["c:dev","c:qa","c:prod"]) or prod(i.e. those with the name ["a:prod","b:prod","c:prod"]).

Is there any sane way to do this with terraform 0.11 (or even 0.12 for that matter)?

I am looking to:

group the dynamo db table resources by some of the inputs (environment or source) so I can apply some policy to each group
Extract the input for each created one so I can apply the correct tags

I was thinking of, potentially, instead of creating the cross-product list, to create maps for each input:

{
  "a": ["dev","qa","prod"],
  "b": ["dev","qa","prod"],
  "c": ["dev","qa","prod"]
}

or

{
  "dev":  ["a","b","c"],
  "qa":   ["a","b","c"],
  "prod": ["a","b","c"]
}

It would make it easy to find the target names for each one, since I can look up by the input, but that only gives me the names, but not make it easy to get the actual resources (and hence the arns).

Thanks!

Martin Atkins Martin Atkins · Accepted Answer · 2019-07-25T18:04:46

A Terraform 0.12 solution would be to derive the cartesian product automatically (using setproduct) and use a for expression to shape it into a form that's convenient for what you need. For example:

locals {
  environments = ["dev", "qa", "prod"]
  sources      = ["a", "b", "c"]

  tables = [for pair in setproduct(local.environments, local.sources) : {
    environment = pair[0]
    source      = pair[1]
    name        = "${pair[1]}:${pair[0]}"
  })
}

resource "aws_dynamodb_table" "foo" {
  count = length(local.tables)
  name  = "foo-${local.tables[count.index].name}"
  tags {
    Name        = "foo-${local.tables[count.index].name}"
    Environment = local.tables[count.index].environment
    Source      = local.tables[count.index].source
  }
}

At the time I write this the resource for_each feature is still in development, but in a near-future Terraform v0.12 minor release it should be possible to improve this further by making these table instances each be identified by their names, rather than by their positions in the local.tables list:

# (with the same "locals" block as in the above example)

resource "aws_dynamodb_table" "foo" {
  for_each = { for t in local.tables : t.name => t }

  name  = "foo-${each.key}"
  tags {
    Name        = "foo-${each.key}"
    Environment = each.value.environment
    Source      = each.value.source
  }
}

As well as cleaning up some redundancy in the syntax, this new for_each form will cause Terraform to identify this instances with addresses like aws_dynamodb_table.foo["a:dev"] instead of aws_dynamodb_table.foo[0], which means that you'll be able to freely add and remove members of the two initial lists without causing churn and replacement of other instances because the list indices changed.

This sort of thing would be much harder to achieve in Terraform 0.11. There are some general patterns that can help translate certain 0.12-only constructs to 0.11-compatible features, which might work here:

A for expression returning a sequence (one with square brackets around it, rather than braces) can be simulated with a data "null_data_source" block with count set, if the result would've been a map of string values only.
A Terraform 0.12 object in a named local value can in principle be replaced with a separate simple map of local value for each object attribute, using a common set of keys in each map.
Terraform 0.11 does not have the setproduct function, but for sequences this small it's not a huge problem to just write out the cartesian product yourself as you did in the question here.

The result will certainly be very inelegant, but I expect it's possible to get something working on Terraform 0.11 if you apply the above ideas and make some compromises.

How do I pick elements from a terraform list

1 Answers