Manage one-time infrastructure jobs in AWS

Question

Terraform is declarative IaC, but sometimes you need to write an imperative fix using, e.g, aws cli, I have terraform project which is maintained by CI/CD pipelines and works fine by maintaining the infrastructure in the main Region we work on.

But the time comes to do something that the Terraform is not yet ready to handle, like enable Aws Config in all AWS regions across tons of accounts.

Since Terraform is declarative and deployed by CI/CD, you don't have any issue adding/changing additional modules, commit, and the CI/CD will take care of the rest.

But what's the corresponding best practice solution for maintaining, say, bash scripts with aws cli commands, and each time new script is created should be run once?

I have a lot of manual solutions, like create a repo and keep updating it with removing applied scripts/adding new script each time. Or build new tool for maintaining the state of which scripts already done and just run new scripts each time. But I'm asking if there is already a best practice solution whether from AWS, 3rd party or there is no.

You can run (bash) scripts from within terraform no problem. We e.g. did this when specifying the lake formation admins. null_resources with proper triggers work quite well if you know what you are doing ;) — luk2302

Helder Sepulveda Helder Sepulveda · Accepted Answer · 2021-11-13T15:55:42

If I was in your shoes I would keep everything in terraform and encourage the team to do the same, you can use null_resource to run scripts and have a lot of fancy triggers...
Here are some examples:

variable "vpc_ids" {
  default = [
    "vpc1",
    "vpc2",
    "vpc3",
  ]
}

resource "null_resource" "tion" {
  count = length(compact(var.vpc_ids))

  triggers {
    vpcs = join(",", var.vpc_ids)
  }

  provisioner "local-exec" {
    when    = "create"
    command = "echo create ${var.vpc_ids[count.index]}"
  }

  provisioner "local-exec" {
    when    = "destroy"
    command = "echo destroy ${var.vpc_ids[count.index]}"
  }
}

resource "null_resource" "sh_test" {
  provisioner "local-exec" {
    when        = "create"
    command     = "if [ '${join(",", var.vpc_ids)}' != '' ]; then echo 'GOOD TO GO'; else echo 'NO VPC FOUND'; fi"
    interpreter = ["/bin/sh", "-c"]
  }
}

Ideally, if you can code, and the AWS feature is available in their API the proper thing to do is to change the AWS terraform provider, the code is open source and hosted at GitHub:
https://github.com/hashicorp/terraform-provider-aws

Manage one-time infrastructure jobs in AWS

1 Answers