use different bucket for terraform s3 backend depending on which aws account is configured

Question

I need my terraform s3 backend to use one bucket for my production AWS account, and another bucket for my development AWS account. We need this because we can't allow users from the development AWS account to access the s3 bucket in the production AWS account. S3 bucket names must be globally unique, so the bucket name field can't be the same.

I tried using a variable here but variables error in terraform backends. This is a very requested feature, and there's a GitHub issue for it, but it's been open for 4 years and there doesn't seem to be any plan to add this feature, so I need a workaround. One suggestion comes from that same GitHub issue. That suggestion is to use a terraform remote state data source. Unfortunately, this doesn't seem to work. Here's what I tried:

// backend.tf

terraform {
  backend "s3" {}
}
data terraform_remote_state "state" {
  backend = "s3"
  config {
    bucket = var.aws_account == "123456789000" ? "my-prod-bucket" : "my-dev-bucket"
    key    = "apps/main-stack.tfstate"
    region = "us-east-1"
  }
}

None of the values get used, and it prompts for all of them to be entered manually.

$ terraform init
Initializing modules...

Initializing the backend...
bucket
The name of the S3 bucket

Enter a value:

I looked around for other solutions, but so far, no luck. Does anyone know how to fix this?

Does this answer your question? How to pass variables for Terraform S3 Backend resource? — Chin Huang
@ChinHuang excellent answer. There was definitely overlap there. I've updated my question to focus more on the specific issue of achieving aws account to s3 bucket parity and have linked to your answer for the more general case of variables in backends. — williamcodes

williamcodes williamcodes · Accepted Answer · 2021-03-09T11:46:21

Seems like the solution is to use a backend-config file. This is called partial configuration. You can keep a tfvars file for each aws account you need a separate backend for, and provide it when you initialize terraform like so:

// prod-backend-config.tfvars

bucket = "my-prod-s3-bucket-for-terraform"

// dev-backend-config.tfvars

bucket = "my-dev-s3-bucket-for-terraform"

// backend.tf

terraform {
  backend "s3" {
    // do not set a bucket name here
    key    = "apps/main-stack.tfstate"
    region = "us-east-1"
  }
}

$ terraform init -backend-config prod-backend-config.tfvars

This allows you to maintain parity between aws accounts and s3 buckets for your backend.

For a more general solution to configuring any backend with arbitrary variables, see How to pass variables for Terraform S3 Backend resource?

use different bucket for terraform s3 backend depending on which aws account is configured

1 Answers