Terraform AWS Provider: SecretsManager can't apply because version was deleted

Question

We have an AWS SecretsManager Secret that was created once. That secret will be updated by an external job every hour. I have the problem that sometimes the terraform plan/apply fails with the following message:

AWS Provider 2.48

    Error: Error refreshing state: 1 error occurred:

        * module.xxx.xxx: 1 error occurred:

        * module.xxx.aws_secretsmanager_secret_version.xxx: 
    aws_secretsmanager_secret_version.xxx: error reading Secrets Manager Secret Version: InvalidRequestException: You can't perform this operation on secret version 68AEABC3-34BE-4723-8BF5-469A44F9B1D9 because it was deleted.

We've tried two solutions: 1) Force delete the whole secret via aws cli, but this has the side effect that one of our dependend resources will also be recreated (ecs template definition depends on that secret). This works, but we do not want the side effect of recreating the ecs thing. 2) Manually edit the backend .tfstate file and set the current AWS secret version. Then run the plan again.

Both solution seem to be hacky in a way. What is the best way to solve this issue ?

Answer 1

You can use terraform import to reconcile the state difference before you run a plan or apply.

In your case, this would look like:

terraform import module.xxx.aws_secretsmanager_secret_version.xxx arn:aws:secretsmanager:some_region:some_account_id:secret:example-123456|xxxxx-xxxxxxx-xxxxxxx-xxxxx

Answer 2

I think perhaps the problem you are having is that by default AWS tries to "help you" by not letting you delete secrets automatically until 7 days have elapsed. AWS tries the "help you" by telling you they give you a grace period of 7 days to update your "code" that may rely on this. Which makes automation more difficult.

I have worked around this by setting the recovery window period to "0 days", effectively eliminating that grace period that AWS provides.

Then you can have terraform, rename, or delete your secret at will, either manually (via AWS CLI) or via terraform.

You can update an existing secret by putting in this value FIRST. Then change the name of the secret (if you wish to), or delete it (this terraform section) as desired and run the terraform again after the recovery window days = 0 has been applied.

Here is an example:

resource "aws_secretsmanager_secret" "mySecret" {
    name = "your secret name"
    recovery_window_in_days = "0"

   // this is optional and can be set to true | false
    lifecycle {
    create_before_destroy = true
    }
}

*Note, there is also an option to "create before destroy" you can set on the lifecyle.

https://www.terraform.io/docs/configuration/resources.html

Answer 3

Also, you can use the terraform resource to update the secret values like this:

This example will set the secret values once and then tell terraform to ignore any changes made to the values (username, password in this example) after the initial creation.

If you remove the lifecyle section, then terraform will keep track of whether or not the secret values themselves have changed. If they have changed they would revert back to the value in the terraform state.

If you store your tfstate files in an s3 protected bucket that is safer than not doing so, because they are plaintext in the statefile, so anyone with access to your terraform state file could see your secret values.

I would suggest: 1) figuring out what is deleting your secrets unexpectedly? 2) having your "external job" be a terraform bash script to update the values using a resource as in the example below.

Hope this gives you some ideas.

resource "aws_secretsmanager_secret_version" "your-secret-data" {
      secret_id     = aws_secretsmanager_secret.your-secret.id
      secret_string = <<-EOF
              {
          "username": "usernameValue",
          "password": "passwordValue"
        }
      EOF

      // ignore any updates to the initial values above done after creation. 
      lifecycle {
            ignore_changes = [
              secret_string
            ]
          }
}