Terraform fails because tfstate (S3 backend) is lost

2
votes

I am creating AWS infrastructure using Terraform, and using S3 backend configuration. Now the issue is, someone deleted the S3 bucket storing the state, and now every time I run terraform it fails saying the resources already exist. The old tfstate is lost, and new has no information about existing resources. Note: I do not have write access to the AWS environment. I trigger terraform via Jenkins CD pipeline, so I cannot manually modify the infrastructure or run any terraform command.

Is there a way to cleanup existing resources or force recreating resources(if they already exist) with tf file? This is the only place I can make changes.

2
amazon-web-servicesterraform
may be you have some Jenkins job to run destroy? – GiorgosDev
Possible duplicate of What does terraform refresh really do? – ydaetskcoR
As mentioned in the target duplicate question, you will need to manually clean up stuff and rebuild or import everything back into your state file. In future you should enable versioning of your state file so you can get back to it at any point and probably prevent the destruction of the S3 bucket using MFA Delete etc. – ydaetskcoR
Thanks for the information @ydaetskcoR. I cannot cleanup the stuff because I do not have permissions. I will get it done by following a process. – Sumit Jindal
If someone has the local state, you can copy it to the new s3 buucker or a combination of terraform import and and terraform state rm will do the trick. – victor m

2 Answers

2
votes

You really are in a mess. You need to restore the S3 bucket or make a new one and point your code at that. You then need to recreate the state you lost, that or delete every object you created via Terraform and start again. Most objects have the ability to import existing objects via the Terraform import command.

This could be a significantly large task.

And you'd be needing write access to the bucket? Terraform refresh is only going to help if you still had the state file. You don't. If you haven't got permission to do that, then maybe give up that or persist in getting sufficient privilege.

If you can't run Terraform locally then you are also wasting your time. Good luck.

However....

You don't want to be here again. How did you delete/lose the bucket? You really need that never to happen again as @ydaetskcoR said some MFA protection on the bucket - definitely do that and adding versioning to it is a REALLY good idea. Also if you haven't added DynamoDB locking to the bucket do so, its really worth it.

1
votes

If you're working in a production account, follow the advice of others and do not mess with anything in the account manually!

If you are just starting out with terraform or terragrunt and you're trying to reset the terraform state:

  1. Ensure that you are logged into the correct AWS account, it is not a production account, and it is not in use by anyone else
  2. Terraform state is saved in two places under the same name: S3 and DynamoDB. In order to reset the state, you will need to delete both.
  3. If there is anything else that was previously created in the account, you will need to delete those manually. Depending on how much you created, this could take a very long time.
  4. Once you have deleted the S3 and DynamoDB named after your terraform state bucket and deleted the infrastructure created by terraform, you should be able to terraform init and terraform plan without further issue.