While using terraform to deploy a fairly large infrastructure in AWS, our remote tfstate got corrupted and was deleted.
From the documentation, I gather that terraform refresh should query AWS to get the real state of the infrastructure and update the tfstate accordigly, but that does not happen: my tfstate is untouched and plan + apply give a lot of Already existing errors.
What does terraform refresh really do?
terraform refresh attempts to find any resources held in the state file and update with any drift that has happened in the provider outside of Terraform since it was last ran.
For example, lets say your state file contains 3 EC2 instances with instance ids of i-abc123, i-abc124, i-abc125 and then you delete i-abc124 outside of Terraform. After running terraform refresh, a plan would show that it needs to create the second instance while a destroy plan would show that it only needs to destroy the first and third instances (and not fail to destroy the missing second instance).
Terraform makes a very specific decision to not interfere with things that aren't being managed by Terraform. That means if the resource doesn't exist in its state file then it absolutely will not touch it in any way. This enables you to run Terraform alongside other tools as well as making manual changes in the AWS console. It also means that you can run Terraform in different contexts simply by providing a different state file to use, allowing you to split your infrastructure up into multiple state files and save yourself from catastrophic state file corruption.
To get yourself out of your current hole I suggest you use terraform import liberally to get things back into your state file or, if possible, manually destroy everything outside of Terraform and start from scratch.
In future I would suggest both splitting out state files to apply for more granular contexts and also to store your remote state in an S3 bucket with versioning enabled. You could also look towards tools like Terragrunt to lock your state file to help avoid corruption or wait for the native state file locking in the upcoming 0.9 release of Terraform.
Plan->
In the aforementioned scenario when the 2nd EC2 instance is deleted and when you try again with terraform plan it does refresh the terraform state but in-memory prior to plan.
The refreshed state will be used to calculate this plan, but will not be
persisted in local or remote state storage.
Refresh->
But if I run individually terraform refresh then it actually refreshes the state file which is present locally or on the remote state storage.
Hope you get your answer now...
I know this is a little late to the party but in my recent dealings with terraform I read it still keeps a local state file tfstate.backup in case the remote state file is corrupted. Apparently, I haven't tried this, you can then do terraform state push and it will push the backup to the remote.
Cheers
@Philip Seems I am the last here, going ahead with your answer.
Since till you are here, you already know that Terraform refresh will update local state for any real world drifts and compare with the actual infra in your terraform templates.
And then Terraform will just lay things off which are not part of your desired configuration.
Ex: 1.) Desired SG with 1 rule, somebody manually add another rule to SG. 2) You run Terraform Refresh or Plan, Terraform notes then down in your remote or local state and generates a plan accordingly. 3.) And in You next apply its going to remove that manually added change.
But in case you wanna retain those manual changes, you need to add a lifecycle block to your resource with ignore_changes.
Ex:
resource "aws_security_group" "default" {
lifecycle {
ignore_changes = [ingress]
}
}
