Need help understand Terraform state management

Question

This is more a set of conceptual things that I'm trying to wrap my head around when it comes to how Terraform works.

1) Here's a scenario. Lets say I create a simple plan that creates a couple machines in AWS. Init, apply, it builds the infrastructure and creates a local state file. Satisfied with the code, I push it up to github. My question here is should I also push up the state file? Without the state file, anyone who pulls down the code and tries to build the instances again won't have the state, so I assume terraform tries to build a new set of machines. But, a state file feels like residual data, and therefore should be gitignored.

2) Same scenario, but lets say the plan includes a setup file that implements an S3 backend. Does the same problem exist, or does any run of the plan honor the S3 backend's state?

3) Now, lets say that I want this plan to run as module in a larger plan that deploys an entire environment and stores its state file in S3. The module has already been run so there's already existing machines. Will running the module as part of a larger module act the same way as (1) since it uses the state file of the umbrella plan, or does it use its existing s3 state file?

4) Should backend setup be implemented as an independent module? My thinking is that if someone ever tries to run terraform destroy, they won't destroy the backend, just the infrastructure.

In general, I'm rather confused about the enterprise workflow of Terraform.

Answer 1

I'll try to answer according to my experience with Terraform:

1) An updated Terraform state is necessary to do any operation with Terraform. It is a requirement of the technology. You can share this state via file sharing or git (not recommended, you will have problems if anyone forgets to make git pull, and also the state includes sensitive data that should be protected). It's much better to use one if the standard backends. As you said, if terraform is executed without an updated state it will recreate the whole infrastructure.

2) Effectively, the problem is solved using S3 (or any other) backend. With a backend, Terraform takes care of pulling the remote state before performing any action.

3) No, if you change your plan the state must be manually updated. You can use terraform refresh or terraform import to import existing infrastructure into the state.

4) There must be a backend per terraform plan, it isn't possible to configure a backend in a module. Modules are meant to be exported and shared, so it has to be agnostic about the backend. The backend is no managed by terraform (you should create it before starting to use terraform), so it will not be destroyed in any case. If you want any other resource to be prevented from destroying, you can use the feature prevent_destroy.

Terraform has some caveats that teams should familiarize with, one of them is that changing the structure of the plan implies extra work to reconcile the existing infrastructure with the state.

Anyway, using a remote backend the team will have no problems to work together in the same plan.