I work with terraform
5 years. I did a lot of mistakes with in my career with modules and environments.
Below text is just share of my knowledge and experience. They may be bad.
Real example project may is hard to find because terraform
is not used to create opensource projects. It's often unsafe to share terraform
files because you are showing all vulnerabilities from your intrastructure
Module purpose and size
You should create module that has single purpose, but your module should be generic.
Example module
You can create bastion host module
, but better idea is to create a module for generic server
. This module may have some logic dedicated to your business problem like, CW Log group
, some generic security group rules
, etc.
Application module
Sometimes it is worth to create more specific module.
Let's say you have application, that requires Lambda
, ECS service
, CloudWatch alarms
, RDS
, EBS
etc. All of that elements are strongly connected.
You have 2 options:
- Create separated modules for each above items - But then your application uses 5 modules.
- Create one big module and then you can deploy your app with single module
- Mix above solutions - I prefer that
Everything depends on details and some circumstances.
But I will show you how I use terraform in my productions in different companies.
Separated definitions for separated resurces
This is project, where you have environment as directories. For each application, networking, data resoruces you have separated state. I keep mutable data in separated directory(like RDS, EBS, EFS, S3, etc) so all apps, networking, etc can be destroyed and recreated, because they are stateless. No one can destroy statefull items because data can be lost. This is what i was doing for last few years.
project/
├─ packer/
├─ ansible/
├─ terraform/
│ ├─ environments/
│ │ ├─ production/
│ │ │ ├─ apps/
│ │ │ │ ├─ blog/
│ │ │ │ ├─ ecommerce/
│ │ │ ├─ data/
│ │ │ │ ├─ efs-ecommerce/
│ │ │ │ ├─ rds-ecommerce/
│ │ │ │ ├─ s3-blog/
│ │ │ ├─ general/
│ │ │ │ ├─ main.tf
│ │ │ ├─ network/
│ │ │ │ ├─ main.tf
│ │ │ │ ├─ terraform.tfvars
│ │ │ │ ├─ variables.tf
│ │ ├─ staging/
│ │ │ ├─ apps/
│ │ │ │ ├─ ecommerce/
│ │ │ │ ├─ blog/
│ │ │ ├─ data/
│ │ │ │ ├─ efs-ecommerce/
│ │ │ │ ├─ rds-ecommerce/
│ │ │ │ ├─ s3-blog/
│ │ │ ├─ network/
│ │ ├─ test/
│ │ │ ├─ apps/
│ │ │ │ ├─ blog/
│ │ │ ├─ data/
│ │ │ │ ├─ s3-blog/
│ │ │ ├─ network/
│ ├─ modules/
│ │ ├─ apps/
│ │ │ ├─ blog/
│ │ │ ├─ ecommerce/
│ │ ├─ common/
│ │ │ ├─ acm/
│ │ │ ├─ user/
│ │ ├─ computing/
│ │ │ ├─ server/
│ │ ├─ data/
│ │ │ ├─ efs/
│ │ │ ├─ rds/
│ │ │ ├─ s3/
│ │ ├─ networking/
│ │ │ ├─ alb/
│ │ │ ├─ front-proxy/
│ │ │ ├─ vpc/
│ │ │ ├─ vpc-pairing/
├─ tools/
To apply single application, You need to do:
cd ./project/terraform/environments/<ENVIRONMENT>/apps/blog;
terraform apply;
You can see there is a lot of directories in all environments. As I can see there are pros and cons of that tools.
Cons:
- It is hard to check if all modules are in sync
- Complicated CI
- Complicated directory structure especially for new people in the team, but it is logic
- There may be a lot of dependencies, but this is not a problem when you think about it from the beginning.
- You need to take care, to keep exactly the same environments.
- There is a lot of initialization required and refactors are hard to do.
Pros:
- Quick apply after small changes
- Separated applications and resources. It is easy to modify small module or small deployment for it without knowledge about overall system
- It is easier to clean up when you remove something
- It's easy to tell what module need to be fixed. I use some tools I wrote to analyze status of particular parts of infrastructure and I can send email to particular developer, that his infrastructure needs resync for some reasons.
- You can have different environments easier than in the monolith. You can destroy single app if you do not need it in environemnt
Monolith infrastructure
Last time I started working with new company. They keep infrastructure definition in few huge repositories(or folders), and when you do terraform apply, you create all applications at the same time.
project/
├─ modules/
│ ├─ acm/
│ ├─ app-blog/
│ ├─ app-ecommerce/
│ ├─ server/
│ ├─ vpc/
├─ vars/
│ ├─ user/
│ ├─ prod.tfvars
│ ├─ staging.tfvars
│ ├─ test.tfvars
├─ applications.tf
├─ providers.tf
├─ proxy.tf
├─ s3.tf
├─ users.tf
├─ variables.tf
├─ vpc.tf
Here you prepare different input values for each environment.
So for example you want to apply changes to prod:
terraform apply -var-file=vars/prod.tfvars -lock-timeout=300s
Apply staging:
terraform apply -var-file=vars/staging.tfvars -lock-timeout=300s
Cons:
- You have no dependency, but sometimes you need to prepare some environment element like domains, elastic IP, etc manually, or you need to have them created before
terraform plan/apply
. Then you have problem
- Its hard to do cleanup as you have hundreds resources and modules at the same time
- Extremely long terraform execution. Here it takes around 45 minutes to plan/apply single environment
- It's hard to understand entire environment.
- Usually you need to have 2/3 repositories if you keep that structure to separate networking,apps,dns etc...
- You need to do much more work to deal with different environments. You need to use count etc...
Pros:
- It's easy to check if your infrastructure is up to date
- There is no complicated directory structure...
- All your environments are exactly the same.
- Refactoring may be easier, because you have all resources in very few places.
- Small number of initialization is required.
Summary
As you can see this is more architectural problem, the only way to learn it, is to get more experience or read some posts from another people...
I am still trying to figure out the most optimal way and I would probably experiment with first way.
Do not take my advantages as sure thing. This post is just my experience, maybe not the best.
References
I will post some references that helped me a lot: