How to tell teamcity agent, which aws account to run terraform to

Question

I spun up TeamCity into a Kubernetes cluster. I have one deployment for the TeamCity server and one deployment for the TeamCity build agent. When I run Terraform on the TeamCity build agent, it creates resources in the same AWS account as the EC2 instance which is hosting TeamCity Kubernetes cluster.

I would like to run a build, which creates AWS resources in a separate AWS account. My thinking is to assign an AWS access key and secret key to the TeamCity server and pass them to the TeamCity build agent, but I don't know how the work flow is going to be.

Currently, I have declared AWS access key and secret keys as environment variables on the build but they are not getting passed to agent. My build steps only contains 3 lines

terraform init
terraform plan
terraform apply -auto-approve

ShaneC ShaneC · Accepted Answer · 2020-05-28T13:10:04

I dealt with a very similar situation. When we started using AWS, we did not follow best practice around how our scripts authenticate to AWS.

We created users in each account and exported the access and secret keys into our TeamCity build automation to allow our scripts to run in the context of these accounts.

You will wind up with several users in different accounts and will wind up manually rotating the keys for these users, then updating the access and secret keys in TeamCity everytime you need to rotate the keys.

As the TeamCity build agent servers are EC2 instances, it is best practice in AWS to use roles instead of users in situations like this. From the AWS documentation:

Applications that run on an Amazon EC2 instance need credentials in order to access other AWS services. To provide credentials to the application in a secure way, use IAM roles. A role is an entity that has its own set of permissions, but that isn't a user or group. Roles also don't have their own permanent set of credentials the way IAM users do. In the case of Amazon EC2, IAM dynamically provides temporary credentials to the EC2 instance, and these credentials are automatically rotated for you.

Source: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#use-roles-with-ec2

What you should do in this situation is create a role in each account that has access to create the resources that terraform will build. You will need another role for the TeamCity build agent servers, this role should be able to assume the roles created in each account.

Let's say you have two accounts:

AccountA is where you want the build agents to run build resources with terraform.
AccountB is the account your build agents are running from.

What you will need is:

A role in AccountA with access to create resources.
A role in AccountB that grants access for ec2 instances to assume the above role.

The instance profile of the role in AccountB should be assigned to the TeamCity build agent servers. The build agent servers are then able to assume the role in AccountA and run in the context of that account with the permissions they need without the need for hard-coded access and secret keys.

You can repeat this process for as many other accounts as you want to build resources in through Terraform.

You will have to add the following block to the aws provider in your terraform scripts to let terraform know what role to assume:

provider "aws" {
assume_role {
role_arn = "arn:aws:iam::<ACCOUNTA>:role/<ROLENAME>"
}
}

Here is some AWS documentation on how to grant access to an S3 bucket in a different account from an EC2 instance: https://aws.amazon.com/premiumsupport/knowledge-center/s3-instance-access-bucket/

Another useful documentation page on that covers IAM Roes, EC2 instance profiles & cross account access via Roles is here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

One last link goes through exactly what you are trying to do: https://blog.e-zest.com/aws-ec2-assume-role-with-terraform

How to tell teamcity agent, which aws account to run terraform to

2 Answers