Terraform and VPC Peering

Question

So we are using terraform pretty extensively at our org and I had a few questions around how others are doing VPC peering. The initial creation of the connection is easy enough. We pull in from the VPC we just created and reference another VPC, it then populates route tables etc.. The problem is with the VPC we just peered with. We now have to manually go to that other networking stack and add the CIDR/PCX id's manually as variables. I wrote a script that kind of handles this a bit more easily for us, but I wanted to ask if anyone is dynamically performing lookups against AWS for any existing VPC's and automatically adding the existing PCX's to the routing table of that VPC.

An example of where this would be valuable would be in an OPS VPC. We have OPS and then dev,prod,qa,stg,uat,cte etc.. So when we create the CTE vpc it auto creates a pcx and links it to ops and routes to ops. However ops doesn't know about this new pcx. So we have to add it manually. I would like for ops to be able to do a resource lookup on it's own and provision it's own resources for any new VPC/PCX's that it finds.

TLDR; A way for bi directional VPC peering to be more dynamic

Answer 1

Assuming you're using a remote state backend, you can pull in the OPS network stack as a remote state data source and then make changes to its routing tables from whichever ephemeral stack you want it to be able to route to.

Will try and do a minimal example (obviously missing lots of boiler plate):

# my_ops_stack.tf

provider "aws" {
  region = "eu-west-1"
}

module "ops_stack" {
  source = "/my/modules/ops_stack"
  cidr   = "10.1.0.0/16"
  // other vars probably
}

// the outputs which will be accessible 
// via the remote state data source:
output "routing_table_id" { 
  value = "${module.ops_stack.routing_table_id}"
}
output "vpc_id" {
  value = "${module.ops_stack.vpc_id}"
}
output "vpc_cidr" {
  value = "10.1.0.0/16"
}

I'll now configure a remote state backend for this stack using terraform cli (this will soon be possible in config):

# Run in the same folder as my_ops_stack.tf
terraform remote config \
  -backend=s3 \
  -backend-config="bucket=my-state-bucket" \
  -backend-config="key=ops-stack/terraform.tfstate" \
  -backend-config="region=eu-west-1"

Now the state backend is configured, any changes you apply to the stack will synchronise to that backend:

terraform apply
# the usual stuff... but now synced with s3!

Now, in the template of your new ephemeral stack (dev,prod,qa,stg,uat,cte etc.):

# my_dev_stack.tf

provider "aws" {
  region = "eu-west-1"
}

// Pull in your ops stack from the remote backend:
data "terraform_remote_state" "ops_stack" {
    backend = "s3"
    config {
        bucket = "my-state-bucket"
        key = "ops-stack/terraform.tfstate"
        region = "eu-west-1"
    }
}

// Create your dev stack
module "dev_stack" {
  source            = "/my/modules/dev_stack"
  cidr              = "10.2.0.0/16"
  // The ops_stack vpc id for creating the peering connection:
  ops_vpc_id        = "${data.terraform_remote_state.ops_stack.vpc_id}"
  // Maybe some security group rules you wanna setup
  allow_access_from = "${data.terraform_remote_state.ops_stack.vpc_cidr}"
  // other vars probably
}

// And use its outputs to add a route to the 
// ops vpc routing table from the dev stack!
resource "aws_route" "ops_to_dev" {
    route_table_id = "${data.terraform_remote_state.ops_stack.routing_table_id}" 
    destination_cidr_block = "10.2.0.0/16" // dev_stack's cidr
    vpc_peering_connection_id = "${module.dev_stack.vpcx_id}"
}

Once you're done with the ephemeral stack, you can safely destroy it and it will even clean up its route in the ops stack.

Hope this is what you were after!

Answer 2

We ended up just writing a wrapper script around this. Whenever we add a new VPC we go to the ops VPC directory and execute this script and it will dynamically populate the variables.tf file with all necessary variables to setup the OPS vpc peering connections/routes.

Example script:

#!/bin/bash
region=$(find . -name "*vars.tf"|cut -d/ -f2|cut -d- -f1-3)
profile=$(find . -name "*vars.tf" -exec grep 'variable "profile"' {} \; |awk '{print $6}'|tr -d '"')
account=$(pwd|cut -d/ -f5|cut -d- -f1)

getData(){
    for id in ${ids[@]}; do
        output=$(aws ec2 describe-vpc-peering-connections --region $region --profile $account --vpc-peering-connection-ids $id)
        cidr=$(echo "$output"|jq '.VpcPeeringConnections[].RequesterVpcInfo.CidrBlock'|tr -d '"')
        if [[ $1 == cidr ]]; then
            echo $cidr
        elif [[ $1 == id ]]; then
            echo $id
        fi
    done
}
checkOps() {
    pwd|grep 'ops' &>/dev/null
}
populateRoutes() {
    if ! checkOps; then
        echo "Must be run from the ops directory"
        exit 1
    fi
    ids=($(aws ec2 describe-vpc-peering-connections --region $region --profile $account --filters "Name=status-code,Values=active"|jq '.VpcPeeringConnections[].VpcPeeringConnectionId'|tr -d '"'))
    if (( ${#ids[@]} == 0 )); then
        echo "No update necessary"
        exit 0
    fi

    cidr_list=($(getData cidr))
    cidr_format=$(echo "${cidr_list[@]}"|tr ' ' ',')
    echo $cidr_format

    id_list=($(getData id))
    id_format=$(echo "${id_list[@]}"|tr ' ' ',')
    echo $id_format

    if (( ${#cidr_list[@]} != ${#id_list[@]} )); then
        echo "CIDR List and ID List do not match"
        exit 1
    fi

    sed -i "/pcx_count/c\variable\ \"pcx_count\"\ \{\ default \=\ \"${#ids[@]}\" \}" ./variables.tf
    sed -i "/ops_cidrs/c\variable\ \"ops_cidrs\"\ \{\ default\ \=\ \"$cidr_format\"\ \}" ./variables.tf
    sed -i "/pcx_ids/c\variable\ \"pcx_ids\"\ \{\ default\ \=\ \"$id_format\"\ \}" ./variables.tf
}

populateRoutes

Answer 3

I am also interested in performing this automation part where terraform accepts a peering request (mostly focusing on inter-account VPC Peering) automatically, when we provide some criteria. I believe it would be fairly easy to create a lookup resource for pending vpc peering request. Using the aws cli, this is doable via something like aws ec2 describe-vpc-peering-connections with some filters like --filter Name=status-code,Values=pending-acceptance but unfortunately i do not see terraform offering such a resource. My initial idea was to perform a lookup for pending peering requests and accept one that has specific criteria: requester_account_id, requester_vpc_id, accepter_vpc_id.. So far, the only way is to either accept manually the peering requests OR providing the peering request ID manually to aws_vpc_peering_connection_accepter