Running Terraform on AWS without VPC endpoints or 0.0.0.0/0 rules

Question

I have an EC2 instance on which I will be running Terraform to provision a bunch of other resources in AWS. I don't want to give this server outbound internet access as the only thing it needs to access is the AWS API, so I was going down the VPC endpoint route.

This was working fine until I needed to provision resources for which there is no VPC endpoint (e.g. RDS & Elasticache). I don't want to give this instance internet access and create an outbound 0.0.0.0/0 rule on port 443, but I can't see another way.

Is there a way of restricting outbound HTTPS access to AWS APIs only?

I assume you've already thought about other things that Terraform would normally need to communicate with? Eg fetching provider plugins etc from the registry. This is an interesting question and I think I have a reasonable answer but might take a bit off playing around with things so might need to wait until tomorrow before I can answer fully. — ydaetskcoR

ydaetskcoR ydaetskcoR · Accepted Answer · 2020-03-11T08:15:02

AWS provide a guide for working out the IP addresses of any service endpoints to allow egress to it. This relies on downloading the IP addresses of all AWS endpoints and then filtering for ones in the AMAZON list but removing the ones from the EC2 list.

Thankfully we can get those IP addresses directly in Terraform using the aws_ip_ranges data source and can filter for both the AMAZON ranges and the EC2 ranges:

data "aws_region" "current" {}

data "aws_ip_ranges" "amazon" {
  regions  = [data.aws_region.current.name]
  services = ["amazon"]
}

data "aws_ip_ranges" "ec2" {
  regions  = [data.aws_region.current.name]
  services = ["ec2"]
}

The above example would get all of the IP addresses for the AMAZON and the EC2 blocks (which are a subset of the AMAZON ones) for the region you are running in.

To remove the EC2 blocks from the AMAZON one we need to use the setsubtract function:

locals {
  aws_control_plane = setsubtract(data.aws_ip_ranges.amazon.cidr_blocks, data.aws_ip_ranges.ec2.cidr_blocks)
}

This should give us just the IP ranges that we want to allow our security group egress to.

Unfortunately this is likely to be over 60 CIDR ranges which would equate to more than 60 rules. And a security group is limited to a max of 60 ingress and 60 egress rules:

You can have 60 inbound and 60 outbound rules per security group (making a total of 120 rules). This quota is enforced separately for IPv4 rules and IPv6 rules; for example, a security group can have 60 inbound rules for IPv4 traffic and 60 inbound rules for IPv6 traffic. A rule that references a security group or prefix list ID counts as one rule for IPv4 and one rule for IPv6.

A quota change applies to both inbound and outbound rules. This quota multiplied by the quota for security groups per network interface cannot exceed 1000. For example, if you increase this quota to 100, we decrease the quota for your number of security groups per network interface to 10.

We can, however, have multiple security groups per interface so we can just spread these ranges across multiple security groups and attach multiple security groups to the instance:

To do this we need to split the list of ranges we have into blocks of 60 and then loop over the security group resource that we're going to create. We can do this with the chunklist function:

locals {
  aws_control_plane_chunked = chunklist(local.aws_control_plane, 60)
}

This returns a list of lists with a max of 60 CIDR blocks in each.

We can then create our multiple security groups by iterating over these lists:

resource "aws_security_group" "aws_only_egress" {
  count = length(local.aws_control_plane_chunked)

  name = "aws-only-egress-example-chunk-${count.index + 1}"

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = local.aws_control_plane_chunked[count.index]
  }
}

And then finally we need to attach these multiple security groups to our instance:

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

resource "aws_instance" "example" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"

  vpc_security_group_ids = aws_security_group.aws_only_egress.*.id
}

Running Terraform on AWS without VPC endpoints or 0.0.0.0/0 rules

1 Answers