How to route between two subnets in an AWS VPC w/ Terraform?

Question

UPDATE: Been working on this off and on among other things. Cannot seem to get a working config w/ two subnets and an SSH bastion. Placing bounty for a full .tf file config that: * creates two private subnets * creates a bastion * spins an ec2 instance on each subnet configured via the bastion (run some arbitrary shell command via the bastion) * has an internet gateway configured * has a nat gateway for the hosts on the private subnets * has routes and security groups configured accordingly

Original post: I am trying to learn Terraform and build a prototype. I have an AWS VPC configured via Terraform. In addition to a DMZ subnet, I have a public subnet 'web' that receives traffic from the internet. I have a private subnet 'app' that is not accessible from the internet. I am trying to configure a bastion host so that terraform can provision instances on the private 'app' subnet. I have not yet been able to get this to work.

When I ssh in to the bastion, I cannot SSH from the bastion host to any instances within the private subnet. I suspect there is a routing problem. I have been building this prototype via several available examples and the documentation. Many of the examples use slightly different techniques and terraform routing definitions via the aws provider.

Can someone please provide the ideal or proper way to define these three subnets (public 'web', public 'dmz' w/ a bastion, and private 'app') so that instances on the 'web' subnet can access the 'app' subnet and that the bastion host in the DMZ can provision instances in the private 'app' subnet?

A snip of my configs are below:

resource "aws_subnet" "dmz" {
    vpc_id = "${aws_vpc.vpc-poc.id}"
    cidr_block = "${var.cidr_block_dmz}"
}

resource "aws_route_table" "dmz" {
    vpc_id = "${aws_vpc.vpc-poc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        gateway_id = "${aws_internet_gateway.gateway.id}"
    }
}

resource "aws_route_table_association" "dmz" {
    subnet_id = "${aws_subnet.dmz.id}"
    route_table_id = "${aws_route_table.dmz.id}"
}

resource "aws_subnet" "web" {
    vpc_id = "${aws_vpc.vpc-poc.id}"
    cidr_block = "10.200.2.0/24"
}

resource "aws_route_table" "web" {
    vpc_id = "${aws_vpc.vpc-poc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        instance_id = "${aws_instance.bastion.id}"
    }
}

resource "aws_route_table_association" "web" {
    subnet_id = "${aws_subnet.web.id}"
    route_table_id = "${aws_route_table.web.id}"
}

resource "aws_subnet" "app" {
    vpc_id = "${aws_vpc.vpc-poc.id}"
    cidr_block = "10.200.3.0/24"
}

resource "aws_route_table" "app" {
    vpc_id = "${aws_vpc.vpc-poc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        instance_id = "${aws_instance.bastion.id}"
    }
}

resource "aws_route_table_association" "app" {
    subnet_id = "${aws_subnet.app.id}"
    route_table_id = "${aws_route_table.app.id}"
}

Answer 1

Here is a snippet that may help you. This was untested but was pulled from one of my terraform files where I provision VMs in a private subnet. I know this works with one private subnet, I tried to implement two here like your original question.

I jump through my NAT instances to hit and provision private subnet boxes with Terraform. If your security groups are set up correctly it does work. It took some experimentation for me.

/* VPC creation */
resource "aws_vpc" "vpc_poc" {
  cidr_block = "10.200.0.0/16"
}

/* Internet gateway for the public subnets */
resource "aws_internet_gateway" "gateway" {
  vpc_id = "${aws_vpc.vpc_poc.id}"
}

/* DMZ subnet - public */
resource "aws_subnet" "dmz" {
    vpc_id = "${aws_vpc.vpc_poc.id}"
    cidr_block = "10.200.1.0/24"
    /* may help to be explicit here */
    map_public_ip_on_launch = true
    /* this is recommended in the docs */
    depends_on = ["aws_internet_gateway.gateway"]
}

resource "aws_route_table" "dmz" {
    vpc_id = "${aws_vpc.vpc_poc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        gateway_id = "${aws_internet_gateway.gateway.id}"
    }
}

resource "aws_route_table_association" "dmz" {
    subnet_id = "${aws_subnet.dmz.id}"
    route_table_id = "${aws_route_table.dmz.id}"
}

/* Web subnet - public */
resource "aws_subnet" "web" {
    vpc_id = "${aws_vpc.vpc_poc.id}"
    cidr_block = "10.200.2.0/24"
    map_public_ip_on_launch = true
    depends_on = ["aws_internet_gateway.gateway"]
}

resource "aws_route_table" "web" {
    vpc_id = "${aws_vpc.vpc_poc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        /* your public web subnet needs access to the gateway */
        /* this was set to bastion before so you had a circular arg */
        gateway_id = "${aws_internet_gateway.gateway.id}"
    }
}

resource "aws_route_table_association" "web" {
    subnet_id = "${aws_subnet.web.id}"
    route_table_id = "${aws_route_table.web.id}"
}

/* App subnet - private */
resource "aws_subnet" "app" {
    vpc_id = "${aws_vpc.vpc_poc.id}"
    cidr_block = "10.200.3.0/24"
}

/* Create route for DMZ Bastion */
resource "aws_route_table" "app" {
    vpc_id = "${aws_vpc.vpc_poc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        /* this send traffic to the bastion to pass off */
        instance_id = "${aws_instance.nat_dmz.id}"
    }
}

/* Create route for App Bastion */
resource "aws_route_table" "app" {
    vpc_id = "${aws_vpc.vpc_poc.id}"
    route {
        cidr_block = "0.0.0.0/0"
        /* this send traffic to the bastion to pass off */
        instance_id = "${aws_instance.nat_web.id}"
    }
}

resource "aws_route_table_association" "app" {
    subnet_id = "${aws_subnet.app.id}"
    route_table_id = "${aws_route_table.app.id}"
}

/* Default security group */
resource "aws_security_group" "default" {
  name = "default-sg"
  description = "Default security group that allows inbound and outbound traffic from all instances in the VPC"
  vpc_id = "${aws_vpc.vpc_poc.id}"

  ingress {
    from_port   = "0"
    to_port     = "0"
    protocol    = "-1"
    self        = true
  }

  egress {
    from_port   = "0"
    to_port     = "0"
    protocol    = "-1"
    self        = true
  }
}

/* Security group for the nat server */
resource "aws_security_group" "nat" {
  name        = "nat-sg"
  description = "Security group for nat instances that allows SSH and VPN traffic from internet. Also allows outbound HTTP[S]"
  vpc_id      = "${aws_vpc.vpc_poc.id}"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    /* this your private subnet cidr */
    cidr_blocks = ["10.200.3.0/24"]
  }
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    /* this is your private subnet cidr */
    cidr_blocks = ["10.200.3.0/24"]
  }
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    /* this is the vpc cidr block */
    cidr_blocks = ["10.200.0.0/16"]
  }
  egress {
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

/* Security group for the web */
resource "aws_security_group" "web" {
  name = "web-sg"
  description = "Security group for web that allows web traffic from internet"
  vpc_id = "${aws_vpc.vpc_poc.id}"

  ingress {
    from_port = 80
    to_port   = 80
    protocol  = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 443
    to_port   = 443
    protocol  = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

/* Install deploy key for use with all of our provisioners */
resource "aws_key_pair" "deployer" {
  key_name   = "deployer-key"
  public_key = "${file("~/.ssh/id_rsa")}"
}

/* Setup NAT in DMZ subnet */
resource "aws_instance" "nat_dmz" {
  ami               = "ami-67a54423"
  availability_zone = "us-west-1a"
  instance_type     = "m1.small"
  key_name          = "${aws_key_pair.deployer.id}"
  /* Notice we are assigning the security group here */
  security_groups   = ["${aws_security_group.nat.id}"]

  /* this puts the instance in your public subnet, but translate to the private one */
  subnet_id         = "${aws_subnet.dmz.id}"

  /* this is really important for nat instance */
  source_dest_check = false
  associate_public_ip_address = true
}

/* Give NAT EIP In DMZ */
resource "aws_eip" "nat_dmz" {
  instance  = "${aws_instance.nat_dmz.id}"
  vpc       = true
}

/* Setup NAT in Web subnet */
resource "aws_instance" "nat_web" {
  ami               = "ami-67a54423"
  availability_zone = "us-west-1a"
  instance_type     = "m1.small"
  key_name          = "${aws_key_pair.deployer.id}"
  /* Notice we are assigning the security group here */
  security_groups   = ["${aws_security_group.nat.id}"]

  /* this puts the instance in your public subnet, but translate to the private one */
  subnet_id         = "${aws_subnet.web.id}"

  /* this is really important for nat instance */
  source_dest_check = false
  associate_public_ip_address = true
}

/* Give NAT EIP In DMZ */
resource "aws_eip" "nat_web" {
  instance  = "${aws_instance.nat_web.id}"
  vpc       = true
}

/* Install server in private subnet and jump host to it with terraform */
resource "aws_instance" "private_box" {
  ami           = "ami-d1315fb1"
  instance_type = "t2.large"
  key_name      = "${aws_key_pair.deployer.id}"
  subnet_id     = "${aws_subnet.api.id}"
  associate_public_ip_address = false

  /* this is what gives the box access to talk to the nat */
  security_groups = ["${aws_security_group.nat.id}"]

  connection {
    /* connect through the nat instance to reach this box */
    bastion_host = "${aws_eip.nat_dmz.public_ip}"
    bastion_user = "ec2-user"
    bastion_private_key = "${file("keys/terraform_rsa")}"

    /* connect to box here */
    user = "ec2-user"
    host = "${self.private_ip}"
    private_key = "${file("~/.ssh/id_rsa")}"
  }
}

Answer 2

Unless that bastion host is also acting as a NAT (I wouldn't advise you to combine roles on the same instance) then the web and app subnets will not have any outbound web access but otherwise that looks fine routing wise as TF will automatically add a local routing record for the VPC.

As long as you have that local routing record that covers your VPC range then routing should be fine. Taking your Terraform configuration file (and adding the minimum necessary resources) allows me to create some basic instances in all 3 subnets and route between them successfully so you're likely missing something else such as security groups or NACLs.

Answer 3

You haven't given your full Terraform, but you'll need to allow SSH into your 'app' VPC instances from either the bastion IP or the CIDR block of your bastion host, so something like this:

resource "aws_security_group" "allow_ssh" {
  name = "allow_ssh"
  description = "Allow inbound SSH traffic"

  ingress {
      from_port = 22
      to_port = 22
      protocol = "tcp"
      cidr_blocks = ["${aws_instance.bastion.private_ip}/32"]
  }
}

Then in your 'app' instance resources, you'll want to add the security group:

...
vpc_security_group_ids = ["${aws_security_group.allow_ssh.id}"]
...

https://www.terraform.io/docs/providers/aws/r/security_group_rule.html

Answer 4

I don't see a reason for Bastion host.

I have something similar using saltstack, I just control the rest using a master server inside a VPC, assign specific security group to it to allow access.

CIDR X/24
subnetX.0/26- subnet for control server. <aster server ip EC2-subnet1/32
subnetX.64/26 - private minions 
subentX.128/26 - public minions
subnetX.192/26- private minions 

then create one route table for EACH subnet for your love of isolation Attach each route to individual subnet. E.g.

rt-1  - subnetX.0/26
rt-2  - subnetX.64/26
rt-3  - subnetX.128/26
rt-4  - subnetX.192/26

make sure your route table has something such this so the route is possible for rt-1 instance connect to everyone

destination: CIDR X/24  Target: local

Then restrict the connectivity through security group inbound.e.g. allow SSH from EC2-subnet1/32

Once I am done all the work with the control server, I can delete the specific route that say CIDR X/24 Target: local in my public subnet, so it no longer able route traffic to my local CIDR.

There is no reason for me to create complicate bastion since I already given the power to remove route in the control server.

Answer 5

You should check the network issues with tcpdump and other debug tools. Please check that:

1. Ips are reacheble and the network setting are right (e.g. 10.200.2.X can reach ip of the bastion host)
2. That iptables/another firewall doesn't block your traffic
3. a ssh server is listen (ssh to an ip of those host from those host)
4. You have right security groups for hosts (you can see this in proterties of EC2 instances)
5. Try to sniff the traffic with tcpdump