Why can't I connect AWS RDS instance from EC2 instance in another VPC after peering

Question

I am running Tableau Server on our EC2 instance in VPC A. Meanwhile, I created a postgres RDS in another VPC B. Now I want to establish the connection between the Tableau Server and RDS. CIDR of RDS VPC is 172.31.0.0/16 and that of EC2 VPC is 10.0.0.0/16.

According to A DB Instance in a VPC Accessed by an EC2 Instance in a Different VPC, I created peering between VPC A and VPC B, pcx-xyz123. Besides, I also created the following route tables for the VPCs.

RDS VPC
Destination      Target
172.31.0.0/16    local
10.0.0.0/16      pcx-xyz123

EC2 VPC
Destination      Target
10.0.0.0/16      local
172.31.0.0/16    pcx-xyz123

Both route tables are main. Each has 0 Subnets though (not sure if this matters).

However I still can't connect RDS from Tableau Server.

The two instances are created by same account. They are both listed under US East(Ohio). So I assume they are in the same region. Plus both have us-east-2 in their hostnames. From my PC, I can connect to RDS with psql command or pgAdmin.

Why can't I connect the two instances?

EDIT: I've created another EC2 Linux instance within the same subnet of the same VPC as the Tableau Server just for debugging purposes. I configured the peering and route table same way and also associate the subnets to the route tables. However, I still can't connect to RDS on the EC2 Linux instance.

Answer 1

VPC Peering works much the same way as how Public Subnets connect to the Internet Gateway -- the Route Tables define how traffic goes in/out of the Subnets.

For VPC Peering to work:

• Invite & Accept the peering connection (Done)
• Create a Route table in each VPC that points to the Peering connection for the other VPC's IP range (Done)
• Associate each subnet that you want able to peer to the Route Table
• Alternatively, edit existing route tables to include the peering entry
• If your RDS database is public, and you are attempting to connect using the public DNS of the database, then you will need to edit the DNS settings of your peering connection to allow DNS resolution.

The routing works as follows:

• When traffic leaves a subnet, the Route Table is consulted to determine where to send the traffic
• The most restrictive (eg /24) is evaluated first, through to the least restrictive (eg /0)
• The traffic is routed according to the appropriate Route Table entry

This means that you can configure some of the subnets to peer, rather than having to include all of them. Traditionally, it is the Private subnets that peer and possibly only specific Private subnets -- but that is totally your choice.

Think of it as directions on a roadmap, telling traffic where it should be directed.

Answer 2

Below are the steps to make private RDS accessible via VPC peering:

Let’s say you have 2 VPCs:

• Production VPC: 10.0.1.0/24
• RDS VPC: 10.0.2.0/24

Step 1: create VPC peering connection between the two VPCs. Then accept the request to establish the connection. You will get a connection ID such as: pcx-e8e8e8e8

Step 2: configure route table in each VPC

• Production VPC: add this route to RDS VPC: 10.0.2.0/24 —> pcx-e8e8e8e8
• RDS VPC: add this route to Production VPC: 10.0.1.0/24 —> pcx-e8e8e8e8

Step 3: configure security group of RDS to accept the IP range of Production VPC, by adding this inbound rule

• Port (MS SQL: 1433, MySQL: 3306, etc) — allow source: 10.0.1.0/24

Should be ready for connection now.

Note: when connecting to RDS, you should use the provided DNS name for better resiliency. AWS VPC DNS will take care of resolving this name to a local IP address of the RDS instance.

Answer 3

VPC peering is all about the details. Here are the items we had to run down to get it to work.

Peer VPC 1 to VPC 2 (obvious, but included for those that did not do this step). From VPC 1, establish peering to VPC 2. Accept request. If different region, switch to VPC 2 region and accept the peer request.

Examples:

VPC 1 CIDR = 10.0.0.0/16
VPC 2 CIDR = 172.16.0.0/16

VPC 1 (VPC With RDS Instance)
1. Route Table Servicing Subnet of RDS Instance - Add route destination to VPC 2 CIDR block (172.16.0.0/16) and target VPC 2 peering connection (select from list - pcx-#####).
2. RDS Security Group - Add inbound rule for DB port with source IP being the VPC 2 CIDR block (172.16.0.0/16). So, you will have two inbound rules for the DB port. One for the VPC 1 (10.0.0.0/16) CIDR Block and one for VPC 2 (172.16.0.0/16).
3. Network Access Control List for Private Route Table - if you are only allowing certain ports, add a rule for the DB Port, source = VPC 2 CIDR block (172.16.0.0/16) and Allow.

VPC 2
1. Route Table Servicing Subnet of EC2 Instance - Add route destination to VPC 1 CIDR block (10.0.0.0/16) and target VPC 1 peering connection (select from list - pcx-#####).
2. Instance Security Group - Add inbound rule for DB port with source IP being the VPC 1 CIDR block (10.0.0.0/16).
3. Network Access Control List for Route Table - if you are only allowing certain ports, add a rule for the DB Port, source = VPC 1 CIDR block (10.0.0.0/16) and Allow.

I think that was it - but if I find another setting, I will update this message.

Just some history, we were doing this for disaster recovery. Our production instances and RDS MS SQL DB are in us-east-1 (VPC 1) and our disaster recovery warm standby instances are in us-west-2 (VPC 2). We mostly get traffic from the US, but we may consider making the standby site a true production copy (scaling group) and then changing the Route 5 records to latency based routing.

Answer 4

The question has already been answered but I wanted to add if you are connecting to public DNS of RDS(eg. prod.upd9999upd.us-east-1.rds.amazonaws.com) then you must enable DNS resolution to private IP. This is done through AllowDnsResolutionFromRemoteVpc.

Example: To connect Vpc EC2_PROD(172.0.0.0/16) to Vpc RDS_PROD(30.0.0.0/16).

1) Create Peering connection from EC2 VPC(Requester) to RDS VPC (Accepter). Make sure to enable AllowDnsResolutionFromRemoteVpc with UI by right clicking on already created peering connection and "Edit DNS Settings". Or with following command

aws ec2 modify-vpc-peering-connection-options --vpc-peering-connection-id "pcx-04a511409bb08ef16" --requester-peering-connection-options '{"AllowDnsResolutionFromRemoteVpc":true}' --accepter-peering-connection-options '{"AllowDnsResolutionFromRemoteVpc":true}' --region us-east-1

Your final peering connection will look like:

aws ec2 describe-vpc-peering-connections --profile aws-work --region us-east-1
{
    "VpcPeeringConnections": [
        {
            "Status": {
                "Message": "Active", 
                "Code": "active"
            }, 
            "Tags": [
                {
                    "Value": "ec2-to-rds-peering-connection", 
                    "Key": "Name"
                }
            ], 
            "AccepterVpcInfo": {
                "PeeringOptions": {
                    "AllowEgressFromLocalVpcToRemoteClassicLink": false, 
                    "AllowDnsResolutionFromRemoteVpc": true, 
                    "AllowEgressFromLocalClassicLinkToRemoteVpc": false
                }, 
                "VpcId": "vpc-RDS", 
                "Region": "us-east-1", 
                "OwnerId": "?", 
                "CidrBlockSet": [
                    {
                        "CidrBlock": "30.0.0.0/16"
                    }
                ], 
                "CidrBlock": "30.0.0.0/16"
            }, 
            "VpcPeeringConnectionId": "pcx-04a511409bb08ef16", 
            "RequesterVpcInfo": {
                "PeeringOptions": {
                    "AllowEgressFromLocalVpcToRemoteClassicLink": false, 
                    "AllowDnsResolutionFromRemoteVpc": true, 
                    "AllowEgressFromLocalClassicLinkToRemoteVpc": false
                }, 
                "VpcId": "vpc-ec2", 
                "Region": "us-east-1", 
                "OwnerId": "?", 
                "CidrBlockSet": [
                    {
                        "CidrBlock": "172.0.0.0/16"
                    }
                ], 
                "CidrBlock": "172.0.0.0/16"
            }
        }
    ]
}

2) Your Requester VPC (Ec2 VPC) route table must have Accepter IP cider(eg 30.0.0.0/16) added. (see Routes tab below)

aws ec2 describe-route-tables --filters Name=tag:Name,Values=EC2_PROD --profile aws-work --region us-east-1
{
    "RouteTables": [
        {
            "Associations": [
                {
                    "RouteTableAssociationId": "rtbassoc-?", 
                    "Main": true, 
                    "RouteTableId": "rtb-?"
                }
            ], 
            "RouteTableId": "rtb-?", 
            "VpcId": "vpc-EC2_PROD", 
            "PropagatingVgws": [], 
            "Tags": [
                {
                    "Value": "EC2_PROD", 
                    "Key": "Name"
                }
            ], 
            "Routes": [
                {
                    "GatewayId": "local", 
                    "DestinationCidrBlock": "172.0.0.0/16", 
                    "State": "active", 
                    "Origin": "CreateRouteTable"
                }, 
                {
                    "Origin": "CreateRoute", 
                    "DestinationCidrBlock": "30.0.0.0/16",    // Accepter IP cider block
                    "State": "active", 
                    "VpcPeeringConnectionId": "pcx-04a511409bb08ef16"
                }, 
                {
                    "GatewayId": "igw-???", 
                    "DestinationCidrBlock": "0.0.0.0/0", 
                    "State": "active", 
                    "Origin": "CreateRoute"
                }
            ]
        }
    ]
}

3) Similarly Acceptor VPC (RDS VPC) route table must have Requester IP cider(eg. 172.0.0.0/16) added. (see Routes tab below)

aws ec2 describe-route-tables --filters Name=tag:Name,Values=RDS_PROD --profile aws-work --region us-east-1
{
    "RouteTables": [
        {
            "Associations": [
                {
                    "SubnetId": "subnet-?", 
                    "RouteTableAssociationId": "rtbassoc-?", 
                    "Main": false, 
                    "RouteTableId": "rtb-?"
                }
            ], 
            "RouteTableId": "rtb-?", 
            "VpcId": "vpc-RDS", 
            "PropagatingVgws": [], 
            "Tags": [
                {
                    "Value": "RDS_PROD", 
                    "Key": "Name"
                }
            ], 
            "Routes": [
                {
                    "Origin": "CreateRoute", 
                    "DestinationCidrBlock": "172.0.0.0/16",    // Requester IP cider block
                    "State": "active", 
                    "VpcPeeringConnectionId": "pcx-04a511409bb08ef16"
                }, 
                {
                    "GatewayId": "local", 
                    "DestinationCidrBlock": "30.0.0.0/16", 
                    "State": "active", 
                    "Origin": "CreateRouteTable"
                }, 
                {
                    "GatewayId": "igw-???", 
                    "DestinationCidrBlock": "0.0.0.0/0", 
                    "State": "active", 
                    "Origin": "CreateRoute"
                }
            ]
        }
    ]
}

4) Finally also update firewall/ security group on Accepter VPC(RDS) to allow connection from Ec2 VPC on port 3306 if its mysql.

aws ec2 describe-security-groups --filters Name=tag:Name,Values=RDS_FIREWALL --profile aws-work --region us-east-1
{
    "SecurityGroups": [
        {
            "IpPermissionsEgress": [
                {
                    "IpProtocol": "-1", 
                    "PrefixListIds": [], 
                    "IpRanges": [
                        {
                            "CidrIp": "0.0.0.0/0"
                        }
                    ], 
                    "UserIdGroupPairs": [], 
                    "Ipv6Ranges": []
                }
            ], 
            "Description": "Dev", 
            "Tags": [
                {
                    "Value": "RDS_FIREWALL", 
                    "Key": "Name"
                }
            ], 
            "IpPermissions": [
                {
                    "PrefixListIds": [], 
                    "FromPort": 3306, 
                    "IpRanges": [
                        {
                            "Description": "EC2_VPC_IP_CIDER", 
                            "CidrIp": "172.0.0.0/16"
                        }
                    ], 
                    "ToPort": 3306, 
                    "IpProtocol": "tcp", 
                    "UserIdGroupPairs": [], 
                    "Ipv6Ranges": []
                }
            ], 
            "GroupName": "RDS_FIREWALL", 
            "VpcId": "vpc-???", 
            "OwnerId": "???", 
            "GroupId": "sg-???"
        }
    ]
}

Answer 5

After following all the above, I was still having issues connecting to my RDS instance. For the first time, well, ever, I actually found the answer on Reddit not SO (here: https://www.reddit.com/r/aws/comments/8hx28w/rds_access_from_a_different_vpc/dyn616i/).

Tl;Dr If you've already peered VPC's, modified route tables, and opened up the DB security group to allow connections from the source VPC CIDR (basically, @John Rotenstein's suggestion https://stackoverflow.com/a/46331624/1830623), then make sure your RDS instance is not marked as Public.

Answer 6

I faced a similar issue, this is what I did:

1. Created VPC peering, as discussed in the thread above.

2. In the security group of the RDS instance, you can add a rule which will allow port 5432 (the Postgres port) Securitygroupid/hostIP/VPC_CIDR.

3. Login to the EC2 instance to install psql (sudo yum install postgresql-server postgresql-contrib) and execute the following command:

   # this will ask for the password and connect.
   psql --host=xxxxxx.us-west-2.rds.amazonaws.com --port=5432 --username=xxxxx --password --dbname=xxxxx
   

Answer 7

Peering will not work if you have the same CIDR. The same CIDR would occur if you cloned the instance from one region to another. Within the route, you won't able to define the same CIDR.