Accessing Postgres RDS from Kubernetes cluster in AWS

Question

My Kubernetes cluster setup has n-tier web application running in dev and test environments on AWS. For the production environment, postgres RDS was chosen, to ensure periodic backup. While creating a postgres RDS instance, kubernetes-vpc was selected for db-subnet to keep networking stuff simple during pilot run. Also, security group selected is the same as kubernetes-minions.

Following is the service and endpoint yaml:

apiVersion: v1
kind: Service
metadata: 
  labels: 
    name: pgsql-rds
  name: pgsql-rds
spec: 
  ports: 
    - port: 5432
      protocol: TCP
      targetPort: 5432

--

apiVersion: v1
kind: Endpoints
metadata:
  name: pgsql-rds
subsets:
- addresses:
  - ip: 52.123.44.55
  ports:
  - port: 5432
    name: pgsql-rds
    protocol: TCP

When web-app service and deployment is created, it's unable to connect to RDS instance. The log is as follows:

java.sql.SQLException: Error in allocating a connection. Cause: Connection could not be allocated because: Connection to pgsql-rds:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

What am I missing? any pointers to resolve the issue appreciated.

Answer 1

This has to do with DNS resolving. When you use the RDS dns name INSIDE the same VPC it will be resolved to a private ip. When you use the same dns name on the internet or another VPC you will get the public ip of the RDS instance.

This is a problem because from another VPC you can not make use of the load balancing feature unless you expose the RDS instance to the public internet.

Answer 2

It's been a while the issue was resolved.
Don't exactly remember now, which step I missed that caused connection problem. But, below are the steps that did work for me.

Pre-requisite: kubernetes cluster is set up with vpc ('k8s-vpc')

1. Create VPC SUBNET
   Go to vpc dashboard, ensure same aws region as k8s minion. (you will see existing 'k8s-vpc')
   Create subnet with each availability zone.
   Select 'k8s-vpc' as vpc from drop-down.
   CIDR could be 172.20.16.0/24 or 172.20.32.0/24

2. Create DB SUBNET and SUBNET GROUP FOR VPC of k8s minion if not already available.
   Go to RDS Dashboard.
   Create subnet group (e.g. my-db-subnet-group) for DB and add all subnet from step 1 to create subnet group.

3. From RDS Dashboard create Parameter Group
   (e.g. my-db-param-group) for Postgres (version 9.5 in this example)
   Copy value for max_connections to the max_prepared_transactions field and save

4. Create RDS instance for Postgres DB
   Launch DB instance -> select Engine Postgres -> Choose stage (Production or Dev/Test) -> Give instance spec.s (instance type & disk space etc.) and specify DB settings (user/password) -> Configure Advanced settings

   1. vpc selection as 'k8s-vpc'
   2. DB subnet should be one created in previous step (my-db-subnet-group)
   3. VPC security group should be from Kubernetes minions - so that no additional config. required for access from minions
   4. Select Publicly Accessible - to connect to postgres from internet
   5. Select Parameter Group as 'my-db-param-group'.
   6. Specify Database options, backup and maintenance options and finally launch the instance

5. Also check security group of VPC and add inbound rule to allow connection to postgres port.

6. You can test connection from one of the k8s pod (kubectl exec -it) where postgres client is installed.
   Make sure to change user to postgres.
   Connect to RDS using psql as shown below:
   $ psql --host=my-rds-dev.cyhi3va0y2or.ap-northeast-1.rds.amazonaws.com --port=5432 --username=<masterUserName> --password --dbname=<masterDB>

If everything is set up correctly, it should prompt you for password of db user.
Providing correct password will finally connect to RDS.

This article was of great help.

Answer 3

Your IP is of the form: 52.123.44.55. This is a public IP. See the official RFC

Since you said both are in the same VPC, you could have used the internal IP address instead.

That said, the error "Connection to pgsql-rds:5432 refused" means that the address was resolved, otherwise you would get "psql: error: could not translate host name "psql-rds" to address: Name or service not known". Therefore, it is not a DNS issue as cited on another answer.

The cause of the block is likely that the security group was not configured to accept requests from the EC2 instance external IP address. This if the official AWS documentation on connecting to RDS scenarios.

You might have already whitelisted all connections from VPC, however double check the security groups. I would not recommend using a whitelist for the external IP address, however it works if you put the external IP address there. It is a security concern when you don't have an elastic IP address and there are data transfer costs unless you have a more complex setup.

That said, you could have avoided the Kubernetes resources and used the DNS address of the RDS instance.

If you had to avoid using the DNS address of the RDS instance directly, you could have used the following:

apiVersion: v1
kind: Service
metadata:
  name: psql-rds
spec:
  externalName: my-rds-dev.cyhi3va0y2or.ap-northeast-1.rds.amazonaws.com
  ports:
  - port: 5432
    protocol: TCP
    targetPort: 5432
  sessionAffinity: None
  type: ExternalName

With the setup above, you don't need a Kubernetes Endpoint. You can just use psql-rds, or any variation using the namespace as the domain or the fully qualified version, such as psql-rds.default. This is the documentation for the ExternalName

I see that the original poster mentioned the problem was solved, however it is not clear or well documented that the problem was a combination of using the external IP address and checking the security group rules.