58
votes

From EC2 instance i-78a8df00, I'm trying to connect to RDS instance mysql.************.us-east-1.rds.amazonaws.com. They are both in the U.S. East region. I added the security group of EC2 instance (sg-********) to the RDS security group, but that didn't help. It appears to be a firewall/DNS issue as it is timing out when running this command:

ubuntu@ip-10-195-189-237:~$ mysql -h mysql.************.us-east-1.rds.amazonaws.com

ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql.************.us-east-1.rds.amazonaws.com' (110)

I can connect to RDS instance fine from my local machine using the same line as above. I tried various forum solutions but those don't help.

8
Are the ports open on the security group for the ec2 box to allow the connection to the rds?david99world
Yes, I added the security group of EC2 instance (sg-**) to the RDS security group.Mark Shust at M.academy
If I do a traceroute from EC2 to the RDS hostname, it times out: traceroute mysql.****.us-east-1.rds.amazonaws.com traceroute to mysql.****.us-east-1.rds.amazonaws.com (10.206.**.***), 30 hops max, 60 byte packets^C 1 * * * 2 * * * 3 * * * 4 * * * 5 * * * This leads me to believe it's a DNS/firewall issue. However, I have a rule for the RDS security group to allow connections from this EC2 security group.Mark Shust at M.academy
I also added the CIDR 0.0.0.0/0 to the RDS security group to confirm all can access that mysql database. Still not a go. This is telling me DNS is not routing right to the correct RDS hostname from my EC2 instance. Anyone else have this issue?Mark Shust at M.academy
So I did a 'dig' on the hostname from outside of AWS and inside the EC2 instance. It appears to be correctly routing to their internal IP from the EC2 instance, but the external IP from outside of it. Is there any diagnose-flag I can add to the mysql connection string to see what is actually happening or some other detailed logging? This is very frustrating.Mark Shust at M.academy

8 Answers

103
votes

I had similar problem, when I spun a new EC2 instance, but didn't change setting in RDS security group of inbound IP address allowed to connect to port 3306 of my RDS instance.

The confusing bit was an option in RDS dashboard, called Security Groups. You don't need it to solve the problem.

What you really need is:

  • Go to list of RDS instances
  • Click on the instance you are trying to connect
  • Click Security group rules section

enter image description here

This should open a new browser tab or window with details of security group. Locate several tabs in bottom part, select Inbound rules tab and click Edit button.

enter image description here

Change value to the IP address of your EC2 instance or IPv4 CIDR blocks, e.g.

174.33.0.0/16

To get this value, you can either ssh into your instance and run ip addr or run EC2 Manager in browser and locate value of Private IPs in your instance details.

40
votes

Additional information for people who might run into similar issues trying to connect to RDS or RedShift:

1) Check security groups

Verify the security group for the RDS instance allows access from the security group your source server belongs to (or its IP added directly if external to AWS). The security group you should be looking at is the one specified in the RDS instance attributes from the RDS console UI (named "security group").

NOTE: Database security groups might be different from AWS EC2 security groups. If your RDS instance is in classic/public EC2, you should check in the "database security group" section of the RDS UI. For VPC users, the security group will be an normal VPC security group (the name sg-xxx will be listed in the RDS instance's attributes).

2) Confirm DNS isn't an issue.

Amazon uses split DNS, so a DNS lookup external to AWS will return the public IP while a lookup internal to AWS will return a private IP. If you suspect it is a DNS issue, have you confirmed different IPs are returned from different availability zones? If different AZs get different IPs, you will need to contact AWS support.

3) Confirm network connectivity by establishing a socket connection.

Tools like tracepath and traceroute likely won't help since RDS currently drops ICMP traffic.

Test port connectivity by trying to establish a socket connection to the RDS instance on port 3306 (mysql, or 5432 for postgres). Start by finding the IP of the RDS instance and using either telnet or nc:

telnet x.x.x.x 3306
nc -vz x.x.x.x 3306

a) If your connection attempt isn't successful and immediately fails, the port is likely blocked or the remote host isn't running a service on that port. you may need to engage AWS support to troubleshoot further. If connecting from outside of AWS, try to connect from another instance inside AWS first (as your firewall might be blocking those connections).

b) If your connection isn't successful and you get a timeout, packets are probably being dropped/ignored by a firewall or packets are returning on a different network path. You can confirm this by running netstat -an | grep SYN (from a different CLI window/session while running and waiting for the telnet/nc command to timeout). Connections in the SYN state mean that you've sent a connection request, but haven't received anything back (SYN_ACK or reject/block). Usually this means a firewall or security group is ignoring or dropping packets.

Check to make sure you're not using iptables or a NAT gateway between your host and the RDS instance. If you're in a VPC, also make sure you allow egress/outbound traffic from the source host.

c) If your socket connection test was successful, but you can't connect with a mysql client (CLI, workbench, app, etc.), take a look at the output of netstat to see what state the connection is in (replace x.x.x.x with the actual IP address of the RDS instance):

netstat -an | grep x.x.x.x

If you were getting a connection established when using telnet or NC, but you see the 'SYN' state when using a mysql client, you might be running into an MTU issue.

RDS, at the time this is written, may not support ICMP packets used for PMTUD (https://en.wikipedia.org/wiki/Path_MTU_Discovery#Problems_with_PMTUD). This can be a problem if you're trying to access RDS or RedShift that's in a VPC from a classic ec2 instance via ClassicLink. Try lowering the MTU with the following, then testing again:

sudo ip link show
# take note of the current MTU (likely 1500 or 9001)
sudo ip link set dev eth0 mtu 1400

If the lower MTU worked, be sure to follow up with AWS customer support for help and mention that you are seeing an MTU issue while trying to connect to your RDS instance. This can happen if TCP packets are wrapped with encapsulation for tunneling, resulting in a lower usable MTU for packet data / payload. Lowering the MTU on the source server allows the wrapped packets to still fit under the limit.

If it didn't work, set your MTU back to it's default and engage AWS support for further troubleshooting.

12
votes

While Mark's problem seemed to have something to do with multi-AZ routing & EC2 classic, I ran into this exact same problem today.

To fix it, I modified the Security Group that was created automatically with my RDS instance by adding the two private IP addresses from my EC2 instance.

Adding inbound rules to my RDS instance

This was a fairly obvious problem, but I'm new to AWS in general, so hopefully this is useful for others like myself.

5
votes

Apparently, multi-AZ screws everything up. Since the default multi-AZ config placed my database in region us-east-1d, and my EC2 instance was in region us-east-1a, the DNS was not routing correctly. I re-created the RDS instance as non-multi-AZ, and made it live in us-east-1a, and all is happy.

If there are any super geniuses out there in regards to DNS routing on AWS with RDS, ELB, and multi-AZ capabilities, it would be pretty awesome to know how to do this, since this isn't documented anywhere in Amazon Web Service's documentation.

2
votes

I had the similar problem today when my EC2 instance suddenly lost access to RDS instance and Wordpress stopped working. The security groups were correct and I could even connect to MySQL from console on EC2 instance but not from PHP. For some reason restarting EC2 server helped me.

1
votes

Looks like sometime between the last posting and this posting, Amazon fixed the DNS routing issue, because everything works fine now for multi-AZ rds...

1
votes

After struggling for 3 days I finally found why mine was not connecting ...

Add outbound rule on your EC2 instance for port 3306 and inbound rules on your RDS server on port 3306. The inbound value should be the security of the EC2 instance

Example:

  • Your EC2 security group is - sg.ec2
  • And RDS security group is - sg.rds

So go to edit outbound rules of sg.ec2 and add Custom TCP at port 3306 and destination to 0.0.0.0/0

Then, go to edit inbound rule of sg.rds and add an inbound rule at port 3306 and source as sg.ec2

-3
votes

According to the Amazon doc, we should use:

PROMPT> mysql -h <endpoint> -P 3306 -u <mymasteruser> -p

where endpoint and mymasteruser(username) are from your RDS instance.

I solved that problem using IP public adress directly(of the endpoint) instead of the endpoint(****.us-east-1.rds.amazonaws.com).You can get the ip public address using "ping" command(ping ****.us-east-1.rds.amazonaws.com)