How can a lambda inside a private subnet access EC2 in a public subnet?

2
votes

I've created an EC2 instance inside a public subnet (so that I can access it from my home network) and I have created some Lambda's inside the private subnets of my VPC.

My 1st lambda can freely access the internet (through a NAT Gateway) and do its job. Which is to fetch a file from the internet and upload it to S3, once per day.

My 2nd lambda is supposed to retrieve this file from S3 (which it does without issue) read the file & then upload the data to MySQL running on the EC2 instance. It is unable to connect to the Database (using either the EC2's public or private IP's) and the Cloudwatch logs show that the session times out, making me think this is a networking issue.

I have a 3rd lambda that will also need to interact with the EC2/DB instance.

My security group allows for all incoming traffic from my home network IP, the NAT gateway & the VPC IP range. All outbound traffic is allowed.

I appreciate its not usual to have an EC2/DB set up this way, in a public subnet, but its my preference to interact it with it this way from home using Sequel Pro.

However, is my set up even possible? Eg can my private subnet lambdas interact with a public subnet ec2 instance? if so, does anybody have any ideas how I can make this happen?

1
amazon-web-servicesamazon-ec2aws-lambdaamazon-vpc
Can you verify that your mysql is listening to all network interfaces (not just localhost)? – jellycsc
Thanks for the quick reply. Yes I can. I can access from any IP address using Sequel Pro and my code executes perfectly when its run on my local machine – Chris Hurst
Ok, I see. The network problem seems to be on the lambda side. Could you also verify that your second lambda is able to access the internet? For example, visit google.com – jellycsc
Can you double check the second Lambda is definitely in the correct VPCs subnets? – Chris Williams
I would add to what jarmod said: The DB instance security group should allow inbound access from the Lambda's security group. Also the Lambda function needs to access MySQL via the EC2 server's private IP address. If this doesn't work then you need to check that your MySQL server is configured to allow connections from sources other than localhost, and make sure there isn't some firewall software on the EC2 server blocking connections also. – Mark B

1 Answers

4
votes

It appears that your situation is:

  • An Amazon EC2 instance running in a public subnet, with MySQL
    • The EC2 instance has a Security Group allowing all incoming traffic from your home network IP, the NAT gateway and the VPC IP range
  • An AWS Lambda function connected to a private subnet of the same VPC
  • A NAT Gateway allowing private subnets to connect to the Internet
  • The Lambda function is unable to connect with the MySQL database running on the EC2 instance

The normal security configuration for this scenario would be:

  • A Security Group on the Lambda function (Lambda-SG) that allows all Outbound access (no Inbound required)
  • A Security Group on the EC2 instance (EC2-SG) that allows inbound access from Lambda-SG on port 3306, plus whatever inbound permissions you want for accessing your instance via SSH, etc.

Given that your Security Group includes "the VPC IP range", this should be sufficient to permit inbound access.

The Lambda function should reference the EC2 instance via its private IP address to keep traffic within the VPC. By default, all subnets within a VPC can communicate with each other unless the Network ACLs have been modified (and they should generally be left at default values).

This means that the only remaining explanation would be that the MySQL database is not accepting traffic from the VPC IP range. (I'm not a MySQL person, but I know that PostgreSQL requires incoming IP ranges to be defined, so this might be true for MySQL too.)

To diagnose what might be happening in your network, I recommend:

  • Launch another Amazon EC2 instance in the public subnet
  • Connect to the new instance and try to connect to the MySQL database via the private IP address
  • If that works, repeat the process but from an EC2 instance in the private subnet. To use this you will need to connect to the 'public' EC2 instance, and from there connect to the 'private' EC2 instance. Then, try and connect to MySQL from that private instance.

These steps will progressively let you identify where the network problem might lie. Let us know what you find!