Connect Lambda to Redshift in Different Availability Zones

1
votes

Our Redshift cluster resides in Zone A.

When our Lambda function uses a Zone A subnet, it can connect to Redshift.

When our Lambda function uses a subnet other than Zone A, it times out.

The work around, where we ALLOW connections for Redshift on port 5439 from 0.0.0.0/0, is not desired.

  • We have our Lambda functions and Redshift cluster in the same VPC.
  • Lambda functions have 4 dedicated subnets (one per zone)
  • Redshift has 4 dedicated subnets per zone as well
  • Lambda functions have their own security group (SG)
  • The Redshift cluster has it's own SG as well.
  • Redshift SG ALLOWs port 5439 from Lambda SG and Admin SG
  • Enhanced VPC Routing is enabled
  • Cluster Subnet Groups include all 4 Redshift subnets (one per zone)
  • No issues when allowing port 5439 from 0.0.0.0/0 on Redshift SG
  • Flow logs for REJECT work fine from Zone A to Zone A, but not from other zones to Zone A when we disable 0.0.0.0/0 rule.
  • All Lambda subnets use a NAT that exists in Zone A
  • All Redshift subnets use an IGW that exists in
  • All Network ACLs currently allow all (default)
1
amazon-redshiftaws-lambdaamazon-vpc
Could you please clarify the current situations that work as expected, and those that do NOT work as expected? For example, does it work when Lambda & Redshift are both in the same AZ? Can you clarify what you mean by "but not from other zones when we disable..."? You say "No issues when allowing port 5439 from 0.0.0.0/0" -- does that mean everything is good in that situation, but you don't wish to open up Redshift so widely? Also, do you have any Network ACLs that block traffic on any of the subnets? - John Rotenstein
Updated the post. Network ACLs are default (allow all). Redshift being open to 0.0.0.0/0 is a work around we can't use. Lambda Zone A to Redshift Zone A works. Issue is that functions in zones other than Zone A cannot connect to Redshift Zone A. - Matt Beckman
How are the Lambda functions referring to Amazon Redshift? Presumably, via a DNS name -- but is that DNS name being resolved to an internal IP address or a Public IP address? You can test this by doing a nslookup from an EC2 instance within the VPC, to see what IP address is returned. Also, have you tried activating VPC Flow Logs to analyze the traffic between Lambda and Redshift? - John Rotenstein
Were you able to solve this issue? I am stuck with exactly the same issue. Any advice appreciated. Thanks. - n.nasa

1 Answers

1
votes

I was stuck in a similar situation. Adding the NAT gateway's elastic ip to the inbound rule of Redshift's security group for port 5439 fixed it for me.

Steps:

  • Check lambda's private subnet using a NAT gateway (subnet-abc)
  • Go to VPC console > subnets > subnet-abc > route-table
  • In Route-table routes, you can find the NAT gateway used (nat-abcdefg)
  • Go to VPC console > NAT Gateways > nat-abcdefg. Get the elastic-ip used by this NAT gateway. (xx.yy.zz.pqr)
  • Add an inbound rule for this elastic-ip in redshift's security group (port = 5439 CIDR xx.yy.zz.pqr/32)

Volla! Lambda connects to redshift.

Though, before doing this, lambda should be configured in the same VPC as redshift and using the appropriate private subnet (configured to use NAT gateway) as OP suggested.