How to unblock Kinesis Firehose CIDR/IP on Redshift Cluster in a VPC?

3
votes

We are trying to use AWS Kinesis Firehose with Redshift output. We have created the Firehose Delivery Stream. We have a Redshift Cluster in a VPC Security Group. But it seems like Firehose is not able to connect to the cluster.

This is a snippet where we create the VPC Security Group.

"RedshiftVPCSecurityGroup": {
  "Type": "AWS::EC2::SecurityGroup",
  "Properties": {
    "SecurityGroupIngress": [
      {
        "CidrIp": "52.19.239.192/27",
        "FromPort": "5439",
        "ToPort": "5439",
        "IpProtocol": "tcp"
      }
    ],
    "VpcId": {
      "Ref": "VpcId"
    },
    "GroupDescription": "Redshift VPC security group"
  }
}

Following this link, We set an Ingress rule that allows connections from "52.19.239.192/27". But this does not seem to have worked and we still get the following error.

The connection to the specified Amazon Redshift cluster failed. Ensure that security settings allow Firehose connections, that the cluster or database specified in the Amazon Redshift destination configuration or JDBC URL is correct, and that the cluster is available.

Does anyone know what are we doing wrong?

2
amazon-web-servicesamazon-redshiftamazon-kinesis-firehose
Is your cluster in a public subnet? - Matt Houser

2 Answers

8
votes

When FireHose accesses your Redshift cluster, it does so from outside of your VPC.

In order for FireHose to access your Redshift cluster, the following must be configured:

  1. The Redshift cluster must be in a public subnet, meaning it's in a subnet with an Internet Gateway.
  2. The Redshift cluster must have a public IP address.
  3. Your security group must allow incoming access to FireHose on port 5439.

Without the above two requirements met, nothing can access the Redshift cluster from outside your VPC.

Finally, to restrict outside access to only FireHose, you can limit the rule for port 5439 to the FireHose CIDR block:

  • 52.70.63.192/27 in us-east-1
3
votes

I have been struggling with this same issue and have just found the solution. Make sure that the cluster subnet group created for the Redshift cluster contains only public subnets. When launching a new Redshift cluster, AWS will randomly select ONE of the subnets assigned to the cluster subnet group. If that subnet is private, then Firehose cannot reach your Redshift cluster (it's always outside your VPC). Choosing the "Make publicly available" option on the Redshift cluster won't help, nor will whitelisting Firehose IP addresses in ACL or security groups, etc. You must make the subnet public or relaunch the cluster using only public subnets.