I have created an AWS MSK cluster and have tested the set up by running a EC2 Producer/Consumer instance in the same VPC.I have used the same VPC and subnets to create an AWS lambda with a MSK cluster topic as an event source.All my subnets are private subnets.
Cluster set up works fine as I have verified the same from EC2 instance .The lambda always gets disabled with an error "Last Processing result: PROBLEM:Connection error.Please check your event source connection configuration.
I read in the below AWS blog that we need a NAT gateway running in a public subnet of each region in which we configured the MSK cluster to ensure connectivity .Is this something I need to try doing .I still fail to understand how a EC2 instance running in the same VPC and subnets is able to access the MSK cluster and not my Lambda .Could somebody help ?
https://aws.amazon.com/blogs/compute/using-amazon-msk-as-an-event-source-for-aws-lambda/
On a different note,I have multiple Kafka topics from which I need to consume data from and sink them to an ElasticSearch cluster.I have the option of running Kafka connect sink in an EC2 and I could also create an AWS lambda with MSK topic as event source.I know I could configure multiple MSK topics in an AWS Lambda.Could somebody advice which approach would be good ?
