Kafka Connect AWS Lambda Sink

4
votes

How does this connector plugin works? Kafka Connect AWS Lambda Sink

In docker-compose.yml file it is pulling docker images of zookeeper, kafka & kafka-connect from confluentinc repository.

Can we utilize this plugin only for connecting aws lambda as a kafka consumer with kafka cluster deployed separately on ec2 instances.

Can we consume this connector plugin without the use of kafka connect?

2
apache-kafkaaws-lambdaapache-kafka-connect
By the way, you don't necessarily need that connector. You can use the existing S3 connector and write a lambda within AWS that's triggered on an S3 file creation event – OneCricketeer

2 Answers

2
votes

Can we consume this connector plugin without the use of kafka connect?

No. Kafka Connect connectors run on Kafka Connect.

But, Kafka Connect is part of Apache Kafka, and gives you a runtime environment which handles things like node failure, scaleout, centralised config & management—so it's actually a good thing to use. From a user's point of view it's 'just' config files that you need to write to use it.

You can read more about Kafka Connect here

Disclaimer: I work for Confluent.

1
votes

You can use the plugin with a separately deployed Kafka cluster. Either deploy the connector jar to your Kafka Connect cluster or run the Kafka Connect container in Docker and deploy the lambda connector there. You just need to modify the docker-compose.yml configuration of Kafka Connect to point to you Kafka cluster. There’s a potentially better example in github.com/llofberg/kafka-connect-rest repository.

Disclaimer: I wrote the connector.