Does Kafka support ELB in front of broker cluster?

9
votes

I had a question regarding Kafka broker clusters on AWS. Right now there is an AWS ELB sitting in front of the cluster, but when I set the "bootstrap.servers" property of my producer or consumer to the "A" record (and correct port number) of my ELB, both the producer and consumer fail to produce and consume messages respectively. I have turned off all SSL on my broker and am connecting through the PLAINTEXT 9092 port, with which my ELB forwards port 1234 to 9092. So in my Producer Configs properties for example, I will have...

bootstrap.servers = ("A" record of ELB):1234

More info:

  • My ELB's protocol is TCP/TCP
  • My "advertised.listeners" property on brokers are PLAINTEXT://(ec2-private-ip):9092

Has anyone had any luck running Kafka behind an ELB? If so, please help me out!

2
apache-kafka

2 Answers

12
votes

You can use an ELB as the bootstrap.servers, but the brokers still need to be directly accessible to the client. The ELB will be used for the initial metadata request the client makes to figure out which topic partitions are on which brokers, but after that it'll use the hostname of the server (or advertised.listeners setting if you need to customize it, which, e.g. might be necessary on EC2 instances to get the public IP of a server).

0
votes

IMHO, It is not suggested to add additional SLB to kafka cluster.

  • Kafka cluster are decentralized in essence, SLB add single point of failure risk and complicate the cluster setup.
  • Message send and receive are network lag sensitive, SLB will reduce the message throughput definitely.