How to expose kafka endpoint from azure and consume it from onpremise application?

1
votes

I am trying to do a POC for the following :

I have hosted a container of kafka on azure. I am able to send messages to this kafka instance from a consumer, when the consumer is within azure.

Kafka configuration in docker-compose:

kafka:
     image: confluentinc/cp-kafka:latest
     depends_on:
       - zookeeper
     ports:
     - "9092:9092"
     environment:
       KAFKA_ADVERTISED_HOST_NAME: kafka
       KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://kafka:9092
       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
       KAFKA_BROKER_ID: 1
       KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
       KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
       KAFKA_HEAP_OPTS: "-Xmx256M -Xms128M"
       KAFKA_LOG_DIRS: /var/lib/winkafka

Producer Application Configuration on azure to talk to kafka:



    "Endpoint": "kafka:9092"

The communication between kafka and consumer on within azure is happening perfectly.

Now, I want to have consumer on-premises, rather than on azure. So what I have done is just expose the port 9092 from azure vm and also updated endpoint to the following:

Producer Application Configuration on local to talk to kafka on azure:

"Endpoint": "Azure Vm's Public IP:9092"

But, it is not able to connect from my local application To Kafka on azure.

Can you please suggest What I am doing wrong here and how I will be able to connect from local application to kafka running in a container on azure (or any cloud)?

1
azureapache-kafkacloudazure-cloud-services
Are you able to telnet to to <VM public IP>:9092 from where the local app is hosted? - Peter Wishart
Please read rmoff.net/2018/08/02/kafka-listeners-explained (replace AWS with Azure) ... @PeterWishart That is the first step, but not enough for it to work - OneCricketeer
I am able to get it to work. Two steps needed to be done: 1. Need to open up the kafka port in firewall too on azure vm. 2. Need to update listerner in docker-compose.yml file to use public ip. - user2142938

1 Answers

1
votes

You need to configure your Kafka listener for the separate networks (internal Azure vs public IP). See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for a detailed explanation.

Disclaimer: I wrote the blog post :)