kafka consumer on single node multi broker configuration

Question

Did somebody worked on kafka python with single node multi broker setup?

I was able to produce and consume the data with single node single broker settings but when I have changed that to single node multi broker data was produced and was stored in the topics but when I run the consumer code data was not consumed.

Any suggestions on the above would be appreciated. Thanks in advance!

Note: all the configurations like producer , consumer and server properties were verified and are fine.

Producer code:

from kafka.producer import KafkaProducer

def producer():
    data = {'desc' : 'testing', 'data' : 'testing single node multi broker'}
    topic = 'INTERNAL' 
    producer = KafkaProducer(value_serializer=lambda v:json.dumps(v).encode('utf-8'), bootstrap_servers=["localhost:9092","localhost:9093","localhost:9094"])

    producer.send(topic, data)

    producer.flush()

Consumer code:

from kafka.consumer import KafkaConsumer

def consumer():  
    topic = 'INTERNAL'
    consumer = KafkaConsumer(topic,bootstrap_servers=["localhost:9092","localhost:9093","localhost:9094"])


    for data in consumer:
        print data

Server 1 config: I have added two more server files like this with same parameters for other brokers with the difference in broker.id, log.dirs values.

broker.id=1
port=9092
num.network.threads=3
log.dirs=/tmp/kafka-logs-1
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
delete.topic.enable=true

Producer config:

metadata.broker.list=localhost:9092,localhost:9093,localhost:9094

Consumer config:

zookeeper.connect=127.0.0.1:2181
zookeeper.connection.timeout.ms=6000

Could you please post your broker config and your python script? Otherwise we can't help you. — shizhz
I re-formatted your config info, could you please have a check, just in case I made mistakes — shizhz
I have added two more server files like this with same parameters for other brokers with the difference in broker.id, log.dirs values, did you change port? — shizhz

ImbaBalboa ImbaBalboa · Accepted Answer · 2017-04-03T14:28:35

Do you receive the messages with a simple Kafka consumer ?

bin/kafka-console-consumer.sh –bootstrap-server localhost:9092,localhost:9093,localhost:9094 –topic INTERNAL –from-beginning

Or with this one :

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic INTERNAL

If you get the messages with the second command, try to delete /tmp/log.dir of your brokers and log files in /tmp/zookeepker/version-2/. Then restart zookeeper and your brokers and create your topic again.

kafka consumer on single node multi broker configuration

1 Answers