Questions about Flume Kafka source

0
votes

When I use Flume Agent get data from Kafka source, and write data to HDFS, but some topics do not work well. The version of Kafka is kafka_2.11-0.8.2.1 . So, I run the command bin/kafka-topics.sh --zookeeper zk:port/kafka --describe, the result is

Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact

Topic: __consumer_offsets Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Topic: __consumer_offsets Partition: 1 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2

...

Topic: __consumer_offsets Partition: 48 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Topic: __consumer_offsets Partition: 49 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2

Topic:uniformAuthTopic PartitionCount:3 ReplicationFactor:3 Configs: Topic: uniformAuthTopic Partition: 0 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1 Topic: uniformAuthTopic Partition: 1 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Topic: uniformAuthTopic Partition: 2 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3

What's mean of Topic:__consumer_offsets? Well, I run the command bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper zk:port/kafka --group flume --topic uniformAuthTopic

Group Topic Pid Offset
logSize Lag Owner flume uniformAuthTopic 0 14737892 20069753 5331861
flume_bj-cuadc-xhm-4f03-p7-hs22-client_a-01-1457010084974-e9d501b9-0 flume uniformAuthTopic 1 14925444
20584430 5658986
flume_bj-cuadc-xhm-4f03-p7-hs22-client_a-01-1457010084974-e9d501b9-0 flume uniformAuthTopic 2 21086401
21086988 587
flume_bj-yz302-f11-r720-8-1457492175405-54ad44d1-0

pid 0 and pid 1's owner is wrong, I do not know why the owner changed. Please forgive my broken English!

1
apache-kafkaflume

1 Answers

0
votes

_consumer_offsets is a special topic used to track consumer offset commits. These used to be kept in Zookeeper, but are now kept this topic. According to the docs:

When the offset manager receives an OffsetCommitRequest, it appends the request to a special compacted Kafka topic named __consumer_offsets.