Flume KafkaChannel without KafkaSink?

0
votes

I'm trying to write from Flume source to Kafka topic. There is kafka channel in flume and in this Cloudera post author says that kafka channel may be used

To write to Kafka directly from Flume sources without additional buffering.

But when I'm trying to exclude sink from my configuration flume says

An error occurred while validating this configuration: Component tier1.sinks: Property value missing.

Do I really need to write to kafka channel and read back to write again to the kafka sink? That seems strange for me...

1
apache-kafkaflumeflume-ng

1 Answers

1
votes

No, you don't need to do that, please show me your config file.

A sample in Flume 1.7 goes like this:

source config etc...

agent1.channels.channel_sample.type = org.apache.flume.channel.kafka.KafkaChannel
agent1.channels.channel_sample.kafka.bootstrap.servers = hostnameorip:9092,hostnameorip:9092
agent1.channels.channel_sample.kafka.topic = topic_sample
agent1.channels.channel_sample.kafka.consumer.group.id = consumer_group_sample

If you don't need any sink binding to this channel_sample, kafka.consumer.group.id is not essential.

See https://flume.apache.org/FlumeUserGuide.html#kafka-channel for more

Take care there is a mistake in the document that default value of kafka.consumer.auto.offset.reset is earliest not latest