How to configure Flume with Kafka channel without source?

0
votes

It complains if a source is not specified in the configuration. According to doc :

The Kafka channel can be used for multiple scenarios:

  1. With Flume source and sink - it provides a reliable and highly available channel for events

  2. With Flume source and interceptor but no sink - it allows writing Flume events into a Kafka topic, for use by other apps

  3. With Flume sink, but no source - it is a low-latency,fault tolerant way to send events from Kafka to Flume sinks such asHDFS, HBase or Solr

https://flume.apache.org/FlumeUserGuide.html

I'm interested in the scenario 3 however there is no example for that in official flume doc.

Regards

2
apache-kafkachannelflume
Why not just use Kafka Connect and the HDFS connector? - Robin Moffatt

2 Answers

0
votes

flume agent source can be omitted in flume config on newer versions of CDH (5.14 in my case). Only a warning is issued.

0
votes

you can provide some dummy name for the source like:

agent.sources = dummySource
agent.sinks = hdfsSink
agent.channels = kafkaChnl

and just provide the configurations for hdfsSink and kafkaChnl