Using flume to import data from kafka topic to hdfs folder

1
votes

I am using flume to load messages from kafka topic HDFS folder. So,

  1. I created a topic TT
  2. I sent messages to TT with a kafka console producer
  3. I configured the flume agent FF
  4. Run the flume agent flume-ng agent -n FF -c conf -f flume.conf - Dflume.root.logger=INFO,console

The Code Execution Stops, without Error and it doesnot write anything to HDFS. The log file contains this warning No broker partitions consumed by consumer thread flume_-0 for topic.

Any help is greatly appreciated. Thanks in advance.:)

1
apache-kafkahdfsflumeflume-ng
Try to use Flafka [ blog.cloudera.com/blog/2014/11/… ], you can find an example in cloudera site with explanation - G.Saleh
You're better off using the HDFS connector for Kafka Connect. It is open source and available standalone or as part of Confluent Platform. - Robin Moffatt
Flume is on its last legs; I agree with @RobinMoffatt that Kafka Connect is the standard way to solve this problem now. If you want to proceed with this, it would be useful to see your flume.conf. - Ben Watson

1 Answers

0
votes

this conf fixed the problem

  • FF.sources=kafka-source
  • FF.channels = kafka-channel
  • FF.sinks =hdfs-sink FF.channels.kafka-channel.type org.apache.flume.channel.kafka.KafkaChannel
  • FF.channels.kafka-channel.kafka.bootstrap.servers = wided-Latitude:9092
  • FF.channels.kafka-channel.kafka.topic = TT
  • FF.channels.kafka-channel.parseAsFlumeEvent = false
  • FF.sinks.hdfs-sink.hdfs.writeFormat = Text
  • FF.sinks.hdfs-sink.channel = kafka-channel
  • FF.sinks.hdfs-sink.type = hdfs
  • FF.sinks.hdfs-sink.hdfs.path = hdfs://***:8020/spark/