What open source solutions exist to move data from Kafka to HDFS3 using Kafka Connect?

0
votes

Other than the Confluent HDFS library (not open-source), is there any completely open-source library to move messages from Kafka (using Kafka Connect) to HDFS 3?

  • Kafka Connect HDFS 2 Sink - Confluent community licence
  • Kafka Connect HDFS 3 Sink - Confluent enterprise license

Related to: Need to move small JSON messages from Kafka to HDFS with Kakfa Connect but without using Confluent libs, if not completely free

1
hadoopapache-kafkahdfsapache-kafka-connect
Asking for external libraries is considered off topic - help center. - OneCricketeer

1 Answers

0
votes

The HDFS2 connector is open source and free to use with the Community License if you're not using it to offer it as a hosted service, or as I said before, Apache Nifi is a more rich workflow product that works well in the Hadoop ecosystem alongside Kafka. Spark or Flink are often used for this too