Kafka source and HDFS sink in Spring cloud Data flow

0
votes

I am using Kafka as the source and I want to write the messages on Kafka to HDFS using HDFS sink.But I see the file getting created on the HDFS but the message on Kafka is not written to the HDFS file.Please find below the Stream DSL.

stream create --definition ":streaming > hdfs --spring.hadoop.fsUri=hdfs://127.0.0.1:50071 --hdfs.directory=/ws/output --hdfs.file-name=kafkastream --hdfs.file-extension=txt --spring.cloud.stream.bindings.input.consumer.headerMode=raw" --name mykafkastream

Please help me resolve this.

1
spring-cloud-dataflow

1 Answers

1
votes

It could be that the data isn't written to the hdfs disk yet. You can force a flush/sync while you are testing. Try setting --hdfs.enable-sync=true --hdfs.flush-timeout=10000 that way the data is written to hdfs every 10s whether the buffer is full or not.