Unable to write streamed data to sink file using Spring cloud dataflow

0
votes

I am trying to create data flow pipeline using spring cloud data flow using shell(Not UI). Source being twitterstream and sink as File. Here is what i did to configure file-sink :

dataflow:>stream create demo --definition "twitterstream --credentials | file --dir=/opt/datastream --mode=APPEND --filename=tweets.txt"

I can consume data from kafka topic but unable to write on above sink location, file is not even created . NO error log while deploying the stream. Eventually i will change it to HDFS from local file system. Is there anything missing ?

PS: I tried default file-sink (without definition), which is supposed to create default file inside /tmp/xd/output, didn't happen either.

1
springhadoopspring-cloudspring-cloud-dataflow

1 Answers

2
votes

On the latest 1.0.0.RELEASE (GA) release, the following stream definition works.

dataflow:>stream create demo --definition "twitterstream | file --directory=/someFolder --mode=APPEND --name=demo.txt"

A couple of things to point out:

1) The twitterstream source does not support --credentials as an OOTB property. See here.

2) The file sink does not support --filename as an OOTB property; you'd have to use --name instead. See here.