How to feed an Apache Flink DataStream

1
votes

I am new to Apache Flink. I wanna create a DataStream and feed it with values from another system.

I found examples how to add a "SourceFunctions", in that function I have to wait for values from a source and publish those values to Flink by calling ctx.collect and then wait again, it's polling.

But I have a data source which calls a function when values arrive (async). So, what I wanna do is: when this async call happens I wanna put the value to a Flink DataStream, Pseudo-Code:

mysystem.connect_to_values( (value) => { myflinkdatastream.put(value.toString) })

Can this be done? otherwise i have to execute my connect and callback in the SourceFunction and do a loop with sleep afterwards, but I don't wanna do it in this way...

I already have seen "Asynchronous I/O for External Data Access" in Flink, but for that I still need a source stream, which is feed with a SourceFunctions (Poll/Loop).

1
apache-flinkflink-streaming

1 Answers

0
votes

If you do not want to add a SourceFunction in your streaming job, I suggest using Kafka or other message queue, to which you can send the data from async source, and connect Flink Streaming Job to the message queue.