What are the ways to access the flink state from outside the flink cluster?

Question

I am new to Apache flink and building a simple application where I am reading the events from a kinesis stream, say something like

TestEvent{
 String id,
 DateTime created_at,
 Long amount
} 

performing aggregation (sum) on field amount on above stream keyed by id. The transformation is equivalent to SQL select sum(amount) from testevents group by id where testevents are all the events received till now. The aggregated result is stored in a flink state and I want the result to be exposed via an API. Is there any way to do so?

PS: Can we store the flink state in dynamoDB and create an API there? or any other way to persist and expose the state to outside world?

Answer 1

I'd recommend to ignore state for now and rather look at sinks as the primary way for a stream application to output results.

If you are already using Kinesis for input, you could also use Kinesis to output the results from Flink. You can then use the Kinesis adapter for DynamoDB that is provided by AWS as further described on a related stackoverflow post.

Coming back to your original question: you can query Flinks state and ship a REST API together with your stream application, but that's a whole lot of work that is not needed to achieve your goal. You could also access checkpointed/savepointed state through the state API, but again that's quite a bit of manual work that can be saved by going the usual route outlined above.

Answer 2

This is Flink's documentation, which provides some use cases queryable_state

You can also use the API to read it offlineState Processor API