Flink async IO operator chaining with with another sync operator

0
votes

I have a usecase where I am using async IO operators with normal mappers in flink. I am using flink 1.8. So, async operator is going to have to be at the head of the operator chain. So my operator flows looks like this:

Source -> Mapper1 -> AsyncOperator -> Mapper2 -> Sink

Because of the requirement of async operator being head, there are two operator chains and hence two tasks- 1. Source + Mapper1 2. AsyncOperator+Mapper2+Sink. I have question regarding the second chain. I think the second chain should be comprised within a single task if they are chained correctly. I am not sure if there is a wait time between async operator and mapper 2 on the task threads or the Mapper2 gets bound to the response handler for the Async Operator internally ? Ideally, it should be second, but I can't find any documentation for the same - hence wondering.

Reference:

  1. https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
2
apache-flinkflink-streaming

2 Answers

1
votes

The AsyncWaitOperator spins up an Emitter in a thread, so as soon as results are available they get sent to the operator's collector. Note though that if you specify ordered results there can be a "wait time" due to completion order not matching the order of incoming elements.

0
votes

BTW, the restriction that the AsyncWaitOperator must be at the head of a chain was removed in Flink 1.11. See FLINK-16219. The only remaining limitation was that it could not follow a SourceFunction. The AsyncWaitOperator can follow the new sources introduced in Flink 1.12.