Improving Flink broadcast performance

0
votes

I've a pipeline where I'm applying transformation rules(from broadcast state) on a stream of events; when I run broadcast stream and original stream in parallel without connecting, stream performance is really good, but the moment I do broadcast performance goes down drastically. How can I achieve better performance. Data passed between operators are in byte[] and data footprint is small as well.

I've attached snapshots of both scenarios:

  1. Top row shows stream consuming events from Kafka and bottom row shows rules consumed from another topic. With this setup I could achieve throughput of upto ~20K msg/sec per task manager  processing 12Gb of data in 4mins

enter image description here 2. I've connected the broadcast stream with the data stream for processing in future . Note that only to measure performance of broadcast I've made sure no records are consumed in the data stream(top row). At the processing side of the broadcast state, i'm only store received messages to MapState. With this setup I can get throughput of upto ~1000 msg/sec per task manager processing 12Gb of data in 18mins. enter image description here

1
apache-flinkflink-streaming
Could you please clarify if you have a sink in the 2. case? (it's cropped so I cannot see) How many elements do you broadcast? - Arvid Heise
Yes I've sink which tracks how many elements in broadcast missed in original stream . In broadcast stream I'm sending about 24 million records. - cryptickp

1 Answers

1
votes

You've done more than simply connect the broadcast and keyed streams. Before, each event went through just one network shuffle (the rebalance, hash, and broadcast connections), and now there are four or five shuffles for each event.

Every shuffle is expensive. Try to reduce the number of times you change parallelism or use keyBy.