Flink: Reading from Kinesis causes ReadProvisionedThroughputExceeded

0
votes

I've got a Flink app with a Kinesis source. I'm seeing a lot of ReadProvisionedThroughputExceeded errors from AWS when running the Flink app. I've tried updating the consumer config with different settings to reduce the number of get record calls and increase time between calls but that doesn't seem to help:

    consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "500")
    consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "30000")

    consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_BACKOFF_BASE, "3000")
    consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_BACKOFF_MAX, "10000")

Are there other setting that I should be tuning? Thanks!

2
apache-flinkamazon-kinesis

2 Answers

0
votes

Try to do the following checks:

  • Check Kinesis monitor to get which metric exceeds limit: number of records or sum of bytes for all records for each poll.
  • Competitive consumers. Are there any other consumers reading from the same shard? If enhanced-fan-out is not enabled, then they will share the same throughputs.

Enable enhance-fan-out(https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kinesis/#using-enhanced-fan-out) is a potential solution.

0
votes

Turns out it's a bug: https://issues.apache.org/jira/browse/FLINK-21661 in the version of I'm using - Flink 1.12