Link map state size and number of keys

Question

I was wondering if there is a way to retrieve the total state size stored in the state backend.

I am currently using flink 1.3, above emr and rocksdb backend, with asynchornous checkpointing and incremental checkpoints.

The flink dashboard under "checkpoints" displays the state size but I assume that due to the fact that I'm using incremental checkpoints I see in the checkpoint history page a fluctuation in state size.

The only current way I use and I'm not sure is a fit, is with "ls" command on the hdfs checkpoint location.

I assume there is a better way, and would appreciate your help

Answer 1

Currently one of the things that may help You with Your problem is Queryable State which allows You to basically query the state of the operator. But it is not available through the WEB-UI but you need to create a separate Queryable State Client for this. More info can be found here