Too many empty chk-* directories with Flink checkpointing using RocksDb as state backend

Question

Too many empty chk-* files exist in the location where I have setup Rocksdb as state backend

I am using FlinkKafkaConsumer to get data from Kafka topic. And I am using RocksDb as state backend. I am just printing the messages received from Kafka. Following are the properties I have to set up the state backend:

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(100);
        env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
        env.getCheckpointConfig().setMinPauseBetweenCheckpoints(50);
        env.getCheckpointConfig().setCheckpointTimeout(60);
        env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
        env.getCheckpointConfig().enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
StateBackend rdb = new RocksDBStateBackend("file:///Users/user/Documents/telemetry/flinkbackends10", true);
env.setStateBackend(rdb);
env.execute("Flink kafka");

In flink-conf.yaml I have also set this property:

state.checkpoints.num-retained: 3

I am using simple 1 node flink cluster(using ./start-cluster.sh) .I started the job and kept it running for 1 hour and I see too many chk-* files created under /Users/user/Documents/telemetry/flinkbackends10 location

chk-10      chk-12667   chk-18263   chk-20998   chk-25790   chk-26348   chk-26408   chk-3       chk-3333    chk-38650   chk-4588    chk-8       chk-96
chk-10397   chk-13      chk-18472   chk-21754   chk-25861   chk-26351   chk-26409   chk-30592   chk-34872   chk-39405   chk-5       chk-8127    chk-97
chk-10649   chk-13172   chk-18479   chk-22259   chk-26216   chk-26357   chk-26411   chk-31097   chk-35123   chk-39656   chk-5093    chk-8379    chk-98
chk-1087    chk-14183   chk-18548   chk-22512   chk-26307   chk-26360   chk-27055   chk-31601   chk-35627   chk-4       chk-5348    chk-8883    chk-9892
chk-10902   chk-15444   chk-18576   chk-22764   chk-26315   chk-26377   chk-28064   chk-31853   chk-36382   chk-40412   chk-5687    chk-9       chk-99
chk-11153   chk-15696   chk-18978   chk-23016   chk-26317   chk-26380   chk-28491   chk-32356   chk-36885   chk-41168   chk-6       chk-9135    shared
chk-11658   chk-16201   chk-19736   chk-23521   chk-26320   chk-26396   chk-28571   chk-32607   chk-37389   chk-41666   chk-6611    chk-9388    taskowned
chk-11910   chk-17210   chk-2       chk-24277   chk-26325   chk-26405   chk-29076   chk-32859   chk-37642   chk-41667   chk-7       chk-94
chk-12162   chk-17462   chk-20746   chk-25538   chk-26337   chk-26407   chk-29581   chk-33111   chk-38398   chk-41668   chk-7116    chk-95

out of which only chk-41668, chk-41667, chk-41666 have data. The rest of the directories are empty.

Is this expected behavior. How to delete those empty directories? Is there some configuration for deleting empty directories?

codeit codeit · Accepted Answer · 2019-08-29T23:08:57

Answering my own question here:

In UI I was seeing 'checkpoint expired before completing' error in the checkpointing section. And found out that to resolve the error we need to increase the checkpoint timeout.

I increased the timeout from 60 to 500 and it started deleting the empty chk-* files.

env.getCheckpointConfig().setCheckpointTimeout(500);

Too many empty chk-* directories with Flink checkpointing using RocksDb as state backend

1 Answers