Flink savepoint with local execution environment (like standalone application)

0
votes

How can I implement flink savepoint with standalone application (local execution env or mini cluster). I configured savepoint directory in flink-config.yaml file but not sure how to take the savepoint before shutdown the application and how to restore with restart the application?

is there any way or have to use flink cluster and then use the CLI.

Appreciate your help. thanks

1
apache-flinkflink-streamingflink-sqlflink-batch

1 Answers

1
votes

You can use either the CLI or the REST API to trigger savepoints.

https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html#savepoints

For example, to trigger a savepoint while leaving the job running:

./bin/flink savepoint <jobId> [savepointDirectory]

or to take a savepoint while stopping the job:

./bin/flink stop [-p targetDirectory] [-d] <jobID>

To restore the state from a savepoint during a restart:

./bin/flink run -s <savepointPath> ...

For a tutorial on this and related topics, see https://ci.apache.org/projects/flink/flink-docs-stable/try-flink/flink-operations-playground.html#upgrading--rescaling-a-job.

The REST API is documented here: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html. For example, you can take a savepoint via

curl -X POST localhost:8001/jobs/:jobid/savepoints -d '{"cancel-job": false}'

If you want to use REST API to trigger savepoints without running a cluster, you can do this in your job to start a local cluster (in a single JVM) with the WebUI and REST API:

Configuration conf = new Configuration();
conf.setString("state.savepoints.dir", "file:///tmp/savepoints");
StreamExecutionEnvironment env =
  StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);

These are the only ways to do this with open source Flink. There are commercial products (such as the free community edition of Ververica Platform) that make this easier.