The Flink official documentation provides a jobmanager high availability solution for Standalone And Yarn Flink clusters. But what should be done for high availability using a Kubernetes Flink Cluster?
From the Kubernetes Setup section of the document, it seems we just deploy a single Jobmanager when deploying to a Kubernetes cluster. So how to achieve HA for a Kubernetes Flink Cluster?
The official doc says that high availability for the job manager is to deal with cases where the job manager crashes. So there is only a single job manager needed but you want to handle the case where it goes down. On Kubernetes if it goes down then Kubernetes should detect this and automatically restart it. So you don't need to run more replicas of it.
(The doc says this explicitly about using yarn for ha. It doesn't seem to state it for Kubernetes but restarting failing Pods is standard behaviour for Kubernetes.)
The task manager is configured by default to run with multiple replicas on Kubernetes in the official k8s resources (see the 'replicas' entries in the resources) but the job manager is not. (And it's the same in the helm chart.) So I believe it is not needed for the job manager - I'd suggest running with a single job-manager unless you hit specific problems with that.
I agree with Ryan's answer regarding the fact that Kubernetes fulfils most of the requirements for HA deployments.
With Kubernetes, you can run a Flink job cluster with file-based HA, instead of using ZooKeeper or its equivalents.
You can find an example of how to set it up correctly (with or without HA) in this github repo.
One might also find useful those blogposts explaining about correctly deploying a flink job cluster on k8s and achieving file based high availability without zookeeper.
(Disclosure: I wrote the posts and set up the repo)
