1
votes

Is there a Cookbook with precise description of what exactly one should change in what config file (job-manager-config/flink-conf.yaml, task-manager-config/flink-conf.yaml, job-manager-config/zoo.cfg, ...) in order to change Flink application deployment from session mode to job mode?

Besides, we have flink cluster setup on openshift via an ansible role that was delievered to us, but unfortunately is not taken from ansible galaxy repository, so I cannot reference you to any published setup. Probably, most of configuration changes has to be done there, outside of the mentioned config flink files.

What I don't understand is how is it controlled that submitting a new job spawns automatically a new pair of job manager and a task manager (1+1) instances per job, instead of just using the available task slots within one job manager and predefined number of task managers? We currently have one job manager and two task managers, each with 10 task slots (taskmanager.numberOfTaskSlots:10 line in task-manager-config/flink-conf.yaml), 20 task slots in total.

1

1 Answers

2
votes

You're right, your question is really outside the scope of what comes with Flink. It’s easy enough to set up a session cluster by hand, but if you want a new cluster for every job, that calls for some automation.

There are various solutions out there for managing Flink. You might want to investigate if something like Ververica Platform or flinkk8soperator would meet your needs.