Flink TaskManagers do not start until job is submitted in YARN cluster

3
votes

I am using Amazon EMR to run Flink Cluster on YARN. My setup consists of m4.large instances for 1 master and 2 core nodes. I have started the Flink CLuster on YARN with the command: flink-yarn-session -n 2 -d -tm 4096 -s 4.

Flink Job Manager and Application Manager starts but there are no Task Managers running. The Flink Web interface shows 0 for task managers, task slots and slots available. However when I submit a job to flink cluster, then Task Managers get allocated and the job runs and the Web UI shows correct values as expected and goes back to 0 once the job is complete.

I would like Task Managers to be running even when no Job is submitted as I want to use Flink's REST API to monitor and modify parallelism based on the available slots value.

Is there a configuration that I've missed which prevents Task Managers from running all the time?

1
yarnapache-flinkamazon-emr

1 Answers

4
votes

Flink's behavior was changed in FLIP-6 to allow dynamic resource allocation. From Flink version 1.5.0, one can put mode: legacy in flink-conf.yaml to run Flink in the old mode where Task Managers will be started along with Job Manager.