Flink: Is the number of Managers related to the number of nodeManagers on yarn

Question

I use the 'Run a single Flink job on YARN' mode to run a flink job on yarn,the command is just like this: ./bin/flink run -m yarn-cluster -yn 5 ./examples/batch/WordCount.jar

The problem is:

1. No matter how I modify parameters： '-yn' ( -yn 2/-yn 3/-yn 4 ....), the yarn cluster only creates 1 TaskManager and 1 JobManager for my job. My yarn cluster has only 3 nodes(one resourceManager,two nodeManager) and I think flink's yarn client may find there is only two nodeManager so the client only creates two manager ,one is jobmanager and one is taskmanager.

2. My flink version is 1.6.

Is the number of Managers related to the number of nodeManagers on yarn?

Answer 1

With -yn you specify the number of YARN containers to use. Every YARN container will have a Flink TaskManager. So the number of TaskManagers is not related to the number of YARN NodeManagers.

Are you sure there is enough memory and vCores available to start a total of 6 containers (1 JobManager and 5 TaskManagers)? If so, what does the logs say about the number of TaskManagers, slots and parallelism while starting the job?

Answer 2

I think I have found the reason.'Run a single Flink job on YARN' mode is just like the 'yarn mode' that flink create a jobManager first and then submit the job to the jobmanager.

In this way,we have two important params: -p(parallelism) and -ys(slots).If p is 6 and slots is 2,then the cluster will create 3(6/2) taskManagers,the param -yn is not effective. The default value of parallelism is 1,so no matter what yn param I set,the cluster only create one taskManager for me.