How to run a Flink Job on a remote YARN cluster

Question

I have some issues to deploy a Flink job remotely through the scala API.

I have no problem with launching a Yarn session on my cluster and then run my job in command line with a jar.

What I want is to directly run my job with my IDE. How to do it in scala ?

val env = ExecutionEnvironment.createRemoteEnvironment("mymaster", 6123, "myjar-with-dependencies.jar")

This is not working, and I do realize that I am not declaring any YARN deployment with it.

Any help ?

Robert Metzger Robert Metzger · Accepted Answer · 2017-03-15T18:12:57

Flink does currently (March 2017, Flink 1.2) not allow to deploy on YARN programmatically through an ExecutionEnvironment.

You could look into Flink's internal, undocumented APIs for deploying it on YARN, and then submit through the remote env.