Spark application behind a NAT using YARN cluster mode

0
votes

In client deploy mode a Spark driver needs to be able to receive incoming TCP connections from Spark executors. However, if the Spark driver is behind a NAT, it cannot receive incoming connections. Will running the Spark driver in YARN cluster deploy mode overcome this limitation of being behind a NAT, because the Spark driver is then apparently executed on the Spark master?

1
apache-sparkyarn

1 Answers

1
votes

Will running the Spark driver in YARN cluster deploy mode overcome this limitation of being behind a NAT, because the Spark driver is then apparently executed on the Spark master?

Yes, it will. Another possible approach is to configure:

  • spark.driver.port
  • spark.driver.bindAddress

and create SSH tunnel to one of the nodes.