Using spark cluster remotely

1
votes

I am trying to use a spark cluster remotely, so basically I have a 3 nodes, 1 master and 2 workers 'far' from my laptop, but I want to use them to make computation. I can connect easily to the driver typing: ./spark-shell --master spark://xxx.xxx.xxx.xxx:7077

And I see my application on spark web interface on the driver, but actions are not executed, I think because workers which should connect to me have networks problems. So I set on spark-env.sh my public IP: SPARK_LOCAL_IP=XX.XX.15.215

but doing that I get:

16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.

16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 16/02/11 16:34:04 ERROR SparkContext: Error initializing SparkContext.

1
apache-spark
What do you see on your spark master web UI? Is it waiting for resources? Do you see the workers connected to the master? Can you post a screenshot of the master web UI? – Saket
The warning you have posted should not cause a problem for you as it will attempt to bind to the next port. – Saket
Saket as you can see it stops trying at some point – Enrico D' Urso

1 Answers

0
votes

It is quite important for the DNS entries to be correct and for the workers and the master to be able to connect to the driver. If they are not directly reachable using the hostname, you can specify the spark.driver.host and spark.driver.port as part of the spark submit command.

Refer to this for a similar problem I had.