What is the master URL in EC2 spark cluster

Question

I have a spark cluster launched using spark-ec2 script. (EDIT: after login into the master), I can run spark jobs locally on the master node as :

spark-submit --class myApp --master local myApp.jar 

But I can't seem to run the job in the cluster mode:

../spark/bin/spark-submit --class myApp --master spark://54.111.111.111:7077 --deploy-mode cluster  myApp.jar 

The ip address of the master is obtained from the AWS console. I get the following errors:

WARN RestSubmissionClient: Unable to connect to server 
Warning: Master endpoint spark://54.111.111.111:7077 was not a REST server. Falling back to legacy submission gateway instead.
Error connecting to master (akka.tcp://[email protected]:7077).
Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://[email protected]:7077
No master is available, exiting.

How to submit to a EC2 spark cluster ?

Answer 1

When you run with --master local you are also not connecting to the master. You are executing Spark operations in the same JVM as the application. (See docs.)

Your application code may be wrong too. So first just try to run spark-shell on the master node. /root/spark/bin/spark-shell is configured to connect to the EC2 Spark master when started without flags. If that works, you can try spark-shell --master spark://ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com:7077 on your laptop. Be sure to use the external IP or hostname of the master machine.

If that works too, try running your application in client mode (without --deploy-mode cluster). Hopefully in the course of trying all these, you will figure out what was wrong with your original approach. Good luck!

Answer 2

This is nothing to do with EC2, I had similar error on my server. I was able to resolve it by overwriting spark-env.sh SPARK_MASTER_IP.