spark-submit in Amazon EC2

Question

I've a linux instance in Amazon EC2 instance. I manually installed Spark in this instance and it's working fine. Next I wanted to set up a spark cluster in Amazon. I ran the following command in ec2 folder:

spark-ec2 -k mykey -i mykey.pem -s 1 -t t2.micro launch mycluster

which successfully launched a master and a worker node. I can ssh into the master node using ssh -i mykey.pem ec2-user@master

I've also exported the keys: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

I've a jar file (which has a simple Spark program) which I tried to submit to the master:

 spark-submit --master spark://<master-ip>:7077 --deploy-mode cluster --class com.mycompany.SimpleApp ./spark.jar

But I get the following error:

 Error connecting to master (akka.tcp://sparkMaster@<master>:7077).
 Cause was: akka.remote.InvalidAssociation: Invalid address:    akka.tcp://sparkMaster@<master>:7077
 No master is available, exiting.

I'm also updated EC2 security settings for master to accept all inbound traffic: Type: All traffic, Protocol: All, Port Range: All, Source: 0.0.0.0/0

Micro instances might be too small for spark. I tend to prefer the c3/c4 instances, and if hefty CPU is needed for a task, the bigger ones like c3.8xlarge reduce the need for networking and are affordable for an hour or two of termporary use using the "spot pricing" with below-retail bids. — Paul

Paul Paul · Accepted Answer · 2015-09-18T20:01:52

A common beginner mistake is to assume Spark communication follows a program to master and master to workers hierarchy whereas currently it does not.

When you run spark-submit your program attaches to a driver running locally, which communicates with the master to get an allocation of workers. The driver then communicates with the workers. You can see this kind of communications between driver (not master) and workers in a number of diagrams in this slide presentation on Spark at Stanford

It is important that the computer running spark-submit be able to communicate with all of the workers, and not simply the master. While you can start an additional EC2 instance in a security zone allowing access to the master and workers or alter the security zone to include your home PC, perhaps the easiest way is to simply log on to the master and run spark-submit, pyspark or spark-shell from the master node.

spark-submit in Amazon EC2

1 Answers