Launch spark master windows7

2
votes

Using win7-64, jdk8, sparks1.6.2. I have spark running, winutils, HADOOP_HOME, etc

Per documentation Note: The launch scripts do not currently support Windows. To run a Spark cluster on Windows, start the master and workers by hand. But does not say how?

How do I launch spark master on windows?

Tried running sh start-master.sh thru git bash : failed to launch org.apache.spark.deploy.master.Master: Even though it prints out Master --ip Sam-Toshiba --port 7077 --webui-port 8080 - So I don't know what all this means. But when I try spark-submit --class " " --master spark://Sam-Toshiba:7077 target/ .jar - I get errors: WARN AbstractLifeCycle: FAILED [email protected]: 4040: java.net.BindException: Address already in use: bind java.net.BindException: Address already in use

WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
17/01/12 14:44:29 WARN AppClient$ClientEndpoint: Failed to connect to master Sam-Toshiba:7077
java.io.IOException: Failed to connect to Sam-Toshiba/192.168.137.1:7077
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)

Also tried spark://localhost:7077 - same errors

1
javaapache-sparkwindows-7-x64

1 Answers

5
votes

On Windows you can launch Master using below command. Open command prompt and go to Spark bin folder and execute

spark-class.cmd org.apache.spark.deploy.master.Master

Above command will print like Master: Starting Spark master at spark://192.168.99.1:7077 in console as per IP of your machine. You can check the UI at http://192.168.99.1:8080/

If you want to launch worker once your master is up you can use below command. This will use all the available cores of your machine.

spark-class.cmd org.apache.spark.deploy.worker.Worker spark://192.168.99.1:7077

If you want to utilize 2 cores of your 4 cores of machine then use

spark-class.cmd org.apache.spark.deploy.worker.Worker -c 2 spark://192.168.99.1:7077