Can't start Apache Spark on Windows using Cygwin

15
votes

I am sorry if this is too trivial but I am very new to Spark. I built Apache Spark on Windows from code using Maven. I used Cygwin to build and it worked fine. Now I am unable to start it using the command (./start-master.sh) from the sbin directory.

I get the below error : failed to launch org.apache.spark.deploy.master.Master:

Error: Could not find or load main class org.apache.spark.launcher.Main
full log in /cygdrive/c/Spark/spark-1.5.1/sbin/../logs/spark-auser-org.apache.spark.deploy.master.Master-1.host

I understand it is looking for the spark-launcher_2.10-1.5.1.jar in the classpath but I am not able to identify in the scripts where is it defining the classpath or the lib directory.

1
apache-spark
Are you using JDK/JRE from cygwin? - kostya
@kostya : Spark would run inside the JVM. I am able to access the JRE from cygwin too. I am running Java 1.7. - Geek
Just an FYI, you can also start a Spark Cluster on Windows itself (e.g. spark-class.cmd org.apache.spark.deploy.master.Master ++ spark-class.cmd org.apache.spark.deploy.worker.Worker spark://localhost:7077) - Patrick McGloin
That was supposed to be two commands, the ++ meaning a newline. They should work. - Patrick McGloin
Can someone who participated in the comment-based conversation above please summarize in an Answer? thx. - shoover

1 Answers

5
votes

You could follow the Building a Runnable Distribution steps as Gimby mentioned, like this for example:

./make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn

As Patrick McGloin mentioned, you could also start a Spark Cluster on Windows itself, like this for example:

spark-class.cmd org.apache.spark.deploy.master.Master
spark-class.cmd org.apache.spark.deploy.worker.Worker spark://localhost:7077

You can read more in Spark Standalone Mode and How to set up Spark cluster on Windows machines?

Finally as OP mentioned, (s)he was able to built a runnable distribution that created a new Custom-spark tar file in the parent directory. (S)he took this file, untarred it in a different directory, went to the bin subdirectory and then was able to bring up the scala-shell.