Debugging Spark standalone cluster with idea

Question

I am trying to debug a Spark Application on a local cluster using a master and a worker nodes. I have been successful at setting up the master node and worker nodes using Spark standalone cluster manager with start-master.sh and it works.But I want to how Spark Application works in the spark cluster, so I want to start the cluster in debug mode. I read the start-master.sh code， mock the args and start org.apache.spark.deploy.master.Master main method.Unfortunately it gets a NoClassDefFoundError,I can't open the webui. I want to know where the problem is.

The Error is :

Exception in thread "dispatcher-event-loop-1" java.lang.NoClassDefFoundError: org/eclipse/jetty/util/thread/ThreadPool
    at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:81)
    at org.apache.spark.deploy.master.ui.MasterWebUI.initialize(MasterWebUI.scala:48)
    at org.apache.spark.deploy.master.ui.MasterWebUI.<init>(MasterWebUI.scala:43)
    at org.apache.spark.deploy.master.Master.onStart(Master.scala:131)
    at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:122)
    at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
    at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
    at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.thread.ThreadPool
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 11 more

my debug configurations is: enter image description here

Thanks!

KrazyGautam KrazyGautam · Accepted Answer · 2018-04-25T12:12:54

I would suggest not to even use a spark standalone cluster for debugging.

You can run spark locally in the your IDE with breakpoints.
Spark provides you option to run locally pointing to local filesystem as HDFS.

Please follow the following link to know more about how to write test cases for local mode in spark http://bytepadding.com/big-data/spark/word-count-in-spark/

Debugging Spark standalone cluster with idea

1 Answers