Add dependency to remote spark on intelliJ

Question

I am running spark in intelliJ remotely but facing difficulties while adding dependency to spark conf.

val conf = new SparkConf()
    .setMaster("spark://IP:7077")
    .set("packages", "com.databricks:spark-avro_2.10:2.0.1:jar")
    .setAppName("localtrial")

Error:

16/02/23 12:27:10 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 172.16.248.156): java.lang.ClassNotFoundException: com.databricks.spark.avro.AvroRelation$$anonfun$buildScan$1$$anonfun$3 at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ```

I have also tried setJars property of conf class. Any help would be appreciated.

wm_eddie wm_eddie · Accepted Answer · 2016-02-23T08:20:48

You need to add the dependency to your build.sbt file so that IntelliJ can compile against it. If you are adding it as an argument to spark-submit then you can set the dependency as prodived else you need to package it inside your jar file by using the sbt-assembly, or similar, plugin.

Add the following line to your build.sbt file and if auto-import is enabled Intellij will download the dependency. If auto-import isn't enabled close the project and import it again or use the refresh button inside the SBT tool window.

libraryDependencies += "com.databricks" %% "spark-avro" % "2.0.1" % "provided"

Add dependency to remote spark on intelliJ

1 Answers