IntelliJ Idea 14: cannot resolve symbol spark

6
votes

I made a dependency of Spark which worked in my first project. But when I try to make a new project with Spark, my SBT does not import the external jars of org.apache.spark. Therefore IntelliJ Idea gives the error that it "cannot resolve symbol". I already tried to make a new project from scratch and use auto-import but none works. When I try to compile I get the messages that "object apache is not a member of package org". My build.sbt looks like this:

name := "hello"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies += "org.apache.spark" % "spark-parent_2.10" % "1.4.1"

I have the impression that there might be something wrong with my SBT settings, although it already worked one time. And except for the external libraries everything is the same... I also tried to import the pom.xml file of my spark dependency but that also doesn't work. Thank you in advance!

6
scalaintellij-ideaapache-sparksbt
scalaVersion := "2.11.7" possibly conflicts with "spark-parent_**2.10**" try spark-parent_2.11 - Sascha Kolberg
I have also been thinking this but this doesnt work. And it already worked with spark-parent 2.10... - Giselle Van Dongen
Does it compile in terminal? Are you using default resolvers or some kind of proxy (Nexus, Artifactory)? What is the log output? - Sascha Kolberg
I also get this message: SBT project import [warn] [FAILED ] org.apache.spark#spark-parent_2.11;1.4.1!spark-parent_2.11.jar(doc): (0ms) [warn] ==== local: tried [warn] C:\Users\Giselle\.ivy2\local\org.apache.spark\spark-parent_2.11\1.4.1\docs\spark-parent_2.11-javadoc.jar [warn] ==== public: tried [warn] repo1.maven.org/maven2/org/apache/spark/spark-parent_2.11/1.4.1/… [warn] [FAILED ] org.apache.spark#spark-parent_2.11;1.4.1!spark-parent_2.11.jar(src): (0ms) - Giselle Van Dongen
I dont use any resolvers (as far as I'm aware of) - Giselle Van Dongen

6 Answers

7
votes

This worked for me->

name := "ProjectName"
version := "0.1"
scalaVersion := "2.11.11"

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.11" % "2.2.0",
  "org.apache.spark" % "spark-sql_2.11" % "2.2.0",
  "org.apache.spark" % "spark-mllib_2.10" % "1.1.0"
)
2
votes

I use 

scalaVersion := "2.11.7"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1"

in my build.sbt and it works for me.

2
votes

I had a similar problem. It seems the reason was that the build.sbt file was specifying the wrong version of scala.

If you run spark-shell it'll say at some point the scala version used by Spark, e.g.

Using Scala version 2.11.8

Then I edited the line in the build.sbt file to point to that version and it worked.

2
votes

Currently spark-cassandra-connector compatible with Scala 2.10 and 2.11.

It worked for me when I updated the scala version of my project like below:

ThisBuild / scalaVersion := "2.11.12"

and I updated my dependency like:

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.4.0",

If you use "%%", sbt will add your project’s binary Scala version to the artifact name.

From sbt run: 

sbt> reload
sbt> compile
1
votes

Your library dependecy conflicts with with the scala version you're using, you need to use 2.11 for it to work. The correct dependency would be:

scalaVersion := "2.11.7"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.4.1"

note that you need to change spark_parent to spark_core

0
votes

name := "SparkLearning"

version := "0.1"

scalaVersion := "2.12.3"

// additional libraries libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"