Eclipse Scala IDE code not compiling

Question

I downloaded eclipse scala ide from scala-ide.org site and trying to compile my first scala word count program. But its gives error "object not a member of package org" in the following import command

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

After some research I found that I need to add the jar file spark-assembly-1.0.0-hadoop2.2.0.jar to overcome this issue

But after doing lot of research I could not locate this jar. Can anyone help here ?

Answer 1

• Install the SBT Scala build+dependency tool
• Create an empty directory. Name it spark-test or whatever you want to name your project.
• Put your source code in the sub-directory src/scala/main. If you have Main.scala in package scalatest, it should be src/scala/main/scalatest/Main.scala

• Make a build.sbt file with the following contents

  name := """sparktest"""

  version := "1.0-SNAPSHOT"

  scalaVersion := "2.11.7"

  libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "1.4.0" )

• Configure the SBT Eclipse plugin. Create ~/.sbt/0.13/plugins/plugins.sbt, with:

  addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")

• Generate an Eclipse project with sbt eclipse

• Import your project into eclipse.
• Run :)

Answer 2

Scala is not a simple language/env to learn. It is important you learn how scala works and then move into spark. There are tons of material available on web. A proper learning path will be to learn SBT > SCALA > Use Scala for Spark

The dependency that you mentioned, can be put in he sbt's build.sbt. You can also use maven, but I recommend learning sbt as way of learning scala. Once you have resolved, the dependency using SBT, your simple code should work fine. But still, I recommend doing a "hello world" first than doing a "word count" :-)

Ando to answer your question, in your SBT you should be adding following library,

libraryDependencies += "org.apache.spark" % "spark-assembly_2.10" % "1.1.1"

This was for spark assembly 1.1.1 for hadoop 2.10. I see you need a different version, you can find the proper version at

Maven Repo details for spark/hadoop

Here's the pure eclipse solutions (I had to download and setup eclipse just to answer this question)

1. Get Scala IDE (it comes with inbuilt Scala Compiler version 2.10.5 and 2.11.6)
2. Create a new project Scala Wizard > Scala Project
3. Right click "src" in the scale project , select "Scala Object", give it a name - I gave WordCount
4. Right click project > Configure > Convert to Maven Project
5. In the body of word count object (I named the object as WordCount) paste the text from Apache Spark Example which finally looks like

```

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object WordCount {
    val sparkConf = new SparkConf().setAppName("SampleTest")
    val spark = new SparkContext(sparkConf)
    val textFile = spark.textFile("hdfs://...")
    val counts = textFile.flatMap(line => line.split(" "))
                                   .map(word => (word, 1))
                                   .reduceByKey(_ + _)
    counts.saveAsTextFile("hdfs://...")
}

``` 6. Add following to your maven

```

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.4.0</version>
</dependency>

```

7. Right click on "Scala Library Container", and select "Latest 2.10 bundle", click ok
8. Once I was done with these, there was no error message displayed on my "problems" list of Eclipse...indicating that it compiled as expected.
9. Obviously, this example won't run as I haven't provided enough info for it to run...but this was just to answer to the question, "how to get the code compiled".

Hope this helps.