How to integrate Spark with Scala project in IntelliJ IDE?

1
votes

I am using IntelliJ IDE to play with Scala and Spark. In this IDE I created a Scala project with a Scala object (see picture).

enter image description here

I used the sample Regression code from this web site.

I also downloaded Spark Framework as a non-Maven user from GitHub (as explained here).

My question is how to link this Spark Framework with my Scala project? Also do I need to integrate some additional libraries to be able to work with Spark MLLib?

1
javascalaintellij-ideaapache-spark

1 Answers

2
votes

I would recommend create simple sbt project with following dependencies in the build.sbt

libraryDependencies ++= Seq(   
  "org.apache.spark" %% "spark-core" % "1.5.1",
  "org.apache.spark" %% "spark-sql" % "1.5.1" )

Then Open it via menu which start importing project from sbt.

When IDEA finishes importing\downloading\indexing libraries you can just ctrl-shift-D and voilà - you have spark console with syntax perfect highlighting multiline editing and completion.

Then you can create directory /src/main/scala if it not exists. Or you could make IDEA create it for you in options:

generate folders

Next you should create new object in the scala directory, select "new scala class - object":

new scala class

Next write some code in the main method in your object, and run it via ctrl-shift-F10 or context command

run

And enjoy your results

results