4
votes

I installed Scala(Version 2.12.8) and Spark(2.4.3) on my Mac OS from homebrew. I already have Java 1.8 installed on my machine.

When I launch spark-shell, I see the logo says:

Spark version 2.4.3, Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)

Why does is says Scala version 2.11.12 instead of Scala(Version 2.12.8) which installed on my machine?

Does Spark 2.4.3 come with Scala 2.11.12?

Thanks.

2

2 Answers

7
votes

As stated in the release notes:

Spark 2.4.3 is a maintenance release containing stability fixes. This release is based on the branch-2.4 maintenance branch of Spark. We strongly recommend all 2.4 users to upgrade to this stable release.

Note that 2.4.3 switched the default Scala version from Scala 2.12 to Scala 2.11, which is the default for all the previous 2.x releases except 2.4.2. That means, the pre-built convenience binaries are compiled for Scala 2.11. Spark is still cross-published for 2.11 and 2.12 in Maven Central, and can be built for 2.12 from source.

Additionally Scala version you happen to have on your machine is completely irrelevant - Spark uses Scala version that has been used to compile it.

4
votes

Once we start writing spark code, we need to import spark-core and spark-sql in project. If right versions are not installed, code compilation or runtime fails with missing definitions.

To choose the right version of spark and scala libraries:

See the spark version installed by running the spark-shell. It shows the spark and scala versions both. Use those versions only while importing in projects.

For example in SBT: Spark 2.4.5 supports 2.11.12 of scala

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.5"

scalaVersion := "2.11.12"