How to make multi-project fat jar with sbt-assembly

Question

I have a scala multi-project using spark and try to use the sbt plugin sbt-assembly 0.14.3 to make a fat jar. My buils.sbt look like that:

lazy val commonSettings = Seq(
  organization := "blabla",
  version := "0.1.0",
  scalaVersion := "2.11.8"
)


lazy val core = (project in file("."))
  .settings(commonSettings: _*)
  .settings(libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.6.1" % "provided",
"org.apache.spark" %% "spark-sql" % "1.6.1" % "provided",
"org.apache.spark" %% "spark-mllib" % "1.6.1" % "provided",...)


lazy val sub_project = project
  .settings(commonSettings: _*)
  .aggregate(core)
  .dependsOn(core)

And I would like to create a fat jar of the sub_project such that this fat jar contains all the library and code from the project core. I tried the following:

sbt
project sub_project
assembly

And i get the following error:

[error] missing or invalid dependency detected while loading class file 'blabla.class'.
[error] Could not access term spark in package org.apache,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'blabla.class' was compiled against an incompatible version of org.apache.
[error] one error found

However, when I use "assembly" on the core project, i can get my fat jar.

Does your specified Scala version match that which Spark was compiled against? — nattyddubbs
Yes but, it's just the packaging, i don't yet try to execute it, so even if the version is different, it should not be a problem for the packaging — Jitsumi

Francois G Francois G · Accepted Answer · 2016-06-02T15:00:36

Your build shows that the library dependency on Spark (irrespective of the provided statement) is not present in the classpath of sub_project, and the error message you get matches this. You might want to add this dependency to common settings.

How to make multi-project fat jar with sbt-assembly

1 Answers