Type mismatch after upgrading SBT

Question

I am trying to upgrade a historic Scala/Spark SBT project to SBT version 1.1.0. It uses Scala 2.10.6, as specified in the build.sbt file. It is unclear for which SBT version the build.sbt file was implemented, but presumably 0.12.x or even earlier.

I fail to migrate the assemblyExcludedJars line in the build.sbt file:

import sbt.Keys.{libraryDependencies, _}
import sbtassembly.AssemblyKeys.assemblyExcludedJars

[...]

lazy val sparkSettings = Seq(
  libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-sql" % "1.6.0",
      [...],
  assemblyExcludedJars in assembly <<= (fullClasspath in assembly) map { cp =>
val excludes = Set(
  "minlog-1.2.jar",
  [...]
)
cp filter { jar => excludes(jar.data.getName) }
  }
)

As pointed out in the migration guide, the <<= is no longer supported, so I changed it to :=.

When I run sbt assembly after the change, the following error is thrown:

$ sbt assembly
[info] Loading settings from idea.sbt ...
[info] Loading global plugins from /home/XXX/.sbt/1.0/plugins
[info] Loading settings from assembly.sbt ...
[info] Loading project definition from YYY/project
YYY/build.sbt:49: error: type mismatch;
 found   : sbt.Def.Initialize[sbt.Task[Seq[sbt.internal.util.Attributed[java.io.File]]]]
 required: sbt.Keys.Classpath
(which expands to)  Seq[sbt.internal.util.Attributed[java.io.File]]
  assemblyExcludedJars in assembly := (fullClasspath in assembly) map { cp =>
                                                              ^
[error] sbt.compiler.EvalException: Type error in expression
[error] sbt.compiler.EvalException: Type error in expression
[error] Use 'last' for the full log.
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? 

I could not find a lot in the SBT documentation about what exactly this line is supposed to do. I find it rather confusing though that the type of assembly seems to have changed.

I am also looking at this in IntelliJ Idea, but it actually indicates an error Cannot resolve symbol assembly.

How can I migrate this build file correctly?

Answer 1

To rewrite the assemblyExcludedJars setting it should look something like this:

assemblyExcludedJars in assembly := { 
  val cp = (fullClasspath in assembly).value
  val excludes = Set("minlog-1.2.jar", [...])
  cp filter { jar => excludes(jar.data.getName) }
}

Just replacing the <<= operator will fail because the := operator takes the result type of the task on the right-hand side instead of an Initialize[Task[...]]

Answer 2

Sometimes assemblyExcludedJars doesn't work.

The dirty way is to use defind in merge strategy.

assemblyMergeStrategy in assembly :={
    case PathList("lib", "static", "Windows", xs @ _*) => MergeStrategy.discard
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case PathList("com","sun", xs @ _*)=> MergeStrategy.discard
    case PathList("mx4j", xs @ _*)=> MergeStrategy.discard
    case PathList("org","aopalliance", xs @ _*)=> MergeStrategy.discard
    case PathList("org","apache","jasper", xs @ _*)=> MergeStrategy.discard
    case PathList("javax", xs @ _*)=> MergeStrategy.discard
    case PathList("org", "apache", "jasper", xs @ _*)=> MergeStrategy.discard
    case PathList("org", "fusesource", xs @ _*)=> MergeStrategy.discard
    case PathList("org", "objenesis", xs @ _*)=> MergeStrategy.discard
    case PathList("org", "scalactic", xs @ _*)=> MergeStrategy.discard
    case PathList("org", "scalactic", xs @ _*)=> MergeStrategy.discard
    case PathList("parquet",  xs @ _*)=> MergeStrategy.discard
    case PathList("javax", xs @ _*)=> MergeStrategy.discard
    case PathList("org", "apache", "jasper", xs @ _*)=> MergeStrategy.discard
    case PathList("org", "fusesource", xs @ _*)=> MergeStrategy.discard
    case PathList("org", "objenesis", xs @ _*)=> MergeStrategy.discard
    case PathList("org", "scalactic", xs @ _*)=> MergeStrategy.discard
    case PathList("parquet",  xs @ _*)=> MergeStrategy.discard
    case PathList("akka",  xs @ _*)=> MergeStrategy.discard
    case PathList("antlr",  xs @ _*)=> MergeStrategy.discard
    case PathList("assets",  xs @ _*)=> MergeStrategy.discard
    case PathList("au",  xs @ _*)=> MergeStrategy.discard
    case PathList("breeze",  xs @ _*)=> MergeStrategy.discard
    case PathList("codegen",  xs @ _*)=> MergeStrategy.discard
    case PathList("contribs",  xs @ _*)=> MergeStrategy.discard
    case PathList("darwin",  xs @ _*)=> MergeStrategy.discard
    case PathList("db",  xs @ _*)=> MergeStrategy.discard
    case PathList("edu",  xs @ _*)=> MergeStrategy.discard
    case PathList("fr",  xs @ _*)=> MergeStrategy.discard
    case PathList("hbase-webapps",  xs @ _*)=> MergeStrategy.discard
    case PathList("html",  xs @ _*)=> MergeStrategy.discard
    case PathList("i18n",  xs @ _*)=> MergeStrategy.discard
    case PathList("images",  xs @ _*)=> MergeStrategy.discard
    case PathList("io",  xs @ _*)=> MergeStrategy.discard
    case PathList("javaewah",  xs @ _*)=> MergeStrategy.discard
    case PathList("javassist",  xs @ _*)=> MergeStrategy.discard
    case PathList("javolution",  xs @ _*)=> MergeStrategy.discard
    case PathList("jersey",  xs @ _*)=> MergeStrategy.discard
    case PathList("jline",  xs @ _*)=> MergeStrategy.discard
    case PathList("jodd",  xs @ _*)=> MergeStrategy.discard
    case PathList("joptsimple",  xs @ _*)=> MergeStrategy.discard
    case PathList("junit",  xs @ _*)=> MergeStrategy.discard
    case PathList("kafka",  xs @ _*)=> MergeStrategy.discard
    case PathList("linux",  xs @ _*)=> MergeStrategy.discard
    case PathList("macrocompat",  xs @ _*)=> MergeStrategy.discard
    case PathList("mozilla",  xs @ _*)=> MergeStrategy.discard
    case PathList("net",  xs @ _*)=> MergeStrategy.discard
    case PathList("nu",  xs @ _*)=> MergeStrategy.discard
    case PathList("ooyala",  xs @ _*)=> MergeStrategy.discard
    case PathList("OSGI-OPT",  xs @ _*)=> MergeStrategy.discard
    case PathList("py4j",  xs @ _*)=> MergeStrategy.discard
    case PathList("scala",  xs @ _*)=> MergeStrategy.discard
    case PathList("schema",  xs @ _*)=> MergeStrategy.discard
    case PathList("shaded",  xs @ _*)=> MergeStrategy.discard
    case PathList("shapeless",  xs @ _*)=> MergeStrategy.discard
    case PathList("snappydata",  xs @ _*)=> MergeStrategy.discard
    case PathList("spark",  xs @ _*)=> MergeStrategy.discard
    case PathList("spire",  xs @ _*)=> MergeStrategy.discard
    case PathList("spray",  xs @ _*)=> MergeStrategy.discard
    case PathList("tables",  xs @ _*)=> MergeStrategy.discard
    case PathList("templates",  xs @ _*)=> MergeStrategy.discard
    case PathList("twitter4j",  xs @ _*)=> MergeStrategy.discard
    case PathList("webapps",  xs @ _*)=> MergeStrategy.discard
    case PathList("win32",  xs @ _*)=> MergeStrategy.discard
    case PathList("org", "antlr",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "bouncycastle",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "cliffc",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "cloudera",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "codehaus",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "datanucleus",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "dmg",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "eclipse",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "eigenbase",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "flywaydb",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "glassfish",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "h2",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "hamcrest",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "I0Itec",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "iq80",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "jamon",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "jboss",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "jcodings",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "jets3t",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "joda",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "joni",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "jpmml",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "junit",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "jvnet",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "j_paine",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "mortbay",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "netlib",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "omg",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "osgi",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "parboiled",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "parboiled2",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "roaringbitmap",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "rogach",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "scalatest",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "stringtemplate",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "supercsv",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "tukaani",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "uncommons",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "w3c",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "xerial",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "znerd",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "clearspring",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "esotericsoftware",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "facebook",xs @ _*)=> MergeStrategy.discard
    case PathList("org", "fasterxml",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "gemstone",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "github",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "jamesmurty",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "jolbox",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "ning",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "pivotal",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "thoughtworks",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "univocity",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "vmware",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "yammer",xs @ _*)=> MergeStrategy.discard
    case PathList("com", "zaxxer",xs @ _*)=> MergeStrategy.discard
    case x => MergeStrategy.first
}