sbt assembly package dependencies in multiple artifacts

1
votes

I am trying to generate, with sbt assembly, from a single project several jars. Each containing some of the dependencies.

So far I have found only this QA that is close to what I am looking for. However I don't need to have separate configs, basically when I run assembly, I just want to generate all the different jars.

To be more concrete. I want to generate:

  • One jar with my code and some general dependencies
  • One jar with hadoop dependencies <- this is the problem, as I don't know how to say, generate another jar that has only those dependencies.
  • One jar with scala
1
scalasbtsbt-assembly

1 Answers

0
votes

Without going deep into complex sbt configurations, you could try another approach. The hadoop dependencies being standard, you could mark them as provided in your build to exclude them. 

"org.apache.hadoop" % "hadoop-client" % "2.6.0" % "provided"

For Scala, the library jar is also standard and can be downloaded separately by your "user". To remove it from the fat jar, use the following setting (assembly 0.13.0):

assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

The user of your fat jar is then aked to provide both Scala and Hadoop libraries in the classpath.

For example, when using Spark this is the correct approach as these two libraries are both provided by the Spark running environment. The same logic applies for the Hadoop MapReduce environment.