What is the maven-shade-plugin used for, and why would you want to relocate Java packages?

Question

I found the maven-shade-plugin being used in someone's pom.xml. I've never used maven-shade-plugin before (and I'm a Maven n00b) so I tried to understand the reason for using this and what it does.

I looked at the Maven docs, however I can't understand this statement:

This plugin provides the capability to package the artifact in an uber-jar, including its dependencies and to shade - i.e. rename - the packages of some of the dependencies.

The documentation on the page doesn't seem very newbie-friendly.

What is an "uber jar?" Why would someone want to make one? What's the point of renaming the packages of the dependencies? I tried to go through the examples on the maven-shade-plugin apache page such as "Selecting contents for Uber Jar," but I still can't understand what is being accomplished with "shading."

Any pointers to illustrative examples/use-cases (with an explanation of why shading was required in this case - what problem is it solving) would be appreciated. Lastly, when should I use the maven-shade-plugin?

Answer 1

Uber JAR, in short, is a JAR containing everything.

Normally in Maven, we rely on dependency management. An artifact contains only the classes/resources of itself. Maven will be responsible to find out all artifacts (JARs etc) that the project depending on when the project is built.

An uber-jar is something that takes all dependencies, and extracts the content of the dependencies and puts them with the classes/resources of the project itself, in one big JAR. By having such an uber-jar, it is easy for execution, because you will need only one big JAR instead of tons of small JARs to run your app. It also eases distribution in some cases.

Just a side-note: avoid using uber-jar as a Maven dependency, as it is ruining the dependency resolution feature of Maven. Normally we create an uber-jar only for the final artifact for actual deployment or for manual distribution, but not for putting to Maven repository.

---

Update: I have just discovered I haven't answered one part of the question : "What's the point of renaming the packages of the dependencies?". Here are some brief updates that will hopefully help people having similar questions.

Creating an uber-jar for ease of deployment is one use case of the shade plugin. There are also other common use cases which involve package renaming.

For example, I am developing Foo library, which depends on a specific version (e.g. 1.0) of Bar library. Assuming I cannot make use of other version of Bar lib (because API change, or other technical issues, etc). If I simply declare Bar:1.0 as Foo's dependency in Maven, it is possible to fall into a problem: A Qux project is depending on Foo, and also Bar:2.0 (and it cannot use Bar:1.0 because Qux needs to use new feature in Bar:2.0). Here is the dilemma: should Qux use Bar:1.0 (which Qux's code will not work) or Bar:2.0 (which Foo's code will not work)?

In order to solve this problem, developer of Foo can choose to use shade plugin to rename its usage of Bar, so that all classes in Bar:1.0 jar are embedded in Foo jar, and the package of the embedded Bar classes is changed from com.bar to com.foo.bar. By doing so, Qux can safely depends on Bar:2.0 because now Foo is no longer depending on Bar, and it is using its own copy of the "altered" Bar located in another package.

Answer 2

I was wondering myself recently why elasticsearch shades and relocates a few (but not all) of its dependencies. Here's an explanation from the project's maintainer, @kimchy:

The shading part is intentional, the shaded libraries we use in elasticsearch are for all intent and purpose part of elasticsearch, the version used is tied closely into what elasticsearch exposes and how it uses the library based on the internals of how the library works (and that changes between versions), netty and guava are great examples.

Btw, I have no problem with actually providing several jars of elasticsearch, one with lucene not shaded, and one with Lucene shaded. Not sure how to do it with maven though. I don't want to provide a version that does not shade netty/jackson for example, because of the deep intimiate usage elasticsearch has with them (for example, using the upcoming bufferring improvement with any previous version of netty except for the current one will actually use more memory compared to using considerably less).

-- https://github.com/elasticsearch/elasticsearch/issues/2091#issuecomment-7156766

And another here from drewr:

The shading is important to keep our dependencies (notably netty, lucene, guava) close to our code so that we can fix an issue even if the upstream provider lags behind. It's possible we will distributed modularized versions of the code, which would help with your particular issue (#2091 for example), but we can't simply remove the shaded dependencies at this time. You can build a local version of ES for your purposes until there's a better solution.

-- https://github.com/elasticsearch/elasticsearch/pull/3244#issuecomment-20125452

So, that's one use case. As for an illustrative example, below is how maven-shade-plugin is used in elasticsearch's pom.xml (v0.90.5). The artifactSet::include lines instruct it what dependencies to pull into the uber JAR (basically, they are unzipped and and re-packaged alongside elasticsearch's own classes when the target elasticsearch jar is produced. (In case you didn't know this already, a JAR file is just a ZIP file that contains the program's classes, resources, etc., and some metadata. You can extract one to see how it's put together.)

The relocations::relocation lines are similar, except that in each case they also apply the specified substitutions to the dependency's classes - in this case, bringing them under org.elasticsearch.common.

Finally the filters section excludes some stuff from the target JAR that oughtn't be in there - such as JAR metadata, ant build files, text files, etc. that are packaged with some dependencies, but which don't belong in an uber JAR.

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.1</version>
        <executions>
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <minimizeJar>true</minimizeJar>
            <artifactSet>
                <includes>
                    <include>com.google.guava:guava</include>
                    <include>net.sf.trove4j:trove4j</include>
                    <include>org.mvel:mvel2</include>
                    <include>com.fasterxml.jackson.core:jackson-core</include>
                    <include>com.fasterxml.jackson.dataformat:jackson-dataformat-smile</include>
                    <include>com.fasterxml.jackson.dataformat:jackson-dataformat-yaml</include>
                    <include>joda-time:joda-time</include>
                    <include>io.netty:netty</include>
                    <include>com.ning:compress-lzf</include>
                </includes>
            </artifactSet>
            <relocations>
                <relocation>
                    <pattern>com.google.common</pattern>
                    <shadedPattern>org.elasticsearch.common</shadedPattern>
                </relocation>
                <relocation>
                    <pattern>gnu.trove</pattern>
                    <shadedPattern>org.elasticsearch.common.trove</shadedPattern>
                </relocation>
                <relocation>
                    <pattern>jsr166y</pattern>
                    <shadedPattern>org.elasticsearch.common.util.concurrent.jsr166y</shadedPattern>
                </relocation>
                <relocation>
                    <pattern>jsr166e</pattern>
                    <shadedPattern>org.elasticsearch.common.util.concurrent.jsr166e</shadedPattern>
                </relocation>
                <relocation>
                    <pattern>org.mvel2</pattern>
                    <shadedPattern>org.elasticsearch.common.mvel2</shadedPattern>
                </relocation>
                <relocation>
                    <pattern>com.fasterxml.jackson</pattern>
                    <shadedPattern>org.elasticsearch.common.jackson</shadedPattern>
                </relocation>
                <relocation>
                    <pattern>org.joda</pattern>
                    <shadedPattern>org.elasticsearch.common.joda</shadedPattern>
                </relocation>
                <relocation>
                    <pattern>org.jboss.netty</pattern>
                    <shadedPattern>org.elasticsearch.common.netty</shadedPattern>
                </relocation>
                <relocation>
                    <pattern>com.ning.compress</pattern>
                    <shadedPattern>org.elasticsearch.common.compress</shadedPattern>
                </relocation>
            </relocations>
            <filters>
                <filter>
                    <artifact>*:*</artifact>
                    <excludes>
                        <exclude>META-INF/license/**</exclude>
                        <exclude>META-INF/*</exclude>
                        <exclude>META-INF/maven/**</exclude>
                        <exclude>LICENSE</exclude>
                        <exclude>NOTICE</exclude>
                        <exclude>/*.txt</exclude>
                        <exclude>build.properties</exclude>
                    </excludes>
                </filter>
            </filters>
        </configuration>
    </plugin>
</plugins>

Answer 3

I think one example of the need for a "shaded" jar is an AWS Lambda function. They seem to only let you upload 1 jar, not an entire collection of .jars like you would find in a typical .war file. So, creating a single .jar with all of the project's dependencies lets you do this.

Answer 4

---

Small warning

Although that's not describe why one would like to use the maven-shade-plugin (since the selected answer describe it pretty well), I would like to note that I had problems with it. It changed the JAR (since that what it's doing) and it caused regression in my software.

So, instead of using this (or the maven-jarjar-plugin), I've used JarJar's binary which seems to work without a problem.

I'm posting here my solution since it took me some time to find a decent solution.

---

Downlaod JarJar's JAR file

You can download the jar from here: https://code.google.com/p/jarjar/ In the left menu you've got a link to download it.

---

How to use JarJar in order to relocate classes of a JAR from one package to another

In this example we'll be changing the package from "com.fasterxml.jackson" to "io.kuku.dependencies.com.fasterxml.jackson". - The source JAR is called "jackson-databind-2.6.4.jar" and new modified (target) JAR is called "kuku-jackson-databind-2.6.4.jar". - The "jarjar" JAR file is in version 1.4

1. Create a "rules.txt" file. The contents of the file should be (watch the period before the '@' character): rule com.fasterxml.jackson.** io.kuku.dependencies.com.fasterxml.jackson.@1

2. Run the following command: java -jar jarjar-1.4.jar process rules.txt jackson-databind-2.6.4.jar kuku-jackson-databind-2.6.4.jar

---

Installing the modified JARs to the local repository

In this case I'm installing 3 files located on "c:\my-jars\" folder.

mvn install:install-file -Dfile=C:\my-jars\kuku-jackson-annotations-2.6.4.jar -DgroupId=io.kuku.dependencies -DartifactId=kuku-jackson-annotations -Dversion=2.6.4 -Dpackaging=jar

mvn install:install-file -Dfile=C:\my-jars\kuku-jackson-core-2.6.4.jar -DgroupId=io.kuku.dependencies -DartifactId=kuku-jackson-core -Dversion=2.6.4 -Dpackaging=jar

mvn install:install-file -Dfile=C:\my-jars\kuku-jackson-databind-2.6.4.jar -DgroupId=io.kuku.dependencies -DartifactId=kuku-jackson-annotations -Dversion=2.6.4 -Dpackaging=jar

---

Using the modified JARs in the project's pom

In this example, this is the "dependencies" element in the projects pom:

<dependencies>
    <!-- ================================================== -->
    <!-- kuku JARs -->
    <!-- ================================================== -->
    <dependency>
        <groupId>io.kuku.dependencies</groupId>
        <artifactId>kuku-jackson-annotations</artifactId>
        <version>2.6.4</version>
    </dependency>
    <dependency>
        <groupId>io.kuku.dependencies</groupId>
        <artifactId>kuku-jackson-core</artifactId>
        <version>2.6.4</version>
    </dependency>
    <dependency>
        <groupId>io.kuku.dependencies</groupId>
        <artifactId>kuku-jackson-databind</artifactId>
        <version>2.6.4</version>
    </dependency>
</dependencies>