Find unresolved maven dependencies in multi-module maven project

Question

I'd like maven to report unresolved dependencies in multi-module maven Java project which has below structure:

multi-module-java-app/
├── app1
│   ├── pom.xml
│   └── src
├── app2
│   ├── pom.xml
│   └── src
└── pom.xml

poms are at the bottom.

Background:

Maven is used as a dependency management and build tool. Artifactory is repository manager. Artifacts may be built using maven locally on a developer’s environment or on Jenkins build server. Artifactory periodically moves artifacts to special archive repository which is part of all repository virtual.

Maven caches locally built artifacts in~/.m2 directory on the computer where Maven runs whether it’s developer environment or build server.

Problem

Several issues may arise:

Local builds on developers’ VMs may succeed, but fail in Jenkins.

Local builds, Jenkins builds may succeed, but fail on another developer’s VM.

Cause

Present/missing artifacts on a developer .m2 cache, missing artifacts in build server’s .m2 cache and/or archive Artifactory repository

Proposed solution

Run [path_to_maven]mvn dependency:list. [path_to_maven] has custom maven installation (with empty .m2 cache) in the root folder of the project. Custom maven is also configured using settings.xml to use special non-archived repository (all repository without the archive). The output is like below:

It reports unresolved dependencies as well as dependent artifacts which miss them. However this solution has 2 main drawbacks:

.m2 missing slows down the detection significantly as all the dependencies have to be downloaded
unresolved artifacts or modules which miss them are always snapshots.

Running mvn -B -q versions:set -DnewVersion=[some_version] solves the second. This command runs during release pipeline anyway.

However it's not clear how to solve the first.

How to find unresolved maven dependencies while using .m2, so that unresolved dependencies may be detected quickly during Jenkins build after each push to feature branch?

The only idea that pops is that .m2 on build server will be synced with Artifactory.

.m2 on developers' machines can be synced as well using some sort of custom plugin that uses rsync. Is there a known plugin that does this?

The ultimate goal is to remove archive repository and let the builds fail. But, first developers need to align the dependencies to the latest versions.

root pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
 
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
 
  <groupId>com.mycompany.app</groupId>
  <artifactId>multi-module-java-app</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>pom</packaging>
 
  <name>multi-module-java-app</name>
  <!-- FIXME change it to the project's website -->
  <url>http://www.example.com</url>
 
  <pluginRepositories>    
      <pluginRepository>
          <id>plugins</id>
          <name>plugins</name>
          <url>http://localhost:8081/artifactory/all</url>
      </pluginRepository>
  </pluginRepositories>
 
  <repositories>
      <repository>
          <id>all</id>
          <name>all</name>
          <url>http://localhost:8081/artifactory/all</url>
      </repository>
  </repositories>
 
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>com.example</groupId>
        <artifactId>spring-boot</artifactId>        
      </dependency>
    </dependencies>
  </dependencyManagement>
 
  <modules>
      <module>app1</module>
      <module>app2</module>
  </modules>
 
</project>

app 1 pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
 
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
        <groupId>com.mycompany.app</groupId>
        <artifactId>multi-module-java-app</artifactId>
        <version>1.0-SNAPSHOT</version>
        <relativePath>../pom.xml</relativePath>
  </parent>
  <groupId>com.mycompany.app.app1</groupId>
  <artifactId>app1</artifactId>
  <version>1.0-SNAPSHOT</version>
 
  <name>app1</name>
 
  <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <finalName>${artifactId}</finalName>
                </configuration>
            </plugin>
        </plugins>
  </build>
  <pluginRepositories>    
      <pluginRepository>
          <id>plugins</id>
          <name>plugins</name>
          <url>http://localhost:8081/artifactory/all</url>
      </pluginRepository>
  </pluginRepositories>
 
  <repositories>
      <repository>
          <id>all</id>
          <name>all</name>
          <url>http://localhost:8081/artifactory/all</url>
      </repository>
  </repositories>
 
  <dependencies>
    <dependency>
      <groupId>com.example</groupId>
      <artifactId>spring-boot</artifactId>
      <version>0.0.1-20200510.095344-1</version>
    </dependency>
  </dependencies>
 
</project>

app2 pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
 
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
        <groupId>com.mycompany.app</groupId>
        <artifactId>multi-module-java-app</artifactId>
        <version>1.0-SNAPSHOT</version>
        <relativePath>../pom.xml</relativePath>
  </parent>
  <groupId>com.mycompany.app.app2</groupId>
  <artifactId>app2</artifactId>
  <version>1.0-SNAPSHOT</version>
 
  <name>app2</name>
  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-jar-plugin</artifactId>
              <configuration>
                  <finalName>${artifactId}</finalName>
              </configuration>
          </plugin>
      </plugins>
  </build>
 
  <pluginRepositories>    
      <pluginRepository>
          <id>plugins</id>
          <name>plugins</name>
          <url>http://localhost:8081/artifactory/all</url>
      </pluginRepository>
  </pluginRepositories>
 
  <repositories>
      <repository>
          <id>all</id>
          <name>all</name>
          <url>http://localhost:8081/artifactory/all</url>
      </repository>
  </repositories>
 
  <dependencies>
    <dependency>
      <groupId>com.example</groupId>
      <artifactId>spring-boot</artifactId>
      <version>0.0.1-20200510.095344-1</version>
    </dependency>
  </dependencies>
 
</project>

Wow, that's a scholastic context. Part of problem 1 is, it's not a problem at all. That's how maven works; you start w/empty local pom and it resolves what you list as dependencies ( and tne transitive dependencies). What you want is dependency:analyze. That aligns your pom with your imports. Clean pom is the way to go. You can also keep module versions aligned by specifying GAV dependencies in root pom but only GA dependencies in sub-modules. — Ian W
I'm not sure why you have this discrepancy between developers repositories and artifactory. Are developers building and installing dependencies locally which haven't been released? — tgdavies
@tgdavies yes, parent and modules have SNAPSHOT versions in pom.xml and developers need to verify that local builds pass before pushing to feature branch after which build will be triggered in Jenkins, which should fail if unresolved dependencies are found — rok
I have not understood yet why you could not just wait till Jenkins builds. If dependencies are missing, the Jenkins build fails and the developer can repair the problem. — J Fabian Meier
@JF Meier. It's because of the .m2 cache which present on Jenkins build server and developers' machines. It may have the archived artifact, but Jenkins .m2 may not and vice versa. .m2 cache differences may exist even between different developer machines — rok

mdabdullah mdabdullah · Accepted Answer · 2020-07-19T17:59:18

I have run into this exact issue where the developer's local build environment was different from Jenkins Slave environment. In the ideal world, the developer needs to baseline their local environment with that of the slave, or depend entirely on Jenkins job builds once initial stages of development are complete.

I appreciate the fact that you are trying to provide an automated sync feature of the .m2 repositories, it is feasible but adds scope for error and additional routine maintenance tasks, not to mention user education issues. For instance, how will you ensure that the .m2 is the latest version? In terms of maven dependencies, the developer knows best, or they may be introducing new dependencies which do not exist on the slave yet. Thereby, I suggest fixing the root problem of developer not aligning their dependencies which is more design related than automation.

Not sure if you want to take this path, but may help someone:

Eliminate the need for .m2 repositories in local machines of developers. The m2 cache creates a problem, if the developer's machine gets wiped out or corrupt and there will be a need for updates, audits and reconciliation.
Eliminate the need for .m2 in Jenkins slaves. The problem here is, multiple slaves tend to have different .m2 cache content and syncing back and forth from Artifactory and then developers also syncing it to their local sounds complicated. There is no saying that all these .m2 will be in sync at any point of time and a builds may still get executed with an n-1 version of it.
Now that there is no .m2 , we still need a place for developers to pull dependencies from. Push all your dependencies to a repo in Artifactory and actively maintain it. Use the setting.xml feature to pull the standard dependencies. If a developer is building in his local machine, via Eclipse or other, the dependencies will be available to be pulled from the IDE itself by using the same xml reference so there is no local cache maintained on the developer's machine.
When the build environment has a minor difference between the local and the Jenkins Slave This causes the .jar to be the same size or slightly different in kilobytes of size.

To identify this difference between jar files use any tools listed here in this post, it also helps developers to identify which dependency is out of sync, by themselves: Comparing two .jar files

If this design is implemented, the Artifactory repository containing the dependencies becomes the single source of truth for dependencies, you need to work with developers to create a cadence as to how the dependencies will be updated and consumed to/from this single source of truth. I hope this is useful.

Find unresolved maven dependencies in multi-module maven project

1 Answers