Corda Dependency Conflicts prevent calling Classloader when using quasar agent

Question

This is a copy of my post on corda's github issues.

I'm trying to include an external dependency to one of the sample projects.

This dependency calls ClassLoader.defineClass, which exits with java.lang.ClassFormatError: Illegal UTF8 string in constant pool in class file org/apache/xerces/impl/xpath/regex/ParserForXMLSchema. The exception is identical to the one mentioned by this quasar issue which suggests it is caused by conflicting dependencies and when using the quasar agent.

After running a dependency report on the project, I could not only see some conflicts between the external library and Corda, but also within Corda itself. So far Gradle has been dealing with them through its transient dependency management. Here is an extract of the report:

+--- org.apache.jena:jena-core:3.6.0                            <-external library
     ...
|    \--- org.apache.jena:jena-base:3.6.0
       ...
|         +--- commons-io:commons-io:2.6      <- external lib dependency
\--- net.corda:corda-node-api:2.0.0
     ...
     +--- commons-beanutils:commons-beanutils:1.9.3       <- corda dependency
     |    +--- commons-logging:commons-logging:1.2
     |    \--- commons-collections:commons-collections:3.2.2
     +--- org.apache.activemq:artemis-core-client:2.1.0
     |    +--- org.jgroups:jgroups:3.6.13.Final
     |    +--- org.apache.activemq:artemis-commons:2.1.0
           ...
     |    |    +--- commons-beanutils:commons-beanutils:1.9.2 -> 1.9.3 (*) <- corda conflict
     ...
     +--- commons-fileupload:commons-fileupload:1.3.3
     |    \--- commons-io:commons-io:2.2 -> 2.6     <-    external lib conflict
    ...

And here is my root build.gradle:

buildscript {
    ext.corda_release_version = '2.0.0'
    ext.corda_gradle_plugins_version = '1.0.0'
    ext.kotlin_version = '1.1.4'
    ext.junit_version = '4.12'
    ext.quasar_version = '0.7.9'

    repositories {
        mavenLocal()
        mavenCentral()
        jcenter()
    }

    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
        classpath "net.corda.plugins:cordformation:$corda_gradle_plugins_version"
        classpath "net.corda.plugins:quasar-utils:$corda_gradle_plugins_version"
        classpath "org.jetbrains.kotlin:kotlin-noarg:$kotlin_version"
    }
}

repositories {
    mavenLocal()
    jcenter()
    mavenCentral()
    maven { url 'https://jitpack.io' }
    maven { url 'https://ci-artifactory.corda.r3cev.com/artifactory/corda-releases' }
}

apply plugin: 'kotlin'
apply plugin: 'net.corda.plugins.cordformation'
apply plugin: 'net.corda.plugins.quasar-utils'

sourceSets {
    main {
        resources {
            srcDir "config/dev"
        }
    }
    test {
        resources {
            srcDir "config/test"
        }
    }
}

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib-jre8:$kotlin_version"
    testCompile "org.jetbrains.kotlin:kotlin-test:$kotlin_version"
    testCompile "junit:junit:$junit_version"

    cordaCompile "net.corda:corda-core:$corda_release_version"
    cordaCompile "net.corda:corda-node-api:$corda_release_version"
    cordaRuntime "net.corda:corda:$corda_release_version"
    testCompile "net.corda:corda-node-driver:$corda_release_version"

    compile "org.apache.jena:jena-core:3.6.0"
}

For reference, the function call from jena-core is a one-line ModelFactory.createDefaultModel() inside a JUnit test. At this point I'm using the minimal required Corda libraries to run other tests since more conflicts arise when compiling unused libraries.

An alternative approach would be to use quasar aot instead of the agent, but I'm new to Gradle let alone Ant, and I haven't been able to find / come up with an example.

Finally, I also tried upgrading ext.corda_gradle_plugins_version = '1.0.0' to 2.0.9 as suggested by the corda docs but gradle is unable to call cordaCompile after the upgrade, and I haven't been able to find any documentation on the plugins.

Edit: I noticed upgrading artemis-core-client from 2.1.0 to 2.4.0 would resolve corda-node-api's internal conflict as you can see here. As for commons-io, sadly commons-fileupload is on the latest version 1.3.3 which hasn't been updated since July '17

Joel Joel · Accepted Answer · 2018-03-02T10:14:12

You need to shade your dependencies, as described here: How can CorDapps deal with transitive dependencies.

Background: What is shading?

Say corda depends on Guava version 1. It will use import statements of the type import com.google.guava.SpecialList, meaning that the class com.google.guava.SpecialList must be present in the classloader when Corda is run.

Now, suppose a developer is writing a CorDapp which uses Guava version 2 instead. When the CorDapp is run, both the Guava version 1 and Guava version 2 JARs will be on the classpath.

At runtime, Java will load all the JARs on its classpath. This means that it's undefined which copy of com.google.guava.SpecialList will be loaded. It depends on order, environment, Java version, etc. And it’s possible that 40% of version 1 will be loaded, whilst 60% of version 2 will be loaded.

If version 1 and 2 are compatible, this is not a problem. Either the CorDapp will use version 1, or the Corda runtime will use version 2. The real fun begins when the two versions are incompatible, because now the application might work in some circumstances, and fail in others – it’s a nightmare.

Shading gives us a way out of this problem. It allows us to re-package dependencies so that their package definitions no longer clash.

As part of the build process, a shade plugin is executed, which will append corda.dep to all classes referenced by Corda and rewrite the bytecode of the importing classes so that they reference the modified package, import corda.dep.com.google.guava.SpecialList.

This results in version 1 of Guava being bundled with the Corda JAR in its own namespace (corda.dep). Any CorDapps that also depend on Guava can now safely import this library and include it on their classpath without conflicting with the version used by Corda.

There is a Gradle plugin for shading called shadow. We would invoke this plugin as the last stage of the build which produces the Corda runtime jar.

Corda Dependency Conflicts prevent calling Classloader when using quasar agent

1 Answers