How to bundle custom NiFi processor which uses multiple bundles?

Question

I have written one Custom NiFi processor which uses some Hadoop classes, processes flow files and serializes flow files to and from Avro.

As per my knowledge

to use Hadoop classes without packaging them as part of NAR I will need nifi-hadoop-nar bundle AND
to serialize flow file contents to/from Avro, I will need RecordSetWriter and reader which are part of record-serialization-services-api.

Please correct me if my assumption is wrong.

I have written code, tested it and everything works as expected until I try to deploy NAR to NiFi.

When I deploy NAR and restart NiFi, it throws java.lang.ClassNotFoundException: org.apache.nifi.serialization.RecordSetWriterFactory exception.

My NAR's pom.xml looks like below:

<dependencies>
        <dependency>
            <groupId>com.my_company.is.data.tools</groupId>
            <artifactId>custom-data-movement-processors</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-hadoop-nar</artifactId>
            <version>1.5.0</version>
            <type>nar</type>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-record-serialization-service-api</artifactId>
            <version>1.5.0</version>
            <scope>provided</scope>
        </dependency>
</dependencies>

and processor's pom.xml file looks like below:

<dependencies>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-record-serialization-services</artifactId>
            <version>1.5.0</version>
            <scope>provided</scope>
        <dependency>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-record-serialization-services-api</artifactId>
            <version>1.5.0</version>
            <scope>provided</scope>
        <dependency>
</dependencies>

Since I have marked nifi-record-serialization-services-api as provided, it doesn't get bundled in resulting NAR file. Now I can do a quick fix and remove scope altogether and create NAR BUT then NiFi complains about next class not found error.

I would like to know:

In NiFi project's code I found multiple nested bundles which have relation something like this nifi-nar-bundles -> nifi--bundle -> . How to refer two bundles in my processor? As per my knowledge custom processor can have only one parent NAR and NAR = bundle?
What is the preferred way to specify multiple NiFi bundles in pom.xml
Ideally I do not want to bundle Hadoop related dependencies or NiFi dependencies in my NAR file. or is it mandetory in NiFi to bundle these dependencies?

daggett daggett · Accepted Answer · 2019-09-25T21:07:46

    <packaging>nar</packaging>
    <dependencies>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-hadoop-nar</artifactId>
            <version>1.5.0</version>
            <type>nar</type>
        </dependency>

should be enough to get in resulted nar META-INF\MANIFEST.MF

attributes Nar-Dependency-*

Nar-Dependency-Id: nifi-hadoop-nar
Nar-Dependency-Group: org.apache.nifi
Nar-Dependency-Version: 1.5.0

How to bundle custom NiFi processor which uses multiple bundles?

1 Answers