Use GCP controller service with custom NiFi processor

Question

I've written a custom NiFi processor for doing some tasks on BigQuery. I would like to be able to re-use the GCP credentials provider controller service that comes bundled with NiFi to use for authentication. Is this possible? I tried adding the nifi-gcp-processors maven dependency to my project, it builds successfully but when I try to start NiFi with my nar it fails with this error message.

2017-12-14 05:09:27,406 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.gcp.storage.PutGCSObject could not be instantiated
java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.gcp.storage.PutGCSObject could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:142)
    at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:117)
    at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:702)
    at org.apache.nifi.NiFi.<init>(NiFi.java:160)
    at org.apache.nifi.NiFi.main(NiFi.java:268)
Caused by: java.lang.NoClassDefFoundError: com/google/cloud/HttpServiceOptions
    at java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
    at java.lang.Class.getConstructor0(Class.java:3075)
    at java.lang.Class.newInstance(Class.java:412)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
    ... 7 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.google.cloud.HttpServiceOptions
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 12 common frames omitted
2017-12-14 05:09:27,408 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2017-12-14 05:09:27,408 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).

James James · Accepted Answer · 2017-12-14T15:56:01

It is possible to reuse the GCPCredentialsControllerService in your custom processor. You may be missing the "NAR dependency" on the nifi-gcp-nar, which will allow NiFi to properly match your processor with the GCP credential service. This is set in the pom.xml file of your custom process bundle's NAR output project, like "my-processors-nar". It might look like:

<dependency>
    <groupId>org.apache.nifi</groupId>
    <artifactId>nifi-gcp-nar</artifactId>
    <version>1.4.0</version>
    <type>nar</type>
</dependency>

See NiFi Archives (NARs) in the NiFi Developers Guide for more details.

Use GCP controller service with custom NiFi processor

1 Answers