Apache NIFI: Running a Java process directly after a file is added to a directory

Question

I'm fairly new to Apache NIFI.

I'd like to set up a flow, where there is a file that gets put into a 'hot folder'. If this folder detects a file put into it, this file then gets put into another folder called 'input'. Once the file is copied into the input folder, I'd like trigger off a Java Program to run.

The way i've approached this is to create a 'GETFILE' processor to get the file from the hot folder. and then create a PUTFILE processor to put it in the input folder. So you can imagine there being a connection link between the 'GETFILE' and 'PUTFILE' processors. This works as expected.

However the challenge I'm faced with, is to trigger my Java process to run when the file is copied into the INPUT folder (i.e. after the PUTFILE processor has been executed). I cannot create a link between the PUTFILE and the EXECUTEPROCESS processor (as a means of telling NIFI to run the Java process after the file is copied from the hot folder to the input folder). I can't seem to get the connection arrow to link between the PUTFILE and the EXECUTEPROCESS processors (as NIFI won't let me).

Based on the above description, is there anyone that can recommend an approach to tell NIFI to trigger the Java application to run after detecting the file being added to the input folder?

Thanks.

Joe Witt Joe Witt · Accepted Answer · 2016-02-05T02:03:23

What you are looking to do makes a lot of sense and we actually used to allow something similar with that processor. It turned out though that there were enough edge cases that it became rather problematic to decide what to do with the input flow file so we have a current very explicit model which basically means that processor combined with cron-scheduling is a fancy cron-tool.

So, what we have moved to instead is coming out in NiFi 0.5.0 release which should be in a matter of days. In that we have https://issues.apache.org/jira/browse/NIFI-210 which is a really exciting feature to allow scripting to occur against the flow in-line. The ExecuteScript processor sounds perfect for your case. If you run this code for example you can have it triggered on the presence of data and can wait to listen for the output and capture it as flow file attributes. You could then even route on the content of the response, etc..

def flowFile = session.get()
if (flowFile == null) {
    return;
}
def procout = new StringBuffer(512), procerr = new StringBuffer(512)
def proc = "java -version".execute()
proc.consumeProcessOutput(procout, procerr)
proc.waitForOrKill(1000)
flowFile = session.putAttribute(flowFile, "Process Output", procout.toString())
flowFile = session.putAttribute(flowFile, "Process Error", procerr.toString())
session.transfer(flowFile, REL_SUCCESS)

Let us know if you have more questions.

Thanks Joe

Apache NIFI: Running a Java process directly after a file is added to a directory

1 Answers