Execute Python Script After PutFile (Apache - NiFi)

0
votes

I am trying to do something that I feel should be relatively simple, but I am struggling immensely to conceptualize how to execute this process.

At a high level I am simply unpacking a zip file into a directory. Once that is complete, I need to execute a python script.

The issue I am having is that I can't determine the best method to achieve this goal. I assume Notify/Wait will be the answer, but setting the release signal identifier and only releasing when all flowfiles have been placed, does not appear to be possible? I have also tried MergeContent using a like attribute, but I only want a single merged flowfile, and I cannot set the minimum # of entries = to ${fragment.count}. Please see below for any screenshots that may be beneficial. Thank you.

enter image description here

1
pythonapache-nifi
What does the python script do? Does it execute once for each FlowFile or once for all 16? - mattyb
It will execute a series of batch scripts which handle logic related to the files. But we only want to execute the batch scripts if all the files are in the directory. So in shorter terms - once for all 16. - lildakota

1 Answers

2
votes

If you are using UnpackContent, all unpacked flowfiles get fragment identifier, index and count attributes - see the Writes Attributes section of the docs.

If you want to collect all of these into a single FlowFile, you can use MergeContent with a Merge Strategy of Defragment. This will take all of the FlowFiles created by the UnpackContent and merge them into a single FlowFile - it will group by the identifier and wait until it has seen every index up to the count - thus if 16 FlowFiles were produced, it will merge all 16 into 1.

You can use that single FlowFile to trigger your script.