Nifi: Merging all of necessary flowfile in one shot with MergeContent Processor

3
votes

This is the question about MergeContent processor in Nifi. Currently, I need to combine all flowfiles with one particular attribute in one shot manner.

But what happens is since there are so many flowFile with the same attribute, the processor produces a few different flowfiles merged with the attribute, and those few should be combined into one big flowfile together.

Are they any ways to wait for all flowfile with same attribute to come into the MergeContent Processor and merge all of them at one time?

If not, what could be an alternative?

2
mergeapache-nifi
do you have count of files to wait for? or any other limit that nifi should wait for files with same attribute? - daggett
No I don't have count of files. It varies from time to time depending on size of input file. And input file is split by some tag (with EvaluateXQuery) to a large number of different flowfiles. So how many small files with same attribute value is unknown. Or can I know somehow? - Micro_Andy

2 Answers

1
votes

You can increase the "Minimum Number of Entries" and "Maximum Number of Entries" to suit your use case and increase the "Max Bin Age" to a suitable time.

You can refer all the properties for merge-content here : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.MergeContent/index.html

Also, are you using nifi in a cluster or stand alone? Because the flowfiles in one node cannot be merged with the flowfiles in the other node.

For instance, if you're using a nifi cluster with 5 nodes, each having 10 flowfiles. After using the merge content processor, you will have 5 files.

Let me know if you need further assistance!

0
votes

Before the MergeContent Processor you should configure your connector with a Load Balancing policy of "Single Node" so that you guarantee that only 1 node will process the defragmentation.