Apache NIFI "Execute Processor"

2
votes

Can Apache NIFI "ExecuteSQL Processor" stream large set of select result in chunks say 'x' MB?

2
apache-nifi
From what I understand it was built for large queries in mind which is why it streams the data. I don't think there is any way to chunk the data. I think chunking or "batching" is reserved specifically for Get and Post processors. - pyRabbit

2 Answers

6
votes

The ExecuteSQL Processor can "stream" large numbers of rows in the sense that it will stream the data directly to FlowFile content (which will not be held in-memory/heap) so it is very memory efficient. It does not, at this time, chunk the results, though. There is a ticket https://issues.apache.org/jira/browse/NIFI-1251 to provide such capabilities, though.

0
votes

You can now use the QueryDatabaseTable processor which supports chunking by the "Max Rows Per Flow File" attribute.