I am trying to understand NiFi data flow mechanism . I read that Nifi has flow file which holds content and metadata (flow file attribute).
So I wanted to understand if I have 1 TB of data placed on edge node and would like to pass it to Nifi processors , is it going to load everything into memory to be used by processor?
FlowFiles (herein referred to as FF) are analogous to HTTP data in that they are comprised of content and attributes (metadata) as you highlight. However, the way these are handled within the NiFi framework is that the metadata resides in memory (up to a configured limit per connection) and the content portion of the FF is actually a pointer to the content on disk. That is once the content is received into NiFi, it is not longer held in memory at any point in time, utilizing a pass by reference approach allowing NiFi to handle arbitrarily large files. The only thing stored in memory is the metadata of FFs which is configurable to limit the number on a per connection basis.
When a processor needs to make a change, this exercises the copy on write approach for modifications.
In general, processors use a streaming approach for reading/writing data to/from the content repository. To that end, the included processors avoid storing the totality of a FF's content in memory as it could prove prohibitive. Simple routing and movement of data for an arbitrarily large file should be no issue; avoiding excess pressure on the heap memory. When looking at doing transformations/modifications on such files, the answer is that it is typically okay, but it depends on the specifics of the data type.
