How to append a JSON file on FTP in Apache NiFi?

Question

In Apache NiFi, I have a flow in which the flowfiles' contents are arrays of JSON objects. Each flowfile has a unique filename attribute.

// flowfile1:

filename: file1.json
[ {}, {}, {}, ... ]

// flowfile2:

filename: file2.json
[ {}, {}, {}, ... ]

Now, I want to put those files into a FTP server, if a file with the given filename does not exist. If such a file does exist, I want to merge those two files together (concatenate the array from the existing FTP file, with the one from the incoming flowfile) and put that updated file into the FTP. The first case (file does not yet exist) is simple, but how can I go about the second one?

Andy Andy · Accepted Answer · 2020-06-04T22:09:46

You will probably want to use ListFTP to gather the list of files which exist, RouteOnAttribute/RouteOnContent to direct flowfiles referencing existing files to a queue, FetchFTP and MergeContent to join the content of the existing file and the new content, and then PutFTP to place the file on the FTP server again. You will need to investigate approaches to identify the filename attribute in the local flowfiles and match that with the remote file names (I'd suggest persisting the local filenames into a cache when you generate them and routing the remote file listing flowfiles through an enrichment processor. The DistributedMapCache and LookupAttribute processor families will probably be useful here. Abdelkrim Hadjidj has written a great article on using them.

How to append a JSON file on FTP in Apache NiFi?

1 Answers