Dynamic parameter for GetFile/ListFile processor - NiFi

Question

My workflow is like below.

ListenHTTP(i get a directory name here) --> SplitText --> ExtractText(directory name added as attribute)

Now after this i will have to use that attribute directoryname and extract all the files in that local dir and put that into HDFS. I understand GetFile/ListFile could do this, but how do we provide a dynamic directory name to that processor?

Answer 1

Unfortunately, both GetFile and ListFile are source processors, which means they do not accept an incoming flowfile. The general pattern is to configure these processors with a static Input Directory value and allow them to read from it and manage their state.

In this case, I believe you need to use FetchFile, which accepts an incoming flowfile and reads the file path provided. By default, the File to Fetch property is set to ${absolute.path}/${filename}, which means it uses Apache NiFi Expression Language to resolve the value of those two attributes on the incoming flowfile. You could pass that flowfile to an ExecuteStreamCommand processor first and perform an ls on the directory, then split the results into individual flowfiles with one filename per line, and process each of these through the FetchFile.

I understand this isn't the most concise way to perform the task. Two other suggestions would be:

1. Open a Jira to request a processor which retrieves all of the files in a directory (at the time of incoming flowfile receipt) and requires an incoming flowfile to determine the directory.
2. Use an ExecuteScript processor. The processor would simply extract the attribute from the incoming flowfile and use Groovy/Ruby/Python/etc. facilities to retrieve the files from the directory, or perform the directory listing and pass individual flowfiles downstream to a FetchFile processor.