NiFi LookupRecord for multiple fields

Question

I have a requirement of filtering the records from the source file if the value of few attributes (e.g., Emp_number, Manager_id, associate_num) matches with the list of values available in a lookup file (e.g., column name ID in a csv lookup file). For this, I have used NiFi LookupRecord processor (version 1.5). Added dynamic property in the lookup processor as "key" and gave the recordpath value as /Emp_number. It is working fine. But if i add additional dynamic properties like "key1": /Manager_id and "key2": /associate_num, it is not filtering the records which has matching values in column Manager_id and associate_num with lookup file.

As per the definition in NiFi document, my understanding is multiple field lookup is supported. "The "coordinates" to use for looking up a value in the Lookup Service are defined by adding a user-defined property. Each property that is added will have an entry added to a Map, where the name of the property becomes the Map Key and the value returned by the RecordPath becomes the value for that key."

Could any one please help me on what am missing here.

Answer 1

The LookupRecord processor will take all of the user defined properties and create a coordinates Map and then call a LookupService.

The LookupService has a method like:

lookup(final Map<String, Object> coordinates)

Each implementation of LookupService may work differently and may or may not accept multiple coordinates.

There is a method on the service Set<String> getRequiredKeys(); where the service defines the required keys that are expected to be in the coordinates Map.

For example, CsvRecordLookupService returns a set of keys with one string in it called "key", so that service only supports a single coordinate.

I haven't gone through all the services, but many of them only support a single coordinate called "key".