I need to use HDFS file as lookup in a NiFi job. This HDFS file contain two fields id, name and I need to pass IDS through my flow and retrieve Name from HDFS file. My HDFS file in parquet snappy format. Could you please let me know can I achieve this through lookup processor or should I follow different approach.
Any example workflow would be really helpful.
Thanks all in advance.
Kumar
HDFS as a lookup can be really inefficient, which is likely why we don't have a controller service for it. Where are you getting the files that you want to do the lookup on? Maybe instead you can get the HDFS files with NiFi and do a "reverse lookup" where you enrich the HDFS file with the fields from what you're trying to do the lookup with?
Having said that, NiFi does have HBase lookup services, so an alternative is to put your HDFS files into HBase and do the lookup that way.
