NiFi ListHDFS cannot find directory, FileNotFoundException

Question

Have pipeline in NiFi of the form listHDFS->moveHDFS, attempting to run the pipeline we see the error log

13:29:21 HSTDEBUG01631000-d439-1c41-9715-e0601d3b971c
ListHDFS[id=01631000-d439-1c41-9715-e0601d3b971c] Returning CLUSTER State: StandardStateMap[version=43, values={emitted.timestamp=1525468790000, listing.timestamp=1525468790000}]
13:29:21 HSTDEBUG01631000-d439-1c41-9715-e0601d3b971c
ListHDFS[id=01631000-d439-1c41-9715-e0601d3b971c] Found new-style state stored, latesting timestamp emitted = 1525468790000, latest listed = 1525468790000
13:29:21 HSTDEBUG01631000-d439-1c41-9715-e0601d3b971c
ListHDFS[id=01631000-d439-1c41-9715-e0601d3b971c] Fetching listing for /hdfs/path/to/dir
13:29:21 HSTERROR01631000-d439-1c41-9715-e0601d3b971c
ListHDFS[id=01631000-d439-1c41-9715-e0601d3b971c] Failed to perform listing of HDFS due to File /hdfs/path/to/dir does not exist: java.io.FileNotFoundException: File /hdfs/path/to/dir does not exist

Changing the listHDFS path to /tmp seems to run ok, thus making me think that the problem is with my permissions on the directory I'm trying to list. However, changing the NiFi user to a user that can access that directory (eg. hadoop fs -ls /hdfs/path/to/dir) by setting the bootstrap.properties value run.as=myuser and restarting (see https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#bootstrap_properties) still seems to produce the same problem for the directory. The literal dir. string being used that is not working is:

"/etl/ucera_internal/datagov_example/raw-ingest-tracking/version-1/ingest"

Does anyone know what is happening here? Thanks.

** Note: The hadoop cluster I am accessing does not have kerberos enabled (it is a secured MapR hadoop cluster).

Update: It appears that the mapr hadoop implementation is different enough that it requires special steps in order for NiFi to properly work on it (see https://community.mapr.com/thread/10484 and http://hariology.com/integrating-mapr-fs-and-apache-nifi/). May not get a chance to work on this problem for some time to see if still works (as certain requirements have changed), so am dumping the link here for others who may have this problem in the meantime.

I would suggest changing the permissions on the HDFS directory to include the nifi user rather than changing the NiFi run.as user because a number of other conditions depend on the OS user NiFi is running under. — Andy
A quick question. Are you using the correct core-site.xml & hdfs-site.xml? Can you check the files when you use /tmp is the same when you check using hdfs dfs -ls /tmp ? — Sivaprasanna Sethuraman

notNull notNull · Accepted Answer · 2018-05-05T00:34:40

Could you once make sure you have entered correct path and directory needs to be exists in HDFS.

It seems to be list hdfs processor not able to find the directory that you have configured in directory property and logs are not showing any permission denied issues.

If logs shows permission denied then you can change the nifi running user in bootstrap.conf and Once you make change in nifi properties then NiFi needs to restart to apply the changes (or) change the permissions on the directory that NiFi can have access.

NiFi ListHDFS cannot find directory, FileNotFoundException

1 Answers