Where HDFS stores files locally by default?

Question

I am running hadoop with default configuration with one-node cluster, and would like to find where HDFS stores files locally.

Any ideas?

Thanks.

Answer 1

You need to look in your hdfs-default.xml configuration file for the dfs.data.dir setting. The default setting is: ${hadoop.tmp.dir}/dfs/data and note that the ${hadoop.tmp.dir} is actually in core-default.xml described here.

The configuration options are described here. The description for this setting is:

Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.

Answer 2

Seems like for the current version(2.7.1) the dir is

/tmp/hadoop-${user.name}/dfs/data

Based on dfs.datanode.data.dir, hadoop.tmp.dir setting from: http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/core-default.xml

Answer 3

As "more recent answer" and to clarify hadoop version numbers:

If you use Hadoop 1.2.1 (or something similar), @Binary Nerd's answer is still true.

But if you use Hadoop 2.1.0-beta (or something similar), you should read the configuration documentation here and the option you want to set is: dfs.datanode.data.dir

Answer 4

For hadoop 3.0.0, the hdfs root path is as given by the property "dfs.datanode.data.dir"

Answer 5

Run this in the cmd prompt, and you will get the HDFS location:

bin/hadoop fs -ls /