Where is the physical path of HDFS system on disk?

Question

I'm a beginner in Hadoop and HBase, I downloaded an example project about HBase for learning, and it seems to use MR to import existed data to HBase. This example puts the .txt data file under this path on HDFS: /user/root/uid_details.txt , and when clicking the 'Import' button, the data in uid_details.txt should be imported to HBase.

The problem is, I do not know the actual physical path of /user/root/ on my disk, so I do not know where the uid_details.txt should be placed on my disk. My core-site.xml and hdfs-site.xml : Screenshots of core-site.xml and hdfs-site.xml

Answer 1

The dfs.datanode.data.dir configuration in the hdfs-site.xml tells you physical location of the files (detailed description of hdfs-site.xml configurations can be found here.

So in your particular case, the files should be located in the /usr/local/hadoop_dfs/data directory.

Answer 2

The data in hadoop(hdfs) lies in the local directory location of dfs.datanode.data.dir(present in hdfs-site.xml). But this data won't be accessible to you directly(i.e. you may not cd into the directory and see it).

The hbase data lies in HBASE root directory mentioned in hbase.rootdir. You may check the location of the base directory in the hbase-site.xml.

Answer 3

I guess you are running all these commands in your local VM ( where hadoop and hbase is already installed ).

Try running the following command to find out the location of the hdfs dir.

hdfs dfs -ls /

It should list all the dirs, in case if you want to create some dirs, you can create also.

Then you can place your file accordingly and start import process.

Refer below link for more information

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html

Answer 4

You need to use the hdfs dfs / hadoop fs (or another hdfs client) to put the files into the dfs - the location of the HDFS blocks on disk will not help you to add files to the HDFS filesystem - eg

hdfs dfs -put /path/to/uid_details.txt /user/root/uid_details.txt

or

hdfs dfs -put /path/to/uid_details.txt hdfs://namenodeaddress/user/root/uid_details.txt