In Hadoop what is stored in dfs.data.dir (in hdfs-site.xml parameter) & the director we create using dfs -mkdir

0
votes

I created a directory as below and create hive table using below syntax

a) 1) dfs -mkdir /app/hadoop/warehouse/

2) Create table employee ( empid int, name string , surname string) row format delimited fields terminated by ‘,’ location ‘/app/hadoop/warehouse/’;

3) load data inpath ‘/input file location .txt’ overwrite into table employee;

if my understanding is correct , all the blocks are stored under /app/hadoop/warehouse/ directory. Please confirm ?

b) what is the purpose of dfs.data.dir parameter in hdfs-site.xml , for the directories we mention here?

c) What ever the directory we create as the part of step a) should be mentioned in Step b) ?

I appreciate , for your clarification

Thanks Naveen

1
hadoophdfs

1 Answers

2
votes

1) Check the Apache documentation here for more details on dfs.data.dir and other properties.

2) File system is an abstraction of raw data, so the blocks are not stored in the /app/hadoop/warehouse/ folder. The blocks are stored in the DN's as specified by the dfs.data.dir property in the hdfs-site.xml.

3) In Hive the logical table should be associated with the data in HDFS. So, the folder name in HDFS should be specified while creating the table.