HDFS path to load data to Hive

1
votes

I am running hadoop as a single node distribution. Following the posts i moved a file to HDFS using 

hadoop fs -put <local path>    </usr/tmp/fileNAme.txt> .

Now I am trying to load the data from HDFS file to Hive table using the command below . Not able to find out what is the HDFS path relative to my local file system that i should be providing in the command below.

Load Command I am using from my java program to load the hive table is

LOAD DATA IN PATH ('HDFS PATH as it relates to my local File System???' ). All my attempts in giving the path including /usr/tmp/fileNAme.txt fails.

How do I resolve the full HDFS path?

3
hadoophive
What error you are getting?? And can you check if the file exists at the path you have specified using hadoop fs -ls /usr/tmp/ – Shivanand Pawar
if your data is already there in hadoop, why don't you create an external table on top it – Amar
When i run fs -ls i do see the file being in there . The problem is i am loading the data to hive from a Java program . static void createTable() Statement stmt) throws SQLException{ stmt.execute("LOAD DATA INPATH '/usr/local/Cellar/hadoop/hdfs/tmp/dfs/data/user/sriramvaradharajan/Acord_2_2013_01_INS.txt' INTO TABLE auto_claims_table"); See the path in there after the 'INPATH' . That path should be full HDFS path . If i specify the path i do in fs -ls it is not recognizing it. To the external table question even there i need to specify the location which is the path at the end of query. – sriram var

3 Answers

1
votes

Syntax is incorrect

 load data local inpath '/tmp/categories01.psv' overwrite into table categories;

You have to specify local inpath in the command.

1
votes

This command loads data from local file system

LOAD DATA LOCAL INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;

'LOCAL' signifies that the input file is on the local file system. If 'LOCAL' is omitted then it looks for the file in HDFS.

This command loads data from HDFS file system.

LOAD DATA INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;

Have a look into this article for more details.

0
votes

The syntax for loading file from hdfs into hive is

LOAD DATA INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;

Please clarify how do i resolve the full HDFS path .

the full hdfs path in your syntax would be 

hdfs://<namenode-hostname>:<port>/your/file/path