Dump data from beeline in HDFS directory

0
votes

i am writing a bash script to export dynamic sql query into a hql file in HDFS directory.I am going to run this bash through oozie.

  1. sql_v= select 'create table table_name from user_tab_columns where ...;'

  2. beeline -u "$sql_v" > local_path

sql_v variable will store dynamic create table command which i want to store in a hql file in hdfs directory. If i run above 2 steps it runs fine because i am storing data in local path but instead of passing local_path i want to store sql in hdfs directory.Is there a way i can pass hdfs path instead of local_path like below but this doesn't work. Can i use any other command instead of beeline to achieve this ?

beeline -u "$sql_v" | hdfs dfs -appendToFile -

1
hadoophivehdfsbeeline
Why not use an INSERT INTO query? – OneCricketeer
INSERT into can store query in hdfs file ? how – vjrock99
Hive isn't a database... It's just a collection of HDFS files that you're querying. So inserting into a table (or creating a table from a selection) will generate HDFS files (though, not "hql files", whatever those are) – OneCricketeer
i only want to store the query in hql file and not the result of query – vjrock99
something like this :sql_v= select 'create table table_name from user_tab_columns where ...;' beeline -u "$sql_v" | hdfs dfs -appendToFile -destination – vjrock99

1 Answers

0
votes

If the goal is to write the output of beeline to hdfs file then below options should work fine since both commands will pipe the standard output of beeline to hadoop commands as input which is recognized by (-).

beeline -u beeline_connection_string .... -e "$sql_v" | hadoop fs -put - /user/userid/file.hql

beeline -u beeline_connection_string .... -e "$sql_v" | hadoop fs -appendToFile - /user/userid/file.hql

Note: 1. It's a little unclear based on your question and comments on why can't you use the suggestion given by @cricket_007 and why to go for a beeline in particular.

echo "$sql_v" > file.hql
hadoop fs -put file.hql /user/userid/file.hql

beeline -u beeline_connection_string .... -e "$sql_v" > file.hql 
hadoop fs -appendToFile file.hql /user/userid/file.hql

beeline -u beeline_connection_string .... -e "$sql_v" > file.hql 
hadoop fs -put file.hql /user/userid/file.hql
  1. If oozie shell action is used to run the bash script which containing the sql_v and beeline command, beeline needs to be present in the node where shell action will run if not you will face beeline not found an error. Refer: beeline-command-not-found-error