Sqoop Import Command Directories

Question

Env: CDH Tool: Sqoop

Version: Sqoop 1.4.6-cdh5.8.0

Objective: Import table from MySQL database

Create hive table with a subset of source data (e.g order_status = 'CLOSED')
Reimport more data in the same directory using order_status not in ('CLOSED')

Results: 1. Objective 1 complete using the command

sqoop import --connect jdbc:mysql://xxx:000/xxxx_db
             --username=xxxx_dba --P
             --warehouse-dir=/user/hive/warehouse/hex.db/
             -m 1
             --table orders --compression-codec=snappy
             --hive-import --as-textfile --create-hive-table
             --hive-table closed_orders
             --hive-overwrite
             --where "order_status='CLOSED'"
             --compress
             --columns "order_id, order_customer_id, order_status"

Creates the directory /user/hive/warehouse/hex.db/closed_orders with a data file and a hive table with "CLOSED" Orders.

I am trying to re import more data - this time order_status not in ('CLOSED') -- This time not creating a hive table and just importing the order_status != 'CLOSED' into different directory (open_orders).

Issue: It creates a directory /user/hive/warehouse/hex.db/open_orders/orders/. 2.a How can import the file into the directory /user/hive/warehouse/hex.db/open_orders?

2.b How can we import the subset date of order_status != 'CLOSED' ie. open orders into the same directory created in the step 1 ie. /user/hive/warehouse/hex.db/closed_orders ?

Command used for step 2:

sqoop import --connect jdbc:mysql://xxxx:0000/retail_db
             --username=xxxx_dba --P --warehouse-dir=/user/hive/warehouse/hex.db/ -m 1 
             --table orders --compression-codec=snappy  --hive-import
             --as-textfile --hive-table open_orders
             --where "order_status not in ('CLOSED')"
             --compress --columns "order_id, order_customer_id, order_status"

2.3 Error with --append command where in I am trying to import the open orders into the directory created from the step 1 /user/hive/warehouse/hex.db/closed_orders

17/04/15 14:24:22 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
17/04/15 14:24:22 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
Append mode for hive imports is not  yet supported. Please remove the parameter --append-mode

Pushkr Pushkr · Accepted Answer · 2017-04-16T04:43:15

when you doing #2 i.e. reimporting data, use

--target-dir /user/hive/warehouse/hex.db/open_orders

instead of

warehouse-dir

Sqoop Import Command Directories

Version: Sqoop 1.4.6-cdh5.8.0

1 Answers