issues with sqoop and hive

0
votes

We are facing the following issues details as follows, please share your inputs.

1) Issue with --validate option in sqoop

if we run the sqoop command without creating a job for it, validate works. But if we create a job first, with validate option the validate doesn't seem to work.

works with

sqoop import --connect "DB connection" --username $USER --password-file $File_Path --warehouse-dir $TGT_DIR --as-textfile --fields-terminated by '|' --lines-teriminated-by '\n' --table emp_table -m 1 --outdir $HOME/javafiles --validate

Does not work with

sqoop job --create Job_import_emp import --connect "DB connection" --username $USER --password-file $File_Path --warehouse-dir $TGT_DIR --as-textfile --fields-terminated by '|' --lines-teriminated-by '\n' --table emp_table -m 1 --outdir $HOME/javafiles --validate

2) Issue with Hive import

If we are importing data for the first time in hive, it becomes imperative to create hive table ( hive internal), so we keep "--create-hive-table" in sqoop command.

Even thouhg if i keep "--create-hive-table" option, Is there any way to skip create table step in hive while importing, if the table is already exists.

Thanks Sheik

1
hivesqoop

1 Answers

0
votes

Sqoop allows --validate option only for sqoop import and sqoop export commands.

From the official Sqoop User guide, the validation has these limitations,

  • all-tables option
  • free-form query option
  • Data imported into Hive or HBase table
  • import with --where argument

No, the table check cannot be skip if --create-hive-table option is set, the job will fail if the target table exists.