Incremental loads in Sqoop

0
votes

I have a table in Teradata which is loaded with new data on daily basis.

I need to import this data to Hive. I'm trying to use Sqoop but how should I do incremental load using Sqoop?

I checked incremental load options available in Sqoop

--check-col

This options expects only numeric and unique data whereas my Teradata table is not having such column

I'm using date column inn --spli-by

How can I do incremental imports to Hive in this case?

Please suggest

2
hadoophiveteradatasqoop

2 Answers

2
votes

If you use --incremental lastmodified mode then your --check-column is a timestamp that does not need to be numeric or unique.

See: Sqoop incremental imports.

0
votes

use one more "--append " command after the check column so that , the new data will not be replaced and you can see it saves in a new file in hdfs without replacing the old file.