How can i see the incremental updated value in sqoop job?

Question

Below is the sqoop job i have created

sqoop job --create sqoop_incremental_job -- import --connect jdbc:mysql://quickstart.cloudera:3306/retail_db --username retail_dba --password cloudera --table categories --target-dir /user/cloudera/sqoop_incremental_job_categories --incremental append --check-column category_id --last-value 10 -m 1

For example: The last value was given as 10

Now i ran sqoop job for 2 times by adding the new records in mysql each time, So that the incremental would perform append to the existing data in HDFS.

The question is how the last-value is getting updated each time when i run job using sqoop job --exec sqoop_incremental_job and how can i see the updated value each time when i run the job???

Answer 1

As per sqoop docs:

If an incremental import is run from a saved job, this value will be retained in the saved job. Subsequent runs of sqoop job --exec someIncrementalJob will continue to import only newer rows than those previously imported.

For seeing last -value,

If an incremental import is run from the command line, the value which should be specified as --last-value in a subsequent incremental import will be printed to the screen for your reference.

Answer 2

Alternatively if the logs are missed. Use the below command

sqoop job --show <jobname> - It will list all the properties of the job

Refer to

incremental.last.value - This will contain the latest value of incremental job performed.

It will be updated each time when we run the job via sqoop job --exec <jobname>