Import Sqoop column names issue

Question

I have a question on Kylo and Nifi. The version of Kylo used is 0.10.1 The version of Nifi used is 1.6.0

When we create a feed for database ingest (using database as source), in the Additional Options step there is no provision to enter the source table column names.

However, in Nifi side, we use an Import Sqoop processor which has a mandatory field called Source Fields and it requires that the columns be entered, separated by commas. If it is not done, we get an error:

ERROR tool.ImportTool: Imported Failed: We found column without column name. Please verify that you've entered all column names in your query if using free form query import (consider adding clause AS if you're using column transformation)

For our requirement, we want Import Sqoop to take all the columns from the table automatically into this property without manual intervention at Nifi level. Is there any option to include all columns of a database table in the background automatically? Or is there any other possibility of giving this value in UpdateAttribute processor?

Answer 1

As mentioned in the Comments, ImportSqoop is not a not a normal Nifi processor. This does not have to be problem, but will mean it is probably not possible to troubleshoot the problem without involving the creator.

Also, though I am still debating whether Nifi on Sqoop is an antipattern, it is certainly not necessary. Please look into the standard options first:

1. Standard way to get data into Nifi from tables is with standard processors such as ExecuteSQL
2. If that doesn't suffice, the standard way to use Sqoop (a batch tool) is with a batch scheduler, such as Oozie or Airflow

This thread may take away further doubts on point 1: http://apache-nifi.1125220.n5.nabble.com/Sqoop-Support-in-NIFI-td5653.html