How to pass column names having spaces to sqoop --map-column-java

Question

I have to import data using sqoop, my source column names are having spaces in between them, so while I am adding it in --map-column-java parameter getting the error.

Sample Sqoop import:

sqoop import --connect jdbc-con --username "user1" --query "select * from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java data col1=String, data col2=String, data col3=String --as-avrodatafile

Column names:

data col1,
data col2,
data col3

Error:

19/03/07 07:31:55 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping should be the form key=value[,key=value]* java.lang.IllegalArgumentException: Malformed mapping. Column mapping should be the form key=value[,key=value]* at org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1355) at org.apache.sqoop.SqoopOptions.setMapColumnJava(SqoopOptions.java:1375) at org.apache.sqoop.tool.BaseSqoopTool.applyCodeGenOptions(BaseSqoopTool.java:1363) at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011) at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435) at org.apache.sqoop.Sqoop.run(Sqoop.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Malformed mapping. Column mapping should be the form key=value[,key=value]*

This answer will help you. sqoop issue with space column name — Data Engineering Simplified
I tried with that by passing data col getting col not found error. — Prabhanj
Tried as below : sqoop import --connect jdbc-con --username "user1" --query "select * from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java "datacol1=String, datacol2=String, datacol3=String"--as-avrodatafile getting error: ERROR tool.ImportTool: Import failed: Cannot convert SQL type 2005 — Prabhanj
Tried passing all columns in --map-column-java , still throwing error: ERROR tool.ImportTool: Import failed: Cannot convert SQL type 2005 — Prabhanj
we have 3 columns of type 2005: Found COLNAME of type [2005, 2147483647, 0]. I see that the datatype for that COL in SQL server is nvarchar with max_length = -1 so i tried with map-col-java on those col to convert them to string. Still getting same error. — Prabhanj

Prabhanj Prabhanj · Accepted Answer · 2019-03-08T11:44:13

Able to resolve this issue:

1. Spaces issue: sqoop import --connect jdbc-con --username "user1" --query "select * from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java "data col1=String, data col2=String, data col3=String" --as-avrodatafile

2. ERROR tool.ImportTool: Import failed: Cannot convert SQL type 2005: 3 columns in source are having 2005 and nvarchar added them in --map-column-java resolved this issue

3. org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["null","long"]: 1****

This is causing due to using * in select query, so modified sqoop query as:

sqoop import --connect jdbc-con --username "user1" --query "select [col1,data col2,data col3] from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java "data col1=String, data col2=String, data col3=String" --as-avrodatafile

How to pass column names having spaces to sqoop --map-column-java

1 Answers