cloudera impala jdbc query doesn't see array<string> Hive column

Question

I have a table in Hive that has the following structure:

> describe volatility2;
Query: describe volatility2
+------------------+---------------+---------+
| name             | type          | comment |
+------------------+---------------+---------+
| version          | int           |         |
| unmappedmkfindex | int           |         |
| mfvol            | array<string> |         |
+------------------+---------------+---------+

It was created by Spark HiveContext code by using a DataFrame API like this:

val volDF = hc.createDataFrame(volRDD)
volDF.saveAsTable(volName)

which carried over the RDD structure that was defined in the schema:

def schemaVolatility: StructType = StructType(
    StructField("Version", IntegerType, false) ::
    StructField("UnMappedMKFIndex", IntegerType, false) ::
    StructField("MFVol", DataTypes.createArrayType(StringType), true) :: Nil)

However, when I'm trying to select from this table using the latest JDBC Impala driver the last column is not visible to it. My query is very simple - trying to print the data to the console - exactly like in the example code provided by the driver download:

String sqlStatement = "select * from default.volatility2";
Class.forName(jdbcDriverName);
con = DriverManager.getConnection(connectionUrl);
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery(sqlStatement);
System.out.println("\n== Begin Query Results ======================");

ResultSetMetaData metadata = rs.getMetaData();
for (int i=1; i<=metadata.getColumnCount(); i++) {
    System.out.println(rs.getMetaData().getColumnName(i)+":"+rs.getMetaData().getColumnTypeName(i));
}
System.out.println("== End Query Results =======================\n\n");

The console output it this:

== Begin Query Results ======================
version:version
unmappedmkfindex:unmappedmkfindex
== End Query Results =======================

Is it a driver bug or I'm missing something?

Michael D Michael D · Accepted Answer · 2016-05-31T20:40:39

I found the answer to my own question. Posting it here so it may help others and save time in searching. Apparently Impala lately introduced the so called "complex types" support to their SQL that include array among others. The link to the document is this:

http://www.cloudera.com/documentation/enterprise/5-5-x/topics/impala_complex_types.html#complex_types_using

According to this what I had to do is change the query to look like this:

select version, unmappedmkfindex, mfvol.ITEM from volatility2, volatility2.mfvol

and I got the right expected results back

cloudera impala jdbc query doesn't see array<string> Hive column

1 Answers