How to use hive warehouse connector in HDP 2.6.5

1
votes

I have a requirement to read hive table from spark which is ACID enabled.

Spark by native doesn't support to read ORC file which is ACID enabled, only option is use spark jdbc.

We can also use hive warehouse connector to read files , can someone explain what is the steps to read using hive warehouse connector.

Is HWC only work in HDP 3 version.Kindly advise.

Spark version :2.3.0

HDP -2.6.5

1
apache-sparkhive

1 Answers

1
votes

Spark can read ORC file, check documentation on it here: https://spark.apache.org/docs/2.3.0/sql-programming-guide.html#orc-files

Here is a sample of code to read orc file:

spark.read.format("orc").load("example.orc")

HWC is made for HDP 3 version, as Hive and Spark catalogs are not compatible anymore in HDP 3, (Hive is in version 3, and Spark in version 2).

See documentation on it here: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/integrating-hive/content/hive_hivewarehouseconnector_for_handling_apache_spark_data.html