Hive table on delta lake

4
votes

I am new to spark & delta lake. I am creating hive table on top of delta table. I have necessary jars delta-core-shaded-assembly_2.11-0.1.0.jar, hive-delta_2.11-0.1.0.jar; in hive class path. Set following properties.

SET hive.input.format=io.delta.hive.HiveInputFormat;
SET hive.tez.input.format=io.delta.hive.HiveInputFormat;

But while creating table

CREATE EXTERNAL TABLE hive_table(col1 INT, col2 STRING) STORED BY 'io.delta.hive.DeltaStorageHandler' LOCATION '/path/to/my/delta/table'

Getting following exception:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.spark.network.util.JavaUtils.byteStringAs(Ljava/lang/String;Lorg/apache/spark/network/util/ByteUnit;)J

Schema of both table matches. Stack details: Spark: 2.4.4 Hive: 1.2.1

Any help is much appreciated. Thanks in advance.

1
apache-sparkhivedelta-lake

1 Answers

1
votes

Hive with delta lake only supporting with spark 3.0 and delta version 0.7.0.

<dependency>
  <groupId>io.delta</groupId>
  <artifactId>delta-core_2.12</artifactId>
  <version>0.7.0</version>
</dependency>