Writing Data into Hive Transactional table

Question

I am trying to write data into Hive transactional table using spark. Following is the sample code that I have used to insert data

dataSet.write().format("orc")
  .partitionBy("column1")
  .bucketBy(2,"column2")
  .insertInto("table");

but unfortunately getting the following error while running the application.

org.apache.spark.sql.AnalysisException: 'insertInto' does not support bucketBy right now;

The spark and hive versions that I have used is 2.4 and 3.1. Have googled a lot but didn't find any solution. I am pretty much new to hive Any help would be appreciated.

Can I ask you to accept the answer? It is a clear answer to your question I think. Or do you disagree? — thebluephantom
Can we expect this feature to be added in future releases of spark any Idea? — Sandhya

thebluephantom thebluephantom · Accepted Answer · 2019-12-16T14:39:35

https://issues.apache.org/jira/browse/SPARK-15348 states clearly that Spark does not allow HIVE ORC ACID processing, currently. A pity, but not possible.

You need to write Hive scripts with TEZ or MR as underlying engine for Hive.

Writing Data into Hive Transactional table

1 Answers