Bucketby Property with Spark saveastable method taking spark 'default' database instead of hive database: HDP 3.0

0
votes

I'm saving Spark DataFrame using saveAsTable method and writing below code.

    val options = Map("path" -> hiveTablePath)
    df.write.format("orc")
    .partitionBy("partitioncolumn")
    .options(options)
    .mode(SaveMode.Append)
    .saveAsTable(hiveTable)

It's working fine and i am able to see data in hive table. but when I'm using one more property bucketby(5,bucketted_column)

    df.write.format("orc")
    .partitionBy("partitioncolumn")
    .bucketby(5,bucketted_column)
    .options(options)
    .mode(SaveMode.Append)
    .saveAsTable(hiveTable)

It's trying to save it in spark 'default' database instead of hive database.

can someone please suggest me why bucketby(5,bucketted_column) is not working with saveAsTable.

    Note: Framework: HDP 3.0
    Spark : 2.1
2
apache-sparkapache-spark-sqlhdp
defaut is also Hive if I remember correctly - thebluephantom

2 Answers

0
votes

You can try to add this parameter:

.option("dbtable", "schema.tablename")
0
votes

df.write.mode...saveAsTable("database.table") should do the trick.

Except that the bucketBy format cannot be read via Hive, Hue, Impala. Currently not supported.

Not sure what you mean by spark db, I think you mean Spark metastore?