Error while using truncateTable method of JDBCUtils in PostgreTable using Spark

0
votes

I am trying to truncate table postgre_table from Spark using JDBCUtils, but it is throwing below error

< console>:71: error: value truncateTable is not a member of object org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils

val trucate_table = JdbcUtils.truncateTable()

I am using the below code:

import org.apache.spark.sql.execution.datasources.jdbc._
import java.sql.DriverManager
import java.sql.Connection

val connection : Connection = DriverManager.getConnection(postgres_host + postgres_database,postgres_username,postgres_password)

val table_existing = JdbcUtils.tableExists(connection, postgres_host + postgres_database, postgre_table)
JdbcUtils.truncateTable(connection, postgres_host + postgres_database, postgre_table)

I am able to drop the table but not truncate it. I can see truncateTable method in https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala

Please suggest a solution and how to use it in databricks.

1
postgresqlscalaapache-sparkjdbcdatabricks

1 Answers

0
votes

Looks like your compile time and runtime spark libraries are of different versions. Please make sure the runtime version match compile time version of spark. Seems the method is available from the version 2.1 alone.

available from this release :

https://github.com/apache/spark/blob/branch-2.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala