Spark Jdbc connection JDBCOptions

2
votes

I am trying to use the method savetable in the Spark JdbcUtils

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala

The defination of the method is the following. It accepts the JDBCOptions as one of the parameter. 

def saveTable(
      df: DataFrame,
      tableSchema: Option[StructType],
      isCaseSensitive: Boolean,
      options: JDBCOptions)

Following is the class of the JDBCOptions

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala

While i am initializing the JDBCOptions with the url,transaction isolation level etc it is throwing errors.

Can you provide insights in initialization of the JDBCOptions

 val options :JDBCOptions = Map("url" -> "JDBCUrl")
Error Type Map doesnt conform to be JDBCOption Type

How to initialize with correct type ?

1
scalaapache-sparkapache-spark-sqlspark-dataframe
Providing the error message would help. – Aydin K.
@AydinK. I am not sure how to initialize it ? – loneStar
Have a look here: docs.databricks.com/spark/latest/data-sources/… – Aydin K.
Or here (chapter „Running Jdbc to other Databases“): spark.apache.org/docs/latest/… – Aydin K.
@AydinK. I looked into them. They are using using Options but i am using the JDBCOptions. Both are different. – loneStar

1 Answers

3
votes

You need to wrap the Map in a new JDBCOptions(...) call

// url and dbtable are required
val options:JDBCOptions = new JDBCOptions(Map("url" -> "JDBCUrl", "dbtable" -> "foo"))