Spark JDBC predicate pushdown disable doesn't work

0
votes

Spark JDBC predicate pushdown disable doesn't work.

sparkVersion = "2.4.2"

tried:

    properties.setProperty(JDBCOptions.JDBC_PUSHDOWN_PREDICATE, "false")
    val dataFrame = sqlContext.read.jdbc(endpoint.connectionString, dbTableName, strictPredicates.get, properties)
    dataFrame.createOrReplaceTempView(tableName)

Predicates (others than strictPredicates) are still pushed down to 3rd party database. In this case its Oracle and it can't handle some of them.

How could I query Oracle from Spark without pushing any predicates down the line and not jumping out of DataFrame context to collect data ?

1
scalaapache-spark

1 Answers

0
votes

Only option I am finding is to use Cache(), which in turn forces the query to evaluate in current state