Apache spark jdbc connect to apache drill error

1
votes

I am sending query to apache drill from apache spark. I am getting the following error:

java.sql.SQLException: Failed to create prepared statement: PARSE ERROR: Encountered "\"" at line 1, column 23.

When traced, I found I need to write a custom sql dialect. The problem I do not find any examples for pyspark. All the examples are for scala or java. Any help is highly appreciated.!

Here is the pyspark code :

`dataframe_mysql = spark.read.format("jdbc").option("url", "jdbc:drill:zk=ip:2181;schema=dfs").option("driver","org.apache.drill.jdbc.Driver").option("dbtable","dfs.`/user/titanic_data/test.csv`").load()`
2
sqlapache-sparkpysparkapache-drill

2 Answers

1
votes

Looks like you have used a double quote in your SQL query (please share your SQL).

By default Drill uses back tick for quoting identifiers - `
But you can change it by setting the system/session option (when you are already connected to Drill by JDBC for example) or you can specify it in JDBC connecting string. You can find more information here: https://drill.apache.org/docs/lexical-structure/#identifier-quotes

1
votes

I navigated to the drill web ui and updated the planner.parser.quoting_identifiers parameter to ". Then I edited my query as below:

dataframe_mysql = spark.read.format("jdbc").option("url", "jdbc:drill:zk=ip:2181;schema=dfs;").option("driver","org.apache.drill.jdbc.Driver").option("dbtable","dfs.\"/user/titanic_data/test.csv\"").load()

And it worked like charm!