Issue connecting to Databricks table from Azure Data Factory using the Spark odbc connector

1
votes

​We have managed to get a valid connection from Azure Data Factory towards our Azure Databricks cluster using the Spark (odbc) connector. In the list of tables we do get the expected list, but when querying a specific table we get an exception.

ERROR [HY000] [Microsoft][Hardy] (35) Error from server: error code: '0' error message: 'com.databricks.backend.daemon.data.common.InvalidMountException: Error while using path xxxx for resolving path xxxx within mount at '/mnt/xxxx'.'.. Activity ID:050ac7b5-3e3f-4c8f-bcd1-106b158231f3

In our case the Databrick tables and mounted parquet files stored in Azure Data Lake 2, this is related to the above exception. Any suggestions how to solve this issue?

Ps. the same error appaers when connectin from Power BI desktop.

Thanks Bart

1
powerbiparquetazure-data-factoryazure-data-lakeazure-databricks
How did you connect to the gen2 lake? If you used spark.conf settings it's probable that the are not available to this session. If you open a sql notebook and select from the table does it work? – simon_dmorias
Hi @simon_dmorias, if I open a SQL notebook I am able to query the tables just fine: use databasename; select * from tablename I'll will inspect our gen2 lake connection setup and provide feedback later on, a colleague did this implementation. – BTV
Hi @simon_dmorias, our datalake gen 2 connection is setup as: configs = {"fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider", "fs.azure.account.oauth2.client.id": "xxx", "fs.azure.account.oauth2.client.secret": "xxx", "fs.azure.account.oauth2.client.endpoint": "login.microsoftonline.com/xxx/oauth2/token"} dbutils.fs.mount( source = "abfss://[email protected]/", mount_point = "/mnt/datalake", extra_configs = configs) dbutils.fs.ls("dbfs:/mnt/datalake") – BTV
I heard that the gen2 datalake integration via Databricks odbc connection might still be in development. Can anyone confirm this ? – BTV
did you find any solution for this? – chathux

1 Answers

0
votes

In your configuration to mount the lake can you add this setting:

"fs.azure.createRemoteFileSystemDuringInitialization": "true"

I haven't tried your exact scenario - however this solved a similar problem for me using Databricks-Connect.