I have the following setup
Databricks Cluster:
8.4 ML (includes Apache Spark 3.1.2, Scala 2.12)
Client side:
databricks==0.2
databricks-cli==0.14.3
databricks-connect==8.1.10 (The latest version up to date 20210813)
When I run databricks-connect test
to test the connection, I got this error message
java.lang.IllegalArgumentException:
The cluster is running server version `dbr-8.4` but this client only supports Set(dbr-8.1).
You can find a list of client releases at https://pypi.org/project/databricks-connect/#history, and install the right client version with `pip install -U databricks-connect==<version>`.
For example, to install the latest 5.1 release, use `pip install -U databricks-connect==5.1.*`.
To ignore this error and continue, set DEBUG_IGNORE_VERSION_MISMATCH=1.
According to the error message, I understand there is a compatibility issue with the client and server versions. However, I am already using the latest client.
- Does it mean the client doesn't add support for server version 8.4?
- Is there any way to circumvent this issue?