Why does Databricks Connect Test not work on Mac?

0
votes

I've gone through the documentation for configuring databricks-connect but still get the error below when running databricks-connect test

Errors From Terminal

java.lang.NoSuchMethodError: org.apache.spark.internal.config.package$.STRING_REDACTION_PATTERN()Lorg/apache/spark/internal/config/ConfigEntry;
error: not found: value spark import spark.implicits._
error: not found: value spark import spark.sql
error: not found: value spark spark.range(100).reduce(_ + _)
Scala command failed to produce correct result

Steps Used to Setup Databricks Connect

  • Created a cluster on Databricks with Runtime of 5.3 and Python 2
  • Set Spark Config to spark.databricks.service.server.enabled true (Restarted)
  • Created Python project using Pyenv and Python 2.7.15
  • Removed pyspark pip uninstall pyspark
  • Installed Databricks Connect pip install -U databricks-connect==5.3.*
  • Configured databricks connect databricks-connect configure
    • Databricks Host: https://<account>.cloud.databricks.com
    • Databricks Token: <secret_token>
    • Cluster ID: <cluster_token>
    • Org ID: <left_blank>
    • Port: 15001

What is causing this error and is there a possible fix?

Used Resources

1
apache-sparkpysparkdatabricks

1 Answers

0
votes

After further investigation, I found another StackOverflow post that discusses how the Java error hints to a version mismatch. I downgraded my Databricks cluster runtime to 5.1 and updated databricks-connect by running:

pip install -U databricks-connect==5.1.*

To validate, I ran:

databricks-connect test

Important items to remember:

  1. Uninstall pyspark first because installing databricks-connect will install a slightly modified version of pyspark.
  2. The databricks-connect version must match the cluster's DBR version.
  3. The Python version of your local environment must match the Python version of your cluster. [E.g. 2.7, 3.5, 3.6, 3.7,...]