try to run airflow on databricks but got error

0
votes

I am trying to use airflow on databricks.

I have installed apache-airflow 1.10.6 from https://pypi.org/project/apache-airflow/.

I am using python3.6 on databricks.

But, I got error:

 import airflow

  ModuleNotFoundError: No module named 'werkzeug.wrappers.json'; 'werkzeug.wrappers' is not a package

I have tried the followings: Apache Airflow : airflow initdb results in "ImportError: No module named json"

Apache Airflow : airflow initdb throws ModuleNotFoundError: No module named 'werkzeug.wrappers.json'; 'werkzeug.wrappers' is not a package error

But, I still got the same problem.

Thanks

1
pythonazureairflowdatabricksazure-databricks

1 Answers

1
votes

Note: By default, "Airflow" and its dependency is not installed on the databricks.

You need to install the package explicitly.

Dependency installation: Using Databricks library utilities.

dbutils.library.installPyPI("Werkzeug")

enter image description here

You can install the packages in different methods.

Method1: Installing external packages using pip cmdlet.

Syntax: %sh /databricks/python3/bin/pip install <packagename>

%sh
/databricks/python3/bin/pip install apache-airflow

enter image description here

Method2: Using Databricks library utilities

Syntax: 

dbutils.library.installPyPI("pypipackage", version="version", repo="repo", extras="extras")
dbutils.library.restartPython()  # Removes Python state, but some libraries might not work without calling this function

To install apache-airflow using databricks library utilities use the below command.

dbutils.library.installPyPI("apache-airflow")

enter image description here

Method3: GUI Method

Go to Clusters => Select Cluster => Libraries => Install New => Library Source "PyPI" => Package "apache-airflow" => Install

enter image description here

Hope this helps. Do let us know if you any further queries.

Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.