Apache Airflow - BigQuery Operator Issue

0
votes

I am using the following library:

airflow.contrib.operators.bigquery_operator

On the Airflow UI, it throws the following error:

Broken DAG: [/home/vagrant/airflow/dags/data/gcp_bq_dag.py] (setuptools 28.8.0 (/home/vagrant/.sandbox/lib/python3.5/site-packages), Requirement.parse('setuptools>=34.0.0'), {'google-api-core'})

I have installed the setuptools library and the version I have is 49.2.0.

Name: setuptools
Version: 49.2.0
Summary: Easily download, build, install, upgrade, and uninstall Python packages
Home-page: https://github.com/pypa/setuptools
Author: Python Packaging Authority
Author-email: [email protected]
License: UNKNOWN
Location: /home/vagrant/.sandbox/lib/python3.5/site-packages
Requires: 
Required-by: zope.deprecation, python-daemon, protobuf, kiwisolver, jsonschema, ipython, google-auth, google-api-core

Can you please suggest what is going wrong there?

2
pythongoogle-bigqueryairflowairflow-operator
Are you using Airflow on-premise or in Cloud Composer? - aga
I'm using Airflow on-premise - S.Nori

2 Answers

1
votes

Ok, the error it was showing on the airflow UI seems to be a redherring:

Broken DAG: [/home/vagrant/airflow/dags/data/gcp_bq_dag.py] (setuptools 28.8.0 (/home/vagrant/.sandbox/lib/python3.5/site-packages), Requirement.parse('setuptools>=34.0.0'), {'google-api-core'})

The actual issue is that it was missing pandas_gbq package. This was logged in the webserver logs. I've installed pandas_gbq and it resolved the issue.

0
votes

This looks like a setup issue around your machine and the various environments of execution. It would be worthwhile taking an a more in depth look with someone technical and familiar with your situation.

Based on reading the error alone. And this could be wrong. But, essentially, the two different contexts are discovering a different version of the installed package. This happens quite often with Python and projects. The version of python that Airflow knows about can only find setuptools 28.0.0. That's just incompatible with the requirements of the DAG.

Edit

You might want to try installing the package against the user pip install --user setuptools