Airflow - ModuleNotFoundError: No module named 'kubernetes'

5
votes

I installed Python, Docker on my machine and am trying to import the from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator but when I connect the docker, I get the message that the module does not exist. I have already done the pip install apache-airflow[kubernetes] and I still have the same error. Is there a specific machine location that I should check if the library is actually installed? What can I do to solve this?

enter image description here

from airflow import DAG
from datetime import datetime, timedelta
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
import logging
import os
from airflow.utils.helpers import parse_template_string

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.utcnow(),
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG(
    'kubernetes_sample', default_args=default_args, schedule_interval=timedelta(minutes=10))


start = DummyOperator(task_id='run_this_first', dag=dag)

passing = KubernetesPodOperator(namespace='default',
                          image="Python:3.6",
                          cmds=["Python","-c"],
                          arguments=["print('hello world')"],
                          labels={"foo": "bar"},
                          name="passing-test",
                          task_id="passing-task",
                          get_logs=True,
                          dag=dag
                          )

failing = KubernetesPodOperator(namespace='default',
                          image="ubuntu:1604",
                          cmds=["Python","-c"],
                          arguments=["print('hello world')"],
                          labels={"foo": "bar"},
                          name="fail",
                          task_id="failing-task",
                          get_logs=True,
                          dag=dag
                          )

passing.set_upstream(start)
failing.set_upstream(start)

webserver_1 | Traceback (most recent call last): webserver_1 |
File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 377, in process_file webserver_1 | m = imp.load_source(mod_name, filepath) webserver_1 | File "/usr/local/lib/python3.6/imp.py", line 172, in load_source webserver_1 | module = _load(spec) webserver_1 | File "", line 684, in _load webserver_1 | File "", line 665, in _load_unlocked webserver_1 | File "", line 678, in exec_module webserver_1 | File "", line 219, in _call_with_frames_removed webserver_1 | File "/usr/local/airflow/dags/example_airflow.py", line 3, in webserver_1 | from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator webserver_1 | File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 21, in webserver_1 | from airflow.contrib.kubernetes import kube_client, pod_generator, pod_launcher webserver_1 | File "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 25, in webserver_1 | from kubernetes import watch, client webserver_1 | ModuleNotFoundError: No module named 'kubernetes'

3
pythonkubernetespipairflow
Try: pip install airflow['kubernetes'] – AC at CA
@FelipeFB - did u resolve? – tooptoop4
I just restart like @Kalil posted! – Felipe FB
i got same error upgrading from 1.10.3 to 1.10.4 but that pip suggestion does not fix it :( – tooptoop4

3 Answers

5
votes

Run the following 

pip install apache-airflow[kubernetes]

Restart Airflow webserver and scheduler after that.

2
votes

It tries to import module named kubernetes. Try that:

pip install kubernetes
1
votes

try any of these imports. one of these got deprecated, so you might have a version that is no longer supported.

    #from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
    from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator