How to run DockerOperator on Airflow?

2
votes

I'm setting up a new airflow server using docker-compose and the airflow image puckel/docker-airflow. BashOperators and PythonOperator are working well, but when I'm trying to use DockerOperator I'm getting [/usr/local/airflow/dags/XXXXXXX.py] No module named 'docker'.

How do I need to change my docker-compose file in order to use docker inside my docker?

I tried to look for the error message on google, but none of the suggested solution worked.

Here is the docker-compose for the WebServer I'm using

webserver:
    image: puckel/docker-airflow:1.10.1
    build:
      context: https://github.com/puckel/docker-airflow.git#1.10.1
      dockerfile: Dockerfile
      args:
        AIRFLOW_DEPS: gcp_api,s3,docker        
    restart: always
    depends_on:
      - postgres
    environment:
      - LOAD_EX=n
      - EXECUTOR=Local
      - FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo=
    volumes:
      - ./LOCALPATHTODAGS/dags:/usr/local/airflow/dags
    ports:
      - "8080:8080"
    command: webserver

My end goal is to be able to trigger a docker container of a local image from my airflow pipelines.

2
dockerairflow

2 Answers

1
votes

There are two methods you could use:

  1. Add before_script to your current current docker-compose:
webserver:
   (...)
   before_script:
   - pip install docker
  1. Create a Dockerfile which uses the one from puckel and install the required package in there.
0
votes

'docker' is actually also a Python module that is probably imported in the source code of the DockerOperator. Assuming that you install your dependencies in a requirements.txt file from within your Dockerfile, you could add docker==4.1.0 into your requirements.txt file which should be in the same directory as your Dockerfile.

So, in your Dockerfile, you need:

COPY requirements.txt requirements.txt
USER root
RUN pip install -r requirements.txt

and in your requirements file, you need:

docker==4.1.0  # Other versions probably also work