Celery worker service ignoring broker url

Question

I'm trying to use Celery in my django app with redis as broker.

In my settings file I set CELERY_BROKER_URL='redis://redis:6379' and CELERY_RESULT_BACKEND='redis://redis:6379' And my docker-compose file looks like this

web: 
    build: 
      context: ./web/
      dockerfile: Dockerfile
    image: &web web
    env_file:
      - .env
    command: "gunicorn web.wsgi:application -w 2 -b :4000"
    volumes:
      - ./web/:/web
    expose:
      - "4000"
    depends_on:
      - db
      - redis
      - worker
      - beat

db:
    build:
      context: ./database/
      dockerfile: Dockerfile
    volumes:
      - data:/var/lib/postgresql/data
    env_file:
      - .env
    expose:
      - "5432"

redis:
    build:
      context: ./cache/
      dockerfile: Dockerfile
    expose:
      - "6379"

worker:
    build:
      context: ./web/
      dockerfile: Dockerfile
    image: *web
    command: "celery -A web worker -l debug"
    ports: []
    depends_on:
      - redis
      - db

beat:
    build:
      context: ./web/
      dockerfile: Dockerfile
    image: *web
    command: "celery -A web beat -l info"
    ports: []
    depends_on:
      - redis
      - db

When I run docker-compose up the beat service starts well but the workerfails with the error

consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused

Somehow the worker service is trying to use rabbitmq as broker

Please someone help shed some lights on what I'm doing wrong here.

@TaranjeetSingh I ended up using rabbitmq as broker and redis as a caching service — Hippolyte Fayol

targhs targhs · Accepted Answer · 2020-09-20T06:11:01

I got it fixed myself.

We need to replace the CELERY_BROKER_URL in the settings file as follows

# Celery
CELERY_BROKER_URL = 'redis://broker/0'

Above, the broker refers to the name of the service in docker-compose.yml.

Earlier it was something like this

CELERY_BROKER_URL = 'redis://localhost:6379/0'

My docker-compose.yml file is as follows

#.... other services here 
broker:
    image: redis:alpine
    expose: 
        - "6379"
celery:
    build: ./myapp
    command: celery -A myapp worker -l info
    env_file:
        - .env
    volumes:
        - ./myapp:/opt/myapp
    depends_on:
        - broker
        - db

Celery worker service ignoring broker url

1 Answers