Docker-compose: bind Celery to Postgres database

Question

My Docker app runs with Flask as backend and Celery as an asyncronous task manager. Task results are then dumped in a SQLalchemy database owned by Postgres.

However, I am not being able to make Celery interact with Postgres.

Configuration:

docker-compose-dev.yml

  web:
    build:
      context: ./services/web
      dockerfile: Dockerfile-dev
    volumes:
      - './services/web:/usr/src/app' 
    ports:
      - 5001:5000
    environment:
      - FLASK_ENV=development
      - APP_SETTINGS=project.config.DevelopmentConfig
      - DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev 
      - DATABASE_TEST_URL=postgres://postgres:postgres@web-db:5432/web_test
      - SECRET_KEY=my_precious  
    depends_on:  
      - web-db
      - redis

  web-db:  
    build:
      context: ./services/web/project/db
      dockerfile: Dockerfile
    ports:
      - 5435:5432
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres

  celery:
    image: dev3_web
    restart: always
    volumes:
     - ./services/web:/usr/src/app
     - ./services/web/celery_logs:/usr/src/app/celery_logs

    command: celery worker -A celery_worker.celery --loglevel=DEBUG --logfile=celery_logs/celery.log -Q cache
    environment:
     - CELERY_BROKER=redis://redis:6379/0
     - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
     - web
     - redis
    links:
     - redis:redis
     - web-db

  redis:
    image: redis:5.0.3-alpine
    restart: always
    expose:
      - '6379'
    ports:
      - '6379:6379'

  monitor:
    image: dev3_web
    ports:
      - 5555:5555
    command:  flower -A celery_worker.celery --port=5555 --broker=redis://redis:6379/0
    depends_on:
      - web
      - redis

Log:

celery_1| Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".

How do I bind Celery tasks to my Postgres database?

Hi data_garden. I wonder if we can edit this again? I edited this because the usages of the quote block here are not quotes, and some of the usages of the inline code formatting are not code. These format types have a specific semantic meaning just like HTML, and it is preferable not to misuse them as general highlighters. This is particularly the case if you have a long tail of posts using the same approach. — halfer
(Things like Dockerfile and postgres are OK in code formatting, as file names and Unix users are frequently rendered to the console. It is also fine to use code formatting for console I/O. However Celery, Docker, Flask, etc are proper nouns, and so should either be rendered with an initial capital (or with special caps rules where appropriate, such as camel-case or all-caps, depending on how each name is normally written)). — halfer
The mis-spelling of "database" has also been re-added to the title. — halfer
Thank you, data_garden! If you could also abide by those conventions, volunteer editors would appreciate it - they may be disinclined to improve the readability of your earlier posts if new ones would be created in the future :-). — halfer

Allen Allen · Accepted Answer · 2019-04-28T19:29:53

celery_1   | Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".

suggests that the error from your celery container, which missing the environment valuable setup, you might need DATABASE_URL setup for it which created based on dev3_web.

changes need apply to your docker-compose-dev.yml:

  celery:
    image: dev3_web
    restart: always
    volumes:
     - ./services/web:/usr/src/app
     - ./services/web/celery_logs:/usr/src/app/celery_logs

    command: celery worker -A celery_worker.celery --loglevel=DEBUG --logfile=celery_logs/celery.log -Q cache
    environment:
     - CELERY_BROKER=redis://redis:6379/0
     - CELERY_RESULT_BACKEND=redis://redis:6379/0
     - APP_SETTINGS=project.config.DevelopmentConfig
     - DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev 
    depends_on:
     - web
     - redis
    links:
     - redis:redis
     - web-db

Docker-compose: bind Celery to Postgres database

1 Answers