Flask+Celery as a Daemon

3
votes

I try to lean Python Flask and want to use celery. The distributed task works fine, but now I would like to config it as Daemon as explained in the celery docs. But I get the celery_worker_1 exited with code 0 error.

Project-Structure:

celery
|-- flask-app
|   `-- app.py
|-- worker
|   |-- celeryd
|   |-- celeryd.conf
|   |-- Dockerfile
|   |-- start.sh
|   `-- tasks.py
`-- docker-compose.yml

Flask-app/ app.py:

from flask import Flask
from flask_restful import Api, Resource

from celery import Celery

celery = Celery(
                'tasks',
                broker='redis://redis:6379',
                backend='redis://redis:6379'
)

app = Flask(__name__)
api = Api(app)

class add_zahl(Resource):
    def get(self):
        zahl = 54
        task = celery.send_task('mytasks.add', args=[zahl])

        return {'message': f"Prozess {task.id} gestartet, input {zahl}"}, 200

api.add_resource(add_zahl, "/add")

if __name__ == '__main__':
    app.run(host="0.0.0.0", debug=True)

worker: tasks.py

from celery import Celery
import requests
import time
import os
from dotenv import load_dotenv

basedir = os.path.abspath(os.path.dirname(__file__))
load_dotenv(os.path.join(basedir, '.env'))

celery = Celery(
                'tasks',
                broker='redis://redis:6379',
                backend='redis://redis:6379'
)

@celery.task(name='mytasks.add')
def send_simple_message(zahl):
    time.sleep(5)
    result = zahl * zahl
    return result

if __name__ == '__main__':
    celery.start()

Dockerfile:

FROM python:3.6-slim

RUN mkdir /worker
COPY requirements.txt /worker/
RUN pip install --no-cache-dir -r /worker/requirements.txt

COPY . /worker/

COPY celeryd /etc/init.d/celeryd
RUN chmod +x /etc/init.d/celeryd

COPY celeryd.conf /etc/default/celeryd
RUN chown root:root /etc/default/celeryd

RUN useradd -N -M --system -s /bin/bash celery
RUN addgroup celery
RUN adduser celery celery

RUN mkdir -p /var/run/celery
RUN mkdir -p /var/log/celery
RUN chown -R celery:celery /var/run/celery
RUN chown -R celery:celery /var/log/celery

RUN chmod u+x /worker/start.sh
ENTRYPOINT /worker/start.sh

celeryd.conf:

CELERYD_NODES="worker1"
CELERY_BIN="/worker/tasks"
CELERY_APP="worker.tasks:celery"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_USER="celery"
CELERYD_GROUP="celery"
CELERY_CREATE_DIRS=1

start.sh

#!/bin/sh
exec celery multi start worker1 -A worker --app=worker.tasks:celery

celeryd: https://github.com/celery/celery/blob/3.1/extra/generic-init.d/celeryd

Docker inspect Log:

Docker inspect 50fbe00fdc5de56dafaf4268f24baed3b47c8519a689f0733e41ec7fdbc86765

[
    {
        "Id": "50fbe00fdc5de56dafaf4268f24baed3b47c8519a689f0733e41ec7fdbc86765",
        "Created": "2019-02-21T23:20:15.017156266Z",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "/worker/start.sh"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-02-21T23:20:40.375566345Z",
            "FinishedAt": "2019-02-21T23:20:41.162618701Z"
        },

Sorry for the "spam" but I can't solve this issue.

EDIT EDIT EDIT

I added the mentioned CMD line, now the worker doesn't start. I am struggling finding a solution for this. any hints? Thank you all.

FROM python:3.6-slim

RUN mkdir /worker
COPY requirements.txt /worker/
RUN pip install --no-cache-dir -r /worker/requirements.txt

COPY . /worker/

COPY celeryd /etc/init.d/celeryd
RUN chmod +x /etc/init.d/celeryd

COPY celeryd.conf /etc/default/celeryd
RUN chown -R root:root /etc/default/celeryd

RUN useradd -N -M --system -s /bin/bash celery
RUN addgroup celery
RUN adduser celery celery

RUN mkdir -p /var/run/celery
RUN mkdir -p /var/log/celery
RUN chown -R celery:celery /var/run/celery
RUN chown -R celery:celery /var/log/celery

CMD ["celery", "worker", "--app=worker.tasks:celery"]
2
pythondockerflaskcelery

2 Answers

2
votes

Whenever a Docker container's entrypoint exits (or, if you don't have an entrypoint, its main command), the container exits. The corollary to this is that the main process in a container can't be a command like celery multi that spawns some background work and immediately returns; you need to use a command like celery worker that runs in the foreground.

I probably would replace the last couple of lines in your Dockerfile with:

CMD ["celery", "worker", "--app=worker.tasks:celery"]

Keeping the entrypoint script and changing it to an equivalent foreground celery worker command should do the job too.

1
votes

You could also use supervisord to run your celery worker. A bonus is supervisord will also monitors and restarts your worker if something went kaput. Below is example from my working image extracted for your situation...

File supervisord.conf

[supervisord]
nodaemon=true

[program:celery]
command=celery worker -A proj --loglevel=INFO
directory=/path/to/project
user=nobody
numprocs=1
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
stopasgroup=true
priority=1000

File start.sh

#!/bin/bash
set -e
exec /usr/bin/supervisord -c /etc/supervisor/supervisord.conf

File Dockerfile

# Your other Dockerfile content here

ENTRYPOINT ["/entrypoint.sh"]
CMD ["/start.sh"]