1
votes

I am going through the setting of the following components on CentOS server. I get supervisord task to get the web site up and running, but I am blocked on setting the supervisor for celery. It seems that it recognizes the tasks, but when I try to execute the tasks, it won't connect to them. My redis is up and running on port 6380

Django==1.10.3
amqp==1.4.9
billiard==3.3.0.23
celery==3.1.25
kombu==3.0.37
pytz==2016.10

my celeryd.ini

[program:celeryd]
command=/root/myproject/myprojectenv/bin/celery worker -A mb --loglevel=INFO


environment=PATH="/root/myproject/myprojectenv/bin/",VIRTUAL_ENV="/root/myproject/myprojectenv",PYTHONPATH="/root/myproject/myprojectenv/lib/python2.7:/root/myproject/myprojectenv/lib/python2.7/site-packages"

directory=/home/.../myapp/
user=nobody
numprocs=1
stdout_logfile=/home/.../myapp/log_celery/worker.log
sterr_logfile=/home/.../myapp/log_celery/worker.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 1200

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; Set Celery priority higher than default (999)
; so, if rabbitmq(redis) is supervised, it will start first.
priority=1000

The process starts and when I go to the project folder and do:

>python manage.py celery status
celery@ssd-1v: OK
1 node online.

When I open the log file of celery I see that the tasks are loaded.

[tasks]
  . mb.tasks.add
  . mb.tasks.update_search_index
  . orders.tasks.order_created

my mb/tasks.py

from mb.celeryapp import app
import django
django.setup()

@app.task
def add(x, y):
    print(x+y)
    return x + y

my mb/celeryapp.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings

# set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mb.settings")
app = Celery('mb', broker='redis://localhost:6380/', backend='redis://localhost:6380/')
app.conf.broker_url = 'redis://localhost:6380/0'
app.conf.result_backend = 'redis://localhost:6380/'
app.conf.timezone = 'Europe/Sofia'
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

my mb/settings.py:

...
WSGI_APPLICATION = 'mb.wsgi.application'
BROKER_URL = 'redis://localhost:6380/0'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
...

when I run:

python manage.py shell
>>> from mb.tasks import add
>>> add.name
'mb.tasks.add'
>>> result=add.delay(1,1)
>>> result.ready()
False
>>> result.status
'PENDING'

And as mentioned earlier I do not see any change in the log anymore. If I try to run from the command line:

/root/myproject/myprojectenv/bin/celery worker -A mb --loglevel=INFO
Running a worker with superuser privileges when the
worker accepts messages serialized with pickle is a very bad idea!

If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).

User information: uid=0 euid=0 gid=0 egid=0

But I suppose that's normal since I run it after with user nobody. Interesting thing is that the command just celery status (without python manage.py celery status) gives an error on connection, probably because it is looking for different port for redis, but the process of supervisord starts normally... and when I call 'celery worker -A mb' it says it's ok. Any ideas?

(myprojectenv) [root@ssd-1v]# celery status                                      
Traceback (most recent call last):                                                          
  File "/root/myproject/myprojectenv/bin/celery", line 11, in <module>                      
    sys.exit(main())                                                                        
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/__main__.py", line 3
0, in main                                                                                  
    main()                                                                                  
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
 81, in main                                                                                
    cmd.execute_from_commandline(argv)                                                      
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
 793, in execute_from_commandline                                                           
    super(CeleryCommand, self).execute_from_commandline(argv)))                             
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/base.py", line 3
11, in execute_from_commandline                                                             
    return self.handle_argv(self.prog_name, argv[1:])                                       
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
 785, in handle_argv                                                                        
    return self.execute(command, argv)                                                      
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
 717, in execute                                                                            
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])                              
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/base.py", line 3
15, in run_from_argv                                                                        
    sys.argv if argv is None else argv, command)                                            
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/base.py", line 3
77, in handle_argv                                                                          
    return self(*args, **options)                                                           
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/base.py", line 2
74, in __call__                                                                             
    ret = self.run(*args, **kwargs)                                                         
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
 473, in run                                                                                
    replies = I.run('ping', **kwargs)                                                       
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
 325, in run                                                                                
    return self.do_call_method(args, **kwargs)                                              
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
 347, in do_call_method                                                                     
    return getattr(i, method)(*args)
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/app/control.py", line 100, in ping
    return self._request('ping')
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/app/control.py", line 71, in _request
    timeout=self.timeout, reply=True,
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/app/control.py", line 316, in broadcast
    limit, callback, channel=channel,
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/pidbox.py", line 283, in _broadcast
    chan = channel or self.connection.default_channel
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/connection.py", line 771, in default_channel
    self.connection
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/connection.py", line 756, in connection
    self._connection = self._establish_connection()
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/connection.py", line 711, in _establish_connection
    conn = self.transport.establish_connection()
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
    conn = self.Connection(**opts)
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/root/myproject/myprojectenv/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
socket.error: [Errno 111] Connection refused

Any help will be highly appreciated.

UPDATE:

when I run

$:python manage.py shell
>>from mb.tasks import add
>>add
<@task: mb.tasks.add of mb:0x**2b3f6d0**>

the 0x2b3f6d0is different from what celery claims to be its memory space in its log, namely:

 [config]
- ** ---------- .> app:         mb:0x3495bd0
- ** ---------- .> transport:   redis://localhost:6380/0
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 1 (prefork)
1
You should probably use celery status -A mb - Håken Lid
Check what settings.INSTALLED_APPS contains when you autodiscover tasks. You import django.settings before you set the DJANGO_SETTINGS_MODULE env variable, which might be a problem. - Håken Lid
I don't know exactly what goes wrong. You might try to get the official django example running, and adjust that to using redis (or use rabbitmq as broker) github.com/celery/celery/tree/master/examples/django - Håken Lid
when in the folder of the project: celery status -A mb * gives ok *celery@ssd-1v: OK tried to remove the settings.INSTALLED_APPS and to remove the import of django.settings, but still no change :/ - palamunder

1 Answers

0
votes

Ok, the answer in this case was that the gunicorn file was actually starting the project from the common python library, instead of the virtual env