Celery error in scheduled task execution

Question

I have an instance of Celery running with Redis server in an archlinux machine. In this instance I have defined some tasks to be executed every X minutes. The schedule is working correctly (if I check the logs, the tasks are being called), but there is a task outputting an error. The task is to run another python script. If I run this script manually, it works perfectly. But when celery try to execute it, I receive this error:

WARNING/ForkPoolWorker-93] Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
TypeError: 'int' object is not callable

The tasks.py is:

@periodic_task(
    run_every=(crontab(minute='*/15')),
    name="sm37_auto",
    ignore_result=True
)
def sm37_auto():
    comando_sm37 = os.system("python /home/user/sm37.py")
    background_thread=threading.Thread(target=comando_sm37)
    background_thread.start()
    return 'ejecuta SM37 ok'

Could somebody help me?

Answer 1

I suppose, you are talking about /home/user/sm37.py which works perfectly. The script task.py will execute fine as well because it has nothing to execute: syntactically the function is correct.

The problem is that when you start a thread, you need to pass something executable as target and you pass a return value instead. Consider making a change:

def sm37_thread():
    comando_sm37 = os.system("python /home/user/sm37.py")

def sm37_auto():
    background_thread=threading.Thread(target=sm37_thread)
    background_thread.start()
    return 'ejecuta SM37 ok'

Answer 2

Here:

comando_sm37 = os.system("python /home/user/sm37.py")

this executes the /home/user/sm37.py scripts and returns the process exit status, which is an integer error code (where 0 means 'no error' and everything else is an error).

Then you have:

background_thread=threading.Thread(
    target=comando_sm37
)

where you try to create a thread (why you want to do so in this context is way beyond my imagination but that's another topic), passing an integer as the target callable.

To make a long story short, this code makes no sense at all.

Using os.system to execute a Python script is already quite questionable (python code can import other python code, you don't need os.system for this) but well, you might have reasons to do so (legacy code, whatever...). But starting a thread from a celery task, really ???