Django: running a few tasks in parallel

1
votes

I have a web application build out of Django version 2.0.1

A user uploads a file, and based on the content, there are tasks which are executed in a serial fashion. After execution the results are shown to the user. Some of the tasks are independent of each other.

I want to execute the independent tasks in parallel. I tried using multiprocessing within views.py but there are some errors thrown when the processes are spawned. These tasks analyse some information and write to a file. The files are then combined to show the results to the user.

These tasks cannot be done asynchronous as the results produced needs to be shown to the user waiting. So I have dropped the idea of using Celery as recommended in other discussions.

Anyone's suggestions would be helpful.

Thanks

Error got

This was the error we gotTraceback (most recent call last): C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\multiprocessing\spawn.py", line 106, in spawn_main exitcode = _main(fd) File "C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\multiprocessing\spawn.py", line 116, in _main self = pickle.load(from_parent) File "G:\work\gitrepo\suprath-github\smartdata\ssd\FinalPlots\uploads\core\views.py", line 6, in from uploads.core.models import Document File "G:\work\gitrepo\suprath-github\smartdata\ssd\FinalPlots\uploads\core\models.py", line 7, in class Document(models.Model): File "C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\site-packages\django\db\models\base.py", line 100, in new app_config = apps.get_containing_app_config(module) File "C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\site-packages\django\apps\registry.py", line 244, in get_containing_app_config self.check_apps_ready() File "C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\site-packages\django\apps\registry.py", line 127, in check_apps_ready raise AppRegistryNotReady("Apps aren't loaded yet.") django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet. Traceback (most recent call last):

2
pythondjangopython-multiprocessing
Your question is missing any kind of information. You say you receive errors, but you did not provide any traceback nor any code to reproduce the problem, so we have no way to tell you whether the issue could be solve with minor tweaks to your original solution. You did not say how much do these tasks take. If they take in the order of minutes then asynchronous requests are a must. For once: the browser or webserver might close connections that last that long, or there may be networking issues. You can submit the work to do by the user and provide the user a link to see the result. – Giacomo Alzetta
Please provide code examples and errors you're getting. I've succesfully used both threads and multiprocesss in Django in the past, so if code is written correctly - it should be possible as well here. – samu

2 Answers

2
votes

These tasks cannot be done asynchronous as the results produced needs to be shown to the user waiting

That doesn't mean you can't use an async queue (celery or other). We have a very similar use case and do use celery to run the tasks. The tasks (part parallel, part serial) store their progress in redis, and the frontend polls to get the current state and display progress to the user, then when the whole process is done (either successfuly or not) we display the result (or errors).

0
votes

I agree with the solution provided by @bruno desthuillieres, however, you can implement some socket solution to reach back to the user.

Since polling from user may have huge performance impacts, the socket solution will ideal for this case.