I have integrated an API with my Django website. For a complete order via the API, there are 5 request and responses that needs to be sent and received. The last 2 requests are not super important to be sent out sequentially. I have assigned those 2 requests to a celery task.
Scenario: I send out a request to process 10 orders, so each order has 5 requests out of which 2 of them are assigned to celery task. Hence celery has a total of 10 tasks which will be assigned by the broker. Each of these tasks will have 2 requests.
Result: Celery is not being assigned all of the 10 tasks while the API call is being made, so for each set of 10 orders, the ones that get completed are either 5 or 4 or 6 and so on. Each time there are 2 or 3 tasks that are not processed. I am not sure what is causing the task not to be assigned by the broker.
I am using delay() command to call the tasks. My task.py file looks somewhat like this
from celery import shared_task
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
@shared_task
def process_jobs(client_obj, job, **kwargs):
# do task related work
The call to this task looks like this
process_jobs.delay(self, job, **response_arg)
process_jobs contains 2 HTTP requests that is made sequentially one after the other with responses saved in the DB.
Please advice
