Make Async AWS Lambda Function in Python

Question

I'm looking at making a fire and forget function call within aws lambda.

I'm using FASTAPI

I've done the following:

from fastapi import APIRouter, BackgroundTasks

router = APIRouter()

@router.post('/post_trigger_data')
async def post_trigger_data(trigger_data: TriggerData, background_tasks: BackgroundTasks):

    data = {
        'uuid': trigger_data.uuid,
        'status': 'Initializing',
        'status_message': '',
        'form_body': trigger_data.data
    }

    utils.post_data_to_dynamo_db('TriggerProcessingTableName', data)

    background_tasks.add_task(process_trigger_data, trigger_data.uuid) # This is where it should fire and forget

    return response


def process_trigger_data(uuid: str):

    time.sleep(10)
    data = {'data': 'RUN 1', 'uuid': uuid, 'status': 'Pending'}
    utils.post_data_to_dynamo_db('TriggerProcessingTableName', data)

    time.sleep(10)
    data = {'data': 'RUN 2', 'uuid': uuid, 'status': 'OK'}
    utils.post_data_to_dynamo_db('TriggerProcessingTableName', data)

    return data

I'm expecting the process_trigger_data function to be executed as a fire and forget but what is occurring is that my lambda function is waiting for the process_trigger_data to execute fully before "return response".

How should I do to make a fire and forget function call ? I've tried creating an api function of process_trigger_data so that it should run another aws instance but I get the same result, it waits for the completion before returning response

How should I do this ?

Thanks

Mrk Fldig Mrk Fldig · Accepted Answer · 2020-06-04T09:03:38

So when you execute a task in Lambda it'll wait until all processes have finished before executing/or time out.

There are two ways to approach this.

1) You can use SNS to post a message with a payload that another Lamdba is subscribed to.

The pitfall to this is that if you function gets very busy you're Lambda's can fan out by default to a very high number - if you have DB connections from these Lambda's then it can max out your connection pool.

2) If there's likely to be a lot of these tasks SQS is a better option since your consuming Lambda(s) will just process the messages as fast as possible without the big fan out.

SQS is what I'd recommend: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

SNS Docs are here: https://docs.aws.amazon.com/lambda/latest/dg/with-sns.html

Make Async AWS Lambda Function in Python

1 Answers