python - Using boto to invoke lambda functions how do I do so asynchronously?

Question

SO I'm using boto to invoke my lambda functions and test my backend. I want to invoke them asynchronously. I have noted that "invoke_async" is deprecated and should not be used. Instead you should use "invoke" with an InvocationType of "Event" to do the function asynchronously.

I can't seem to figure out how to get the responses from the functions when they return though. I have tried the following:

payload3=b"""{
"latitude": 39.5732160891,
"longitude": -119.672918997,
"radius": 100
}"""

client = boto3.client('lambda')
for x in range (0, 5):
    response = client.invoke(
        FunctionName="loadSpotsAroundPoint",
        InvocationType='Event',
        Payload=payload3
    )
    time.sleep(15)
    print(json.loads(response['Payload'].read()))
    print("\n")

Even though I tell the code to sleep for 15 seconds, the response variable is still empty when I try and print it. If I change the invokation InvokationType to "RequestResponse" it all works fine and response variable prints, but this is synchronous. Am I missing something easy? How do i execute some code, for example print out the result, when the async invokation returns??

Thanks.

Answer 1

An asynchronously executed AWS Lambda function doesn't return the result of execution. If an asynchronous invocation request is successful (i.e. there were no errors due to permissions, etc), AWS Lambda immediately returns the HTTP status code 202 ACCEPTED and bears no further responsibility for communicating any information about the outcome of this asynchronous invocation.

From the documentation of AWS Lambda Invoke action:

Response Syntax

HTTP/1.1 StatusCode
X-Amz-Function-Error: FunctionError
X-Amz-Log-Result: LogResult

Payload

Response Elements

If the action is successful, the service sends back the following HTTP response.

StatusCode

The HTTP status code will be in the 200 range for successful request. For the RequestResponse invocation type this status code will be 200. For the Event invocation type this status code will be 202. For the DryRun invocation type the status code will be 204.

[...]

The response returns the following as the HTTP body.

Payload

It is the JSON representation of the object returned by the Lambda function. This is present only if the invocation type is RequestResponse.

Answer 2

There is a difference between an 'async AWS lambda invocation' and 'async python code'. When you set the InvocationType to 'Event', by definition, it does not ever send back a response.

In your example, invoke() immediately returns None, and does not implicitly start up anything in the background to change that value at a later time (thank goodness!). So, when you look at the value of response 15 seconds later, it's still None.

It seems what you really want is the RequestResponse invocation type, with asynchronous Python code. You have a bunch of options to choose from, but my favorite is concurrent.futures. Another is threading.

Here's an example using concurrent.futures:

(If you're using Python2 you'll need to pip install futures)

from concurrent.futures import ThreadPoolExecutor
import json

payload = {...}

with ThreadPoolExecutor(max_workers=5) as executor:
    futs = []
    for x in xrange(0, 5):
        futs.append(
            executor.submit(client.invoke,
                FunctionName   = "loadSpotsAroundPoint",
                InvocationType = "RequestResponse",
                Payload        = bytes(json.dumps(payload))
            )
        )
    results = [ fut.result() for fut in futs ]

print results

Another pattern you might want to look into is to use the Event invocation type, and have your Lambda function push messages to SNS, which are then consumed by another Lambda function. You can check out a tutorial for SNS-triggered lambda functions here.

Answer 3

The following are a python function that accepts lambda-function-Name to invoke and payload to send to that function.

It invokes the lambda function by boto3 client.

import boto3, json, typing

def invokeLambdaFunction(*, functionName:str=None, payload:typing.Mapping[str, str]=None):
    if  functionName == None:
        raise Exception('ERROR: functionName parameter cannot be NULL')
    payloadStr = json.dumps(payload)
    payloadBytesArr = bytes(payloadStr, encoding='utf8')
    client = boto3.client('lambda')
    response = client.invoke(
        FunctionName=functionName,
        InvocationType="RequestResponse",
        Payload=payloadBytesArr
    )
    return response

And usage:

if __name__ == '__main__':
    payloadObj = {"something" : "1111111-222222-333333-bba8-1111111"}
    response = invokeLambdaFunction(functionName='myLambdaFuncName',  payload=payloadObj
    print(f'response:{response}')