aws lambda invoke multiple lambda

Question

I need my python lambda to invoke multiple lambda so they can run in parallel. I have it working - kind of. To test it, I invoke the code below for 5 different lambda calls (could be more in the future)... when i look at the Step Function console, I only see 2 lamdbas running concurrently and the other 3 are ignored after the first 2 finished. In researching concurrency issues, I didnt find a limit that is this low (5 should be nothing for AWS!)..

                response = lambdaClient.invoke(
                    FunctionName=lambdaToInvoke  <---this var is an ARN,
                    InvocationType='Event',
                    Payload=json.dumps(event)   <--just passing state
                )

All IAM is provisioned...Any thoughts are appreciated!

How does Step Functions come into play here? The code you posted invoked Lambdas directly, you would not see that in Step Functions console. — Milan Cermak
hi folks....thanks for your input...sorry for the belated answers.... i could have been more clear in the step functions reference: I am invoking lambdas directly that instantiate step function processes. therefore, when i look at the step function console, I see the step function jobs for the ones that ran but none for the others.... — deant
as far as Lambdas are concerned, they all work fine independently but when i try to stack them up for instantiating a job for each one, i only see the first 2.... @dpmcgarry, what information in the cloudwatch log would be helpful to see? happy to get the data just dont want to back and forth on what to capture :).... thanks again both for your input! — deant
I'm pressuming your Step Functions are already created and you're calling StartExecution on them, is that correct? — Milan Cermak
Hi Milan, yes all lambda functions work fine when executed independently but when they are stacked up in a "fan out" operation (as termed by @dpmcgarry) they are not even invoked if they are after position 2 in the list of lambdas to be invoked...weird. seems like it could be a limitation in the service but i would think we can run more than 2 lambdas at the same time. does it matter that 1 or more of the lambdas being invoked are the the same lambda as the calling (i.e. the calling lambda is creating instance of itself with with different event data....) — deant

dpmcgarry dpmcgarry · Accepted Answer · 2019-03-05T03:35:08

Usually when I need to do a "fan out" operation like you are describing I don't invoke the lambda functions directly. Rather I would recommend you setup your 5 "slave" lambda functions to get activated from either SNS or SQS. From your "master" function you can use the SDK to create a SNS alert or SQS message.

What happens when you try to manually invoke the 3 functions that don't run?

Do you have logging setup in CW logs? Can you post?

aws lambda invoke multiple lambda

1 Answers