concurrent requests to AWS Lambda

1
votes

I have an AWS lambda function that launches an AWS Batch job. I call the lambda function within R like this:

result <- httr::POST(url, body = toJSON(job, auto_unbox = TRUE))

Where url is (some details redacted):

https://XXXXXXXXXX.execute-api.ca-central-1.amazonaws.com/Prod/job"

This works great when the requests are submitted sequentially. However, if I submit the job from even a small cluster (i.e. 10 nodes), I get a lot of 502 responses, which IIUC means the Lambda API endpoint is refusing the connection due to excessive traffic.

If I throttle the requests it works as desired.

But that does not seem like very high traffic (at most, 10 concurrent requests). My questions are: 1) am I interpretting the 502 response correctly and 2) What are the concurrent request limits for Lambda requests via the API Gateway?

1
aws-lambdaaws-api-gatewayhttr
Have you reviewed CloudWatch Logs for this Lambda function to determine if there's some back-end bug in your code that only exhibits under load? Lambda failures could conceivably cause responses that API Gateway considers malformed causing it to return 502 to the client. - jarmod
@jarmod yes. I see a lot of "timed out after 3 seconds". I think the problem is that I am logging to S3 and use a file locking scheme. As a result the concurrent jobs need to "wait their turn" for the log file likely leading to delay greater than 3 seconds. Increasing the timeout fixed the issues, but I should really do something more sophisticated for logging like kinesis firehose (which I wish exposed a simple web api, but would require another lambda service to feed data to kinesis if I understand correctly) - Eric
Note that if your locking scheme uses S3 objects, that isn't a safe and viable strategy. S3 does not guarantee that created and deleted objects will appear and disappear immediately except when an object is created for the first time, and not even then if you check for its existence before creating it. If it uses actual files in the Lambda container, that will not work either because no two concurrent invocations will ever see the same /tmp. DynamoDB conditional writes are the official mechanism for distributed locking. - Michael - sqlbot
@Michael-sqlbot yes figured that out the hard way. The simplest thing seems to be just to write to stdout which then shows up in the cloudwatch logs. - Eric

1 Answers

0
votes

Based on helpful comments from above, it became apparent the problem was not concurrent requests, but timeouts from the lambda function. This was evident in the logs. So when receiving 502 responses from your lambda API endpoint, inspect the cloudwatch logs for further details, including timeouts.