AWS Lambda timeout when another (long) Lambda is invoked

Question

ers

I am using Step Functions that link together multiple Lambdas. In one of these lambdas I invoke another lambda. Both lambdas have 15 min timeout. However, after some mins the primary lambda that invokes the secondary lambda times out (the secondary lambda happily continues its work).

Error

com.amazonaws.SdkClientException
Cause

{"errorMessage": "Unable to execute HTTP request: Read timed out",
  "errorType": "com.amazonaws.SdkClientException",
  "stackTrace": [
    cause": {
      "errorMessage": "Read timed out",
   "errorType": "java.net.SocketTimeoutException",
   "stackTrace": [
  "java.net.SocketInputStream.socketRead0(Native Method)",
  "java.net.SocketInputStream.socketRead(SocketInputStream.java:116)",
  "java.net.SocketInputStream.read(SocketInputStream.java:171)",
  "java.net.SocketInputStream.read(SocketInputStream.java:141)",
  "sun.security.ssl.InputRecord.readFully(InputRecord.java:465)",
  "sun.security.ssl.InputRecord.read(InputRecord.java:503)",
  "sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)",
  "sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)",
  "sun.security.ssl.AppInputStream.read(AppInputStream.java:105)",

This is the code, as you can see I try to raise the timeout using 4 different commands.

  // (2) Instantiate AWSLambdaClientBuilder to build the Lambda client
        AWSLambdaClientBuilder builder = 
 AWSLambdaClientBuilder.standard().withRegion(region);
        // (3) Build the client, which will ultimately invoke 
   the function
        AWSLambda client = builder.build();

        // (4) Create an InvokeRequest with required parameters
        InvokeRequest req = new 
   InvokeRequest().withFunctionName(random_arn).withPayload(jsonString); 

        // (5) Invoke the function and capture response
        int timeout = (15*60*1000);
        req.setSdkClientExecutionTimeout(timeout);
        req.setSdkRequestTimeout(timeout);
        req.withSdkClientExecutionTimeout(timeout);
        req.withSdkRequestTimeout(timeout);

        InvokeResult result = client.invoke(req);

Any idea how to tackle this timeout?

Answer 1

There is more than one timeout you might be interested when working with AWS Lambda. The one you are referring to is the function execution timeout - the timeout people usually mean while talking about Lambda. However, there are also other timeouts you want to configure. Take a look at AWS CLI docs:

https://docs.aws.amazon.com/cli/latest/reference/index.html

Here you have --cli-read-timeout and --cli-connect-timeout. If I'd create a Lambda function with say 5 mins function execution timeout, and invoke it using AWS CLI using RequestResponse (synchronous) invocation type, if it happens it takes more than 1 min to execute (default value for --cli-read-timeout), AWS Lambda will fail this execution and trigger automatic re-try, up to a max number of retries. From what you are describing, I believe you are facing that kind of trouble. Set all the timeouts and try again.

A few more links you may find helpful:

AWS Lambda using Java - setting the configuration connection/socket timeouts

https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html

https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#setSocketTimeout-int-

Answer 2

As you don't explicitly specify the invocation type, you invoke your second AWS Lambda function from your first AWS Lambda synchronously. So while your second AWS Lambda function is running, the first one will keep running as well, waiting for the response of the second function. That's why your first AWS Lambda function will time out if the sum of the run time of the first and the second AWS Lambda function is higher than the configured maximum timeout of the first AWS Lambda function (15 minutes in your case).

Invoking an AWS Lambda function synchronously from another AWS Lambda function is bad practice as you might run in timeout issues as you experience right now and spend money for running the first AWS Lambda function while the second AWS Lambda function is running.

What you could do is to invoke the second AWS Lambda function asynchronously, by setting the proper invocation type for your InvokeRequest:

InvokeRequest req = new InvokeRequest().withFunctionName(random_arn)
                                       .withPayload(jsonString)
                                       .withInvocationType(InvocationType.Event); 

That of course only works if you don't depend on the output of the second AWS Lambda function to finish your first function.

The probably better option, as you're already using AWS Step functions, is to use a Step Functions state machine to orchestrate the invoking of the second AWS Lambda function, as described in the following tutorial: https://aws.amazon.com/de/getting-started/tutorials/create-a-serverless-workflow-step-functions-lambda/

Depending on how your AWS Lambda functions it might make sense to split the first AWS Lambda function, which is currently invoking the second AWS Lambda function into two parts running before and after the invoked Lambda function.