4
votes

I am triggering a Lambda function from an SQS event with the following code:

@Override
public Void handleRequest(SQSEvent sqsEvent, Context context) {
    for (SQSMessage sqsMessage : sqsEvent.getRecords()) {
        final String body = sqsMessage.getBody();
        try {
            //do stuff here
        } catch (Exception ex) {
            //send to DLQ
        }
    }
    return null;
}

The "do stuff" is calling another Lambda function with the following code:

private final AWSLambda client;
    private final String functionName;

    public LambdaService(AWSLambdaAsync client, String functionName) {
        this.client = client;
        this.functionName = functionName;
    }

    public void runWithPayload(String payload) {
        logger.info("Invoking lambda {} with payload {}", functionName, payload);
        final InvokeRequest request = new InvokeRequest();
        request.withFunctionName(functionName).withPayload(payload);
        final InvokeResult invokeResult = client.invoke(request);
        final Integer statusCode = invokeResult.getStatusCode();
        logger.info("Invoked lambda {} with payload {}. Got status code {} and response payload {}",
                functionName,
                payload,
                statusCode,
                StandardCharsets.UTF_8.decode(invokeResult.getPayload()).toString());
        if(statusCode.equals(200) == false) {
            throw new IllegalStateException(String.format("There was an error executing the lambda function %s with payload %s", functionName, payload));
        }
    }

I am using the following libraries:

<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-lambda-java-core</artifactId>
  <version>1.2.0</version>
</dependency>
<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-lambda-java-events</artifactId>
  <version>2.2.6</version>
</dependency>
<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-java-sdk-sqs</artifactId>
  <version>1.11.505</version>
</dependency>
<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-java-sdk-lambda</artifactId>
  <version>1.11.505</version>
</dependency>

The problem is that it looks like the SQS message is not removed from the queue and it gets reprocessed over and over. It happens every 30 seconds which is exactly the value of Default Visibility Timeout. Now, as far as I know, if the lambda consuming the sqs messages is terminating properly it should automatically delete the message from the queue, but this is not happening. I don't think there is any error happening in the lambda because I am not getting any message in the DLQ (and I have a catch-all block) and I cannot see any stacktrace in the logs in Cloudwatch. I am bit confused about what's happening here, anyone has some good idea?

1
The only thing I can see is that the role assumed by the Lambda function does not have permissions to invoke sqs:DeleteMessage on the source queue. Would this be the case? - Thales Minussi
@ThalesMinussi that's is a good idea and I will double check, but I would expect to get an error if that was the case. - burm87
I am not sure if you are able to create the the SQS event source if your role doesn't have permissions to delete messages. I know it fails if I try to create an event source with Terraform if the right permissions are not set. Since Terraform just uses API Calls, I assume it should also fail in the console. But maybe when the event source was created, the role had permissions and then the permission was removed afterwards. I don't think it should error out though since the code "deleting" the message is not part of your function itself, but I may be wrong. - Thales Minussi
OK! that has enlightened my path a little bit now! Since it seems to be a random issue (meaning some messages do get deleted) I would bet on the visibility timeout. Maybe your Lambda is taking longer than 30 seconds to process (given your Lambda has such a long timeout) but the VT is only 30 seconds, which makes the message visible again while one Lambda is still processing it. So you kind of get into a loop and, eventually, they get deleted. I would try to check these things out first: message processing time and visibility timeout. Maybe increase it to 5 minutes or so to begin with - Thales Minussi
Those are great news! Yep, definitely worth checking out why your functions are running for such a long time, but I am glad we have partially solved the issue - Thales Minussi

1 Answers

1
votes

Unless something changed recently, I don't think the AWS SDK for Java automatically deletes the message from the queue. You need to write the code to do that.

I would love to be proven wrong on that one, please share the doc excerpt I missed.

Code sample :