Not receiving all messages at receiver lambda from SQS with AWS serverless and NodeJS

Question

We created two lamdba functions using AWS Serverless and NodeJS one is sender and other is receiver, and in between these two there in one Standard SQS which is receiving data from sender lambda and it automatically triggers the receiver lambda function if any message arrives at SQS.

Here, we are able to send all messages successfully from sender lambda to SQS(as per the logs), but at receiver lambda we are not getting all the messages.

To test this scenario I sent 1000 messages from postman and tested that sender was sending all messages perfectly to SQS, but receiver was only getting random 986 messages out of it, some random messages have been missed in between.

I also tried the SQS type to FIFO instead of standard but it is not supported by lambda.

Here is the code for sender and receiver function:

SENDER:

sender: () => {
    const sqs = new AWS.SQS({ apiVersion: "2019-08-09" });
    let body = [
        { id : "1" },
        { id : "2" },
        { id : "3" }
    ]
    let params = {
        DelaySeconds: 10,
        QueueUrl: url,
        MessageBody: JSON.stringify(body)
    };
    sqs.sendMessage(params, function (err, data) {
        if (err) {
            callback(true, null);
        } else {
            callback(false, data);
        }
    });
}

RECEIVER :

receiver: () => {
    event.Records[0].body = JSON.parse(event.Records[0].body);
    async.timesSeries(event.Records[0].body.length, (i, next) => {
        const params = {
            TableName: "user",
            Key: {
                id: "1"
            }
        };
        dynamoDb.get(params).promise()
            .then(result => {
                //save user
            })
            .catch(error => {
                //throw err
                next();
            });
    }, () => {
        console.log("deleting message");
        const deleteParams = {
            QueueUrl: "queue_url",
            ReceiptHandle: event.Records[0].receiptHandle
        };
        sqs.deleteMessage(deleteParams, function (err, data) {
            if (err) {
                //throw error
            } else {
                //success
            }
        });
    });
}

Mark B Mark B · Accepted Answer · 2019-09-21T13:21:33

In this code:

event.Records[0].body = JSON.parse(event.Records[0].body);
async.timesSeries(event.Records[0].body.length, (i, next) => {

By explicitly taking Records[0] you are only processing the first record in the list. event.Records is a list, and you need to iterate over all records in the list. See an example SQS event in the documentation here that includes multiple records.

Alternatively, in your Lambda SQS integration settings, change the Batch Size to 1 so that only 1 record at a time will be passed to your Lambda function.

Not receiving all messages at receiver lambda from SQS with AWS serverless and NodeJS

1 Answers