AWS Lambda subscribed to SNS topic doesn't process all messages to the topic

Question

I am building an alerting system powered by Amazon services.

I drop a file onto S3 daily which spawns a lambda function (let's call it the Generator function) to process the file.

Generator builds alerts based on this file and posts multiple messages to an SNS topic (let's call it Outbox) - one message for each recipient calculated by Generator.

I have a second lambda function (let's call it Courier) subscribed to Outbox which should take each message and do something with it.

The Generator code:

// 'Generator' function

exports.handler = function (event, context) {
  console.log('Reading options from event:\n', util.inspect(event, {depth: 5}));

  var users = {};
  var userSubscriptions = {};
  var alerts = {};
  var artists = {};
  var tracks = {};

  async.waterfall([
    function downloadSubscribersFile (next) {
      // Do stuff
    },
    function downloadAndFormatActionsFile (next) {
      // Download data file and analyse
    },
    function publishAlerts (next) {
      // Now we have alerts built, we need to mail them out
      var recipients = Object.keys(alerts);

      async.each(recipients, function (recipient, callback) {
        var recipientAlert = alerts[recipient];
        console.log(util.inspect(recipientAlert, { depth: 10 }));

        if (alerts[recipient].actions.artists.length < 1) {
          return callback();
        }

        var params = {
          TopicArn: SNS_TOPIC_ARN,
          Subject: recipient,
          Message: JSON.stringify(recipientAlert)
        };

        sns.publish(params, function (err, data) {
          if (err) {
            console.log(err);
            return callback(err);
          } else {
            console.log('PUBLISHED MESSAGE: \n', util.inspect(data, { depth: 10 }));
            console.log('MESSAGE WAS: \n', util.inspect(params, { depth: 10 }));
          }

          return callback();
        })
      }, function (err) {
        if (err) return next(err);
        next(); 
      })
    }
  ], function (err) {
    if (err) {
      console.log('Error: ', err);
    } else {
      console.log('Process successful');
    }
    context.done();
  })

}

And the other function:

// 'Courier' function

console.log('Loading function');

exports.handler = function(event, context) {
    console.log(JSON.stringify(event, null, 2));
    console.log('From SNS:', event.Records[0].Sns.Message);
    context.succeed();
};

When my Generator function is invoked, I can see 12 messages should be posted to the SNS topic. There are no errors recorded when publishing these messages, and yet the Courier function only fires once.

I'm wondering if anyone's had any similar issues with this and whether there's something I'm missing here. It could be that there's something I haven't configured correctly in AWS but I'm pretty confident everything is set up as it should be.

UPDATE:

After looking at the messages I'm attempting to send, it seems the message that WAS picked up by SNS seemed to be the one with the smallest payload. I'm wondering if SNS is able to cope with lots of small, frequent messages to a topic...?

tster tster · Accepted Answer · 2015-05-15T17:41:48

Yes, SNS can cope with high throughput on a single topic. However, there is a maximum message size 256KB, so if your messages are larger than that it might be the cause.

I see your Generator function is logging the messages, are you seeing 12 logged messages with message ids? I see you have a variable alerts which you are hoping to get multiple recipients from, but I don't see where you are setting it.

My advice: Add more logging before you sent the message to verify that what you think should be happening is actually happening.

AWS Lambda subscribed to SNS topic doesn't process all messages to the topic

1 Answers