How can I have more than 5 Lambdas consume the same Kinesis stream?

Question

I would like to have one Kinesis stream with lots of Lambda consumers (different Lambda functions, not just many instances of the same one).

Kinesis has a limit of 5 read transaction per second (docs). Lambda functions will poll a shard every second.

Does this mean that I should expect my reads to get throttled when I add a sixth Lambda consumer to a stream?

I understand that more than 5 consumers on one stream are doable with the new Enhanced Fanout feature, but I can't find any mention of Lambda functions with regard to this new feature. They only talk about KCL2.

Instead I have found a few older articles of people describing their own fan out implementations to avoid lagging Lambda consumers, e.g., this one. I wonder whether this is still needed or Lambda consumers can take advantage of the new enhanced fanout too.

Can you have one Lambda Parent (consuming), invoke many lambda children by the parent passing the stream data to the children? — Ryan Breece

Ryan Breece Ryan Breece · Accepted Answer · 2018-10-25T20:36:04

Can you try something like this?

/* in the parent lambda */

var consumers = [
  {
    id: "consumer",
    eventData: someData // stream data inserted here
  }
]

var params = {
  ClientContext: "MyApp", 
  FunctionName: "MyFunction", 
  InvocationType: "Event", 
  LogType: "Tail", 
  Payload:  null,        // <Binary String> 
  Qualifier: "1"
 };

let invoke = () => {
  return new Promise((resolve, reject) => {
    lambda.invoke(params, function(err, data) {
      if (err) reject(err);
      else     resolve(data);
    });
  })
}

invokeArr = []
consumers.map((consumer) => {
  params.Payload = consumer.eventData
  invokeArr.push(invoke)
})

Promise.all(invokeArr).then((data) => console.log('were done'))

How can I have more than 5 Lambdas consume the same Kinesis stream?

1 Answers