AWS Lambda SSM calls randomly goes in timeout

Question

I've a lambda deployed on AWS, in a VPC that has internet access via NAT. The deploy is made using Serverless. The lambda uses some Middy middlewares and fetches some credentials from SSM.

The problem is that the SSM fetch randomly goes in timeout!

Here's the lambda code:

/* requirements are omitted */

const authorize = async (_event, _context) => {
  try {
    const ssm = new SSM({
      maxRetries: 6, // lowers a chance to hit service rate limits, default is 3
      retryDelayOptions: { base: 200 }
    })
    const params = {
      Names: ["param1", "param2"],
      WithDecryption: true
    }
    const fetch = () => new Promise(resolve => {
      ssm.getParameters(params, function(err, data) {
        if (err) resolve(err, err.stack); // an error occurred
        else     resolve(data);           // successful response
      })
    })
    const res = await fetch()
    return {
      statusCode: 200,
      body: JSON.stringify(res)
    }
  } catch (_err) {
    console.error(_err)
    return {
      statusCode: 500,
      body: 'error'
    }
  }
}

export default middy(authorize)
  .use(warmup({ waitForEmptyEventLoop: false }))
  .use(doNotWaitForEmptyEventLoop({ runOnError: true }))
  .use(httpSecurityHeaders())

Steven Steven · Accepted Answer · 2021-01-29T09:43:59

The lambda is timing out, because ssm is throttling you with your current configuration (6 retries 200ms) it takes around 26 seconds before your lambda will give up.

You are running here against the SSM standard throughput limits.

You can enable increased throuhgput with:

aws ssm update-service-setting --setting-id arn:aws:ssm:*region*:*account-id*:servicesetting/ssm/parameter-store/high-throughput-enabled --setting-value true

Be aware an extra cost will be incurred for every getParameter call afterwards (0.05$/10.000 requests).

AWS Lambda SSM calls randomly goes in timeout

1 Answers