AWS Lambda: where to store temporary secret data?

Question

I am using AWS Lambda with SQS trigger. My Lambda function is getting data from SQS and sending it to an external source. This external source is requiring authentication. So, before sending data, my function going to get two AWS SSM parameters - key data expiration and API key. With key data expiration my Lambda understanding - should we update API key or not.

Now I am starting to getting these errors:

[ERROR] ClientError: An error occurred (ThrottlingException) when calling the GetParameter operation (reached max retries: 4): Rate exceeded

I never know how much data I will get in SQS, so, looks like sometimes I am getting data with too big frequency.

My question is - how I can solve this issue? Which AWS service or way I can start to use instead of AWS SSM? Is it possible to cache data between AWS Lamdas?

Answer 1

Yes, you can cache data within Lambda functions.

When an AWS Lambda function is created, it is provisioned within a container. Any code that runs in the Lambda function outside of the handler function will be run when the container is first created. This code could, for example, retrieve the parameters from SSM and store it in a global variable within the Python program.

Then, when the Lambda function is invoked, the Handler function is called and it can access the global variable.

(Or, instead of retrieving the data when the container is provisioned, the handler function could check to see whether the data was fetched into a global variable and, if not, it can fetch the parameters from SSM. Similar activity, but in this case it runs within the handler rather than outside of the handler.)

To read more on this topic, see: AWS Lambda execution environment - AWS Lambda

Another option is to enable a higher throughput limit for Parameter Store. See: AWS Systems Manager Now Supports Use of Parameter Store at Higher API Throughput

Or, you could consider using the AWS Secrets Manager. See: AWS Secrets Manager now supports larger size for secrets and resource polices and higher request rate for GetSecretValue API