22
votes

I have a AWS Lambda function that creates an object from a s3 call in cold start. I then hold the object in the cache while the function is warm to keep load times down. When files are changed in s3, I have a trigger to run the lambda, but not all the running instances of lambda restart and pull from s3.

Is there a way to bring down all instances of lambda forcing a full cold start?

Also, I don't want to use python.

8
Depending on the size of the s3 object your caching, maybe you can verify that the etag still matches to determine if you should pull the object again. If the objects themselves aren't much larger than etags, then this strategy doesn't make sense of course. - RaGe
There are multiple folders and files that are looped through to create the object. Is the etag for the whole bucket or the individual files? It might make sense to do an async check on the etag. - Marc
etag is per s3 object - RaGe
wait a sec, now I'm confused. Your lambda triggers on s3 update events? which means everytime your lambda runs, s3 has changed and you need to re-pull? what's the point of caching? - RaGe
The lambda is a router, that redirects traffic. The routes are updated and stored in s3. To keep downtime low, the object is cached so that a majority of the time, when the lambda runs the routes are already mapped in an object. s3 is only checked on a cold start of the lambda. s3 has a trigger for the lambda to be called, but it doesn't restart all the instances of the lambda. So some of the lambdas have an outdated route. - Marc

8 Answers

15
votes

I made an Answer based on my comment and verification from @DejanVasic

aws lambda update-function-configuration --function-name "myLambda" --description "foo"

This will force the next invokation of the lambda to "cold start".

To verify:

@timestamp, @message | sort @timestamp desc | limit 1000 | filter @message like "cold_start:true"
13
votes

Use the UpdateFunctionCode API endpoint to force a refresh of all containers. AWS SDKs wrap this up to make it easier for you to call the API using your preferred language.

5
votes

Currently, there is no way to force restarts on running Lambda containers.

You can, however, redeploy the function so that it will start using new containers from that point onwards.

4
votes

If you are using the Lambda versioning system, another way to do this is by publishing a new version and using an alias to direct all traffic to it.

Here's an example:

Publish version: aws lambda publish-version --function-name your-function-name-here

Update the alias pointing to the new version: aws lambda update-alias --function-name your-function-name-here --name alias-name-here --function-version 123 (use the function version in the output message from the first command above)

2
votes

The only way force lambda to discard existing containers is to redeploy the function with something different.

Check out my answer here: Force Discard AWS Lambda Container

Good luck, Moe

2
votes

In addition to some of the valid answers above: I happened to run an experiment on the (average) AWS Lambda instance lifetime. I could not find instances that ran for much longer than (on average) two hours: https://xebia.com/blog/til-that-aws-lambda-terminates-instances-preemptively/.

TL;DR: AWS Lambda is preemptively terminating instances (even those handling traffic) after two hours, with a standard deviation of 30 minutes.

1
votes

Easiest way I found was changing something in Basic Settings like timeout: Basic Settings

I've upped+1 by a second, saved, and the function got refreshed Memory and timeout settings

0
votes

Simply add a new environment variable and / or change an existing one. I created one named BOGUS and gave it a number that I increment whenever I want to force a cold start.