I'd like to invoke an AWS Lambda function every 5 seconds, for some monitoring purposes.
According to the docs,
Rate frequencies of less than one minute are not supported.
STFW.
Can I autonomously invoke an AWS Lambda function every 5 seconds?
Although I can not recommend this way, but if you really need to execute a Lambda function every 5 seconds, you can try this:
Please carefully consider if you really need this.
you can use step functions and lambda for that - look at https://aws.amazon.com/blogs/architecture/a-serverless-solution-for-invoking-aws-lambda-at-a-sub-minute-frequency/
Thought I'd still answer this.
Your whole problem is that serverless is not a great fit for near constant workloads. They are more designed for quickly scaling in and out. But if that isn't your use-case then you will almost certainly be better off cost wise using an ec2 instance, you can even reserve one. But I do get that you lose the high-availability and some of the other benefits of Lambda, which might want you to still go with this approach.
If you want to use a sledge hammer to make this approach work you could use a StepFunction. StepFunctions can wait in terms of seconds, but the only catch is that they are not suppose to run indefinitely and will die after 1 year, so your StepFunction will need to start another StepFunction before it dies, or you will have to by some other mechanism, i.e. cloud watch alarms.
This approach also costs a bit of money for the number of invocations you are talking about, would ring up a cost of over
(31,540,000 [seconds/year] / 5 [seconds/invocation] x $0.000025 = $157.7)
Plus your Lambda Costs, (and I think you actually need 2 state transitions so I think you double that figure)
Might be fine if you are only going to run 1. But running lets say 100 would be well over 10k! Most certainly an ec2 instance will cost you less.
Here's the full take in serverless with a step function executing every second - forever and a day:
service:
name: your-service
plugins:
- serverless-python-requirements
- serverless-step-functions
- serverless-pseudo-parameters
custom:
pythonRequirements:
dockerizePip: non-linux
region: eu-west-1
stage: dev
package:
exclude:
- node_modues/**
- venv/**
provider:
name: aws
iamManagedPolicies:
- arn:aws:iam::aws:policy/AWSStepFunctionsFullAccess
- arn:aws:iam::aws:policy/AWSLambdaFullAccess
runtime: python3.7
versionFunctions: false
region: ${self:custom.region}
stage: ${self:custom.stage}
environment:
PUSH_FUNCTION: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${self:service}-${self:custom.stage}-push
functions:
schedulerStart:
handler: scheduler.start
events:
- schedule: rate(1 minute)
environment:
STATEMACHINE_ARN: ${self:resources.Outputs.SchedulerStateMachine.Value}
scheduler:
handler: scheduler.handle
push:
handler: scheduler.push
stepFunctions:
stateMachines:
everySecond:
name: SchedulerStateMachine
definition:
Comment: "Step Function invoked every minute"
StartAt: ConfigureCount
States:
ConfigureCount:
Type: Pass
Result:
index: 0
count: 60
ResultPath: "$.iterator"
Next: "Iterator"
Iterator:
Type: Task
Resource: "arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${self:service}-${self:custom.stage}-scheduler"
ResultPath: "$.iterator"
Next: "IsCountReached"
IsCountReached:
Type: Choice
Choices:
- Variable: "$.iterator.continue"
BooleanEquals: true
Next: "Wait"
Default: "Done"
Wait:
Type: Wait
Seconds: 1
Next: "Iterator"
Done:
Type: Pass
End: true
resources:
Outputs:
SchedulerStateMachine:
Description: The ARN of the state machine
Value:
Ref: SchedulerStateMachine
scheduler.py
import os
import boto3
lambda_client = boto3.client('lambda')
step_functions_client = boto3.client('stepfunctions')
def start(event, context):
step_functions_client.start_execution(stateMachineArn=os.environ['STATEMACHINE_ARN'])
return {}
def handle(event, context):
index = event['iterator']['index'] + 1
lambda_client.invoke(
FunctionName=os.environ['PUSH_FUNCTION'],
InvocationType='Event'
)
return {
'index': index,
'continue': index < event['iterator']['count'],
'count': event['iterator']['count']
}
def push(event, context):
print("Executed every seconds!")
return{}
Assuming that a retry or a slight time drift isn't the end of the world. Another possible method would be to cron an orchestration lambda once a minute that then turns around and creates delayed sqs messages. You can then use the sqs lambda trigger to invoke your target lambda.
Use the SQS SendMessage DelaySeconds API (it's native likely all of AWS sdk's) as an off set for the number of sub minute calls one wants to make. For example if you want to make a call every 5 seconds you would create 12 messages
delaySeconds: 00delaySeconds: 05delaySeconds: 10delaySeconds: 15With this approach one must pay special attention to sqs's lambda invoke and retry logic. If multiple tries is a problem be sure to setup a DLQ and consider using a FIFO queue. AWS, by design, may invoke the target lambda more then once. Note that standard SQS queues are: at-least-once delivery, and FIFO queues are: exactly-once processing. It's also possible to have some time drift with this approach as CloudWatch may invoke the orchestration lambda a few seconds "after" the minute.
Is this approach better then long running the orchestration lambda, or even a step function? Eh, that depends on a lot of things. Personally I find the idea of leaving a lambda running in a wait state wasteful, you have to pay for that lambda doing nothing most of the time, and if it's a large lambda those costs are not trivial. Using a step function seems like overkill and a much more complex setup then a sqs integration.
At least to me, it seems better to have it spin up, schedule a set of tasks via sqs and die. This is more complex then a long running lambda, has some concurrency issues one must contend with, but it is all "serverless" if that's where one wants to be.
With all that said, lambda isn't the best fit for everything- if this is a "forever" kind of task consider using ECS/Fargate/EC2, as there is a good chance it's cheaper and ultimately less complex.
Cron only allows for a minimum of one minute. What you could do is write a shell script with an infinite loop that runs your task, and then sleeps for 5 seconds. That way your task would be run more or less every 5 seconds, depending on how long the task itself takes.
But it really does sound like you're doing something that you probably shouldn't be doing though. This feels wrong.
There is now a workaround using CloudWatch Events Rule which would trigger a Step Function every minute. The step function would then trigger a given amount of times an "iterator" Lambda, which will invoke your lambda. Here is the AWS blog post.
