Scalable cron architecture on top of AWS

Question

We have web application used by customers and they have option to create reports. Report contains email and scheduled time(Ex: every day, 9 a.m.).

When the scheduled time matches the current time web app does some work and sends result to the email.

I have one requirement where I need to implement scalable cron architecture on top of swf.

My needed architecture looks like below:

1. User creates report (DONE)
2. Webapp saves report to the db and sends report data and timer to the cron microservice through SQS(simple query service). (DONE)
3. Cron microservice reads incoming SQS messages and sends back SQS message when timer elapsed. (NEED THIS)
4. Webapp reads SQS message and triggers Data Analyzer and Emailer function to send analyzed data. (DONE)

From I read about SWF service is we can create cron jobs and SWF autoscales. How can I create scalable cron microservice using SWF?

Open to any suggestions...

P.S. Web app is written in nodejs, it will be very good to write microservice in nodejs.

Update 1: After spending some time researching possible solutions, I found https://github.com/capside/CloudCron project. But it relies on cloudwatch event. For many scheduled tasks it may charge a lot IMO.

Update 2: Banjo Obayomi suggested to use Lambda function with SQS and CWE.

Solution 1:

1. Webapp sends SQS message.
2. SQS message triggers Lambda function1
3. Lambda function creates CWE rule
4. CWE rule triggers Lambda function2
5. Lambda function2 sends SQS message back to webapp.

Limitations: We can only create 100 CWE rules per region. It means webapp can not generate more than 100 reports with scheduled date.

Links: cron-in-aws-with-lambda-function

Answer 1

Sounds like you can use AWS Lambda to be your ingesting point, have the function trigger from incoming SQS messages, do your processing and then send SQS message back.

https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/

Answer 2

It would be overkill to use the SWF just for the step 3. But it would make sense to use it to implement steps 2 to 4 as a single workflow. This way you don't need the SQS dependency, the application is simpler and you get much better visibility into your process.

Unfortunately SWF doesn't provide supported nodejs client side library. It also doesn't support autoscaling as you have to run the worker processes. But autoscaling can be implemented on top of it.

Answer 3

I did not find the exact solution to my problem, but found another way to achieve my goal with CloudWatchEvent and SQS services.

Instead of receiving individual cron SQS messages from cron microservice with params, and doing analyzing, notification work. We moved logic to backend app like below:

1. Created SQS queue ("cron-microservice-dev") which receives messages from CloudWatchEventRule
2. Created CloudWatchEventRule that sends SQS queue every 1 hour( we can change timer with cron like syntax). So every 1 hour CWE rule sends message to SQS queue.
3. Now backend listens to "cron-microservice-dev" queue and checks from DB if there exists scheduled reports that should be triggered. If there exists reports it will analyze each report and notifies the receiver.

Answer 4

You don't need a cron, you just have to use the correct queue type.

With SQS and other queues (i.e. RabbitMQ) they have what is called a "dead letter exchange" and you can add a time-to-live (TTL) to specific queues. SQS' solution is called Delay Queues and when you push a message to them, your subscribers do not receive them until the delay has passed.

• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html

You must declare the delay interval when creating the queue, but it's very straightforward. I believe you have to first choose a FIFO type queue, then set delay interval.

Other queues you can set a TTL on messages and declare a dead letter exchange (meaning after message expires, it will push it to target queue). That's a nice trick to create a delay and your subscribers are listening for new messages in the queue it gets pushed to after the delay.

1. Choose FIFO queue and "Configure"
2. Set the Delivery Delay