Kubernetes job scheduling using work queue

Question

We are writing a package delivery tracker application using Kubernetes/go-lang. Everything's fine until we've reached a road block where we have certain scheduling needs, and couldn't figure out what to use. The use case is:

1. Once a package is assigned and out for delivery, its location should be tracked.
2. If the package is not assigned after 3 hours of reaching the warehouse, we should mark it as "delivering tomorrow".
3. Some usual cron-type jobs.

for #1 we are planning to schedule a job which will poll the location of package after every 10 seconds. But this is a one time job. Once the package has been delivered. It should die out.

for #2 we are planning to schedule a timed/cron job which will be triggered by our API. This job will be scheduled for 3 hours after the receipt of package at warehouse.

Our #3 above is a usual cron-job for various internal purposes.

We want to use a single scheduling platform to serve all these type of requirements.

Since we are using Kubernetes exclusively, we want to leverage it's job scheduling functionality. But we have certain doubt about it.

1. Can we create these jobs from our source code ? It is possible as mentioned here. But I am not sure whether our system administrator would allow us to do that.

2. I've read about work queues in Kubernetes, where we can push our jobs to a work queue and a consumer will create jobs for these work items.

   But, I am unaware of how to create a permanent consumer daemon for that work queue, which will poll that queue and create jobs for each work item.

   Another doubt is how to schedule simple cron jobs(#3 above) here.

I've also heard about Kala, but not sure how will this fit in Kubernetes world.

Any reference, pointers, links or suggestions would be highly appreciated as I am pretty new to Kubernetes/Golang in general, and not finding anything concrete on google as well.

Answer 1

How many packages are there? How important are the time intervals you mention? There are several approaches that come to mind:

Kubernetes cron jobs can run on a schedule, and presumably solve your third need. If you have "some" packages (I wouldn't necessarily try this with more than "hundreds") then for each package you could create a new cron job which runs once then deletes itself. Cron jobs can't run more often than once a minute. If you can relax your "poll once every 10 seconds" requirement to "poll once a minute" and the number of packages is small, you could do everything this way.

For the second requirement, you can check every package for expiration on an interval. This works better if the "after 3 hours" requirement isn't particularly strict. Write a program that looks at every package, and if the three-hour delay has passed, changes its state, then write a cron job that runs this every 15 minutes. The scaling issues around this are different, but "find expired packages" sounds like a single SQL query; it depends on your backend technology.

You can write one long-running program that does all of the interval-based checks. You hint at using Go, and you can set up a time.Timer that triggers every 10 seconds (either one for the entire program, or one per package). I would probably have the program know, in memory, each package ID, its current state, and when it needs to change state to "delivering tomorrow", and run all of this in a single process. An ordinary Deployment would manage this. You should also keep this state in a database, and that might introduce some scaling limits.

Finally, you can use a message queueing system. RabbitMQ is a popular open-source choice for this. You can arrange for messages to be delayed (send a message to exchange package.in that routes to a queue package.wait with a TTL of 10 seconds, set that queue's dead-letter exchange to package.ready, send that to a queue package.out, and have applications consume from that queue). Again you'd write a long-running process; this one gets items from the queue, retrieves their state from the database, polls for their current location, and marks them as "delivered tomorrow" or requeues them in package.in as appropriate. This is the most scalable answer (you can launch many workers via a Kubernetes Deployment) but also makes the least use of inbuilt Kubernetes parts.