Dynamic Scheduling

This is a bonus edition of Cloud Musings. Every week, I try to write about the challenges we solve whether technical or business. Last week, I finished working on the scheduling aspect of cloud deployments. Here is a demo of how it works.

Looking forward to deep dive into next week's newsletter with with code. This is a very high level approach on how we deal with this problem.

Dynamic scheduling is one of the hardest problems developers face. As a developer, I want to let users choose a date and time to deploy their cloud infrastructure. For example, I wan to deploy the Oatfin API on February 10, 2023 at 3:00 AM. Dynamic scheduling is hard because there is little support for it out of the box. To get it right, you also need a firm understanding of distributed systems. With that also, as every developer knows, timezone is very hard to deal with.

Static scheduling on the other hand is very simple, every programming language provides some kind of support for writing cron jobs. An example of a cron job: I want to import data from a vendor every day at 8:00 PM.

Frontend:

1. To keep it simple, a user specifies year, month, day, hour, and minute from their own point of view.

2. We use the moment-timezone npm package to guess a user's timezone from the browser.

Backend:

1. We run 3 docker containers: automata-celery, automata-scheduler and automata-api. automata-celery is the base image and the other 2 containers extend the base image and override the CMD directive in the Dockerfile.

2. We use MongoDB to store the exact schedule in the database with the user's timezone.

3. We use celery as a task queue and celery-redbeat as the scheduler. Celery-redbeat provides a RedBeatSchedulerEntry object to insert schedules in Redis. When we insert a schedule in Redis, we translate the user's schedule to UTC date and time.

4. Once the task is complete, we mark it complete in MongoDB, which removes it from the list of scheduled deployments.

5. When a user cancels a task, we delete this entry from Redis and it won't run.

Thanks for reading!

Jay