JobMaster achieving horizontal scaling for .NET jobs while maintaining a centralised audit log
The Problem When it comes to dealing with the problem of background task processing in a distributed .NET system, you are usually presented with two options. Message brokers (such as Kafka, RabbitMQ, SQS, and others) scale well and are capable of handling high throughput, but they were not designed to provide the kind of answers that a job system needs to give. For example, what actually happened…
In a distributed .NET system, background task processing can be challenging. Message brokers are good at handling high throughput, but they do not provide an audit trail like a job system requires. This leads to issues such as identifying the status of a particular job, tracking retries, and determining if a job failed silently.
Traditional .NET job schedulers like Hangfire and Quartz.NET offer features such as retries, recurring schedules, and an audit trail, but they scale poorly as more workers compete for the same database rows. To address this issue, a new architecture called JobMaster is proposed.
JobMaster separates auditing from job execution by dividing jobs into partitions and assigning a dedicated worker to each partition. This approach eliminates lock contention and ensures high performance. The system uses an "Agent" layer for execution and transport, as well as a separate "Master DB" for audit history and coordination. The Agent layer only temporarily stores jobs that are ready to run, while the Master DB acts as the long-term store.
A bucket system is employed to asynchronously record job details in the Master DB while keeping throughput high. Jobs are first written into a bucket and then persistently stored in the Master DB, allowing for high throughput. Jobs scheduled far in advance are evicted from the bucket once they reach the Master DB, where they remain until their execution time approaches.
The architecture also includes a Coordinator that picks up jobs from the Master DB and assigns them to buckets based on priority and worker lane. Each worker owns a set of buckets, ensuring that jobs assigned to a specific worker's Critical bucket do not contend with jobs in other workers' buckets. If a job fails, it is sent back to the Master DB and re-dispatched to a bucket for another attempt, up to a configurable retry limit.
Both succeeded and failed outcomes are recorded in the Master DB, maintaining a centralized history of job execution.
JobMaster is an open-source solution currently in development.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
