HTTP Data Cleanup Triggers: A Recurring Method with Bounded External Work
Short answer: use an external recurring trigger to send one small, authenticated HTTP request to a cleanup endpoint, and make that endpoint claim a bounded time window before doing work. The trigger stays stateless. Your application does not need an internal task executor. The trade-off is clear: a shorter interval reduces cleanup latency but creates more request pressure, so the endpoint must…
The article discusses using a recurring HTTP trigger to efficiently clean up old data in a system. The key points are:
1. An external trigger sends a small authenticated HTTP request to a cleanup endpoint. The trigger stays stateless on the application side.
2. A bounded cleanup window is specified, limiting the amount of work done in each request. This prevents the endpoint from becoming overwhelmed by too many concurrent requests.
3. The cleanup process only advances a cleanup watermark to the next unprocessed slice of records, rather than processing all stale data in one go. A scheduler sends a request like POST /internal/cleanup-tick.
4. A separate worker pool drains the claimed cleanup intervals at its normal rate. The timer is good for triggering but poor for managing application state or retries.
5. A database row or lease records the cleanup window, status, and expiration time. If the request times out, the same window can be reclaimed instead of duplicated.
6. The first pass should be small, with a short schedule interval, safety delay, and cleanup window. This exposes the contract without making it too large.
7. The endpoint should return useful status codes, like 202 Accepted for successful claims or 429 Too Many Requests when the worker pool is full. Duplicate keys can be treated as already accepted if the original claim is still visible.
8. Latency and cost affect the interval chosen. The trigger interval controls how often the service notices work, while the worker pool controls how quickly claimed work leaves the system. The watermark is the honest signal of the cleanup progress.
9. A simple decision rule is to choose an interval below the maximum acceptable cleanup age, and ensure one tick's window is smaller than the pool's sustainable capacity. Adjust the window size, add capacity, or accept a slower freshness target as needed.
10. Monitoring metrics such as trigger-to-claim delay, oldest unclaimed record age, window size, claim conflicts, worker drain time, response status, and retry count are important. The age of the oldest unclaimed record is the key metric to watch.
In summary, using a bounded recurring HTTP trigger with a cleanup watermark allows efficient and controlled data cleanup, with careful consideration of interval, worker pool capacity, and monitoring metrics to ensure the system remains performant and avoids overwhelming the endpoint.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
