How to fix 'too many missed start times' in Kubernetes CronJobs (and why your job silently stopped running)
Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run You check on a Kubernetes CronJob that should run every minute and find that it hasn't created a Job in hours. Somewhere in the controller logs or the CronJob's events there's a message like this: Cannot determine if job needs to be started: too many missed start times (> 100). Set or decrease…
When a Kubernetes CronJob doesn't run as expected, the controller logs a message stating "Cannot determine if job needs to be started: too many missed start times (100)." This occurs when the controller can't determine if the job needs to be started due to missed start times. The default Kubernetes version will stop scheduling the job entirely after 100 missed times, while newer clusters will log a warning event and still attempt to run the most recent missed job.
There are several reasons why missed start times may occur:
1. The CronJob was suspended and then resumed, causing the controller to count every scheduled time during the suspension as missed.
2. The controller itself wasn't running, for example, if kube-controller-manager was down or overloaded.
3. There was clock skew between the control plane node and the lastScheduleTime recorded by the controller, leading to a large number of missed slots.
4. The schedule was very frequent and runs were blocked, causing a large number of missed times.
To resolve the issue, you can follow these steps:
1. Determine how the problem arose by examining the CronJob's logs and events using commands like kubectl describe cronjob my-job and kubectl get events.
2. Set the startingDeadlineSeconds field in the CronJob's spec. This tells the controller to skip runs that are more than a specified number of seconds late. Choose a value that is larger than the controller's evaluation interval, typically 10 seconds. For frequent jobs, a value close to the schedule interval is sufficient, while daily or weekly jobs may require a larger value, such as a few hours.
3. If the job is stuck on an older cluster with the legacy controller, adding startingDeadlineSeconds should be sufficient. The next evaluation will only look within the specified window, finding at most a couple of missed times and starting scheduling again. Trigger the job manually using kubectl create job if immediate execution is required.
4. Decide whether missed runs matter for your specific job. If missed runs could lead to important issues, such as missed invoices or backups, consider adding monitoring to detect when a job hasn't run. This could involve implementing a heartbeat mechanism or checking for a timestamp of the last processed data in your own system. Additionally, you may want to modify your job to determine what it missed out on, rather than relying on the scheduler to replay every slot.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.