Urgent.News

What's breaking now, across thousands of outlets.

Tech

How to fix 'too many missed start times' in Kubernetes CronJobs (and why your job silently stopped running)

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run You check on a Kubernetes CronJob that should run every minute and find that it hasn't created a Job in hours. Somewhere in the controller logs or the CronJob's events there's a message like this: Cannot determine if job needs to be started: too many missed start times (> 100). Set or decrease…

When a Kubernetes CronJob doesn't run as expected, the controller logs a message stating "Cannot determine if job needs to be started: too many missed start times (100)." This occurs when the controller can't determine if the job needs to be started due to missed start times. The default Kubernetes version will stop scheduling the job entirely after 100 missed times, while newer clusters will log a warning event and still attempt to run the most recent missed job.

There are several reasons why missed start times may occur:

1. The CronJob was suspended and then resumed, causing the controller to count every scheduled time during the suspension as missed.

2. The controller itself wasn't running, for example, if kube-controller-manager was down or overloaded.

3. There was clock skew between the control plane node and the lastScheduleTime recorded by the controller, leading to a large number of missed slots.

4. The schedule was very frequent and runs were blocked, causing a large number of missed times.

To resolve the issue, you can follow these steps:

1. Determine how the problem arose by examining the CronJob's logs and events using commands like kubectl describe cronjob my-job and kubectl get events.

2. Set the startingDeadlineSeconds field in the CronJob's spec. This tells the controller to skip runs that are more than a specified number of seconds late. Choose a value that is larger than the controller's evaluation interval, typically 10 seconds. For frequent jobs, a value close to the schedule interval is sufficient, while daily or weekly jobs may require a larger value, such as a few hours.

3. If the job is stuck on an older cluster with the legacy controller, adding startingDeadlineSeconds should be sufficient. The next evaluation will only look within the specified window, finding at most a couple of missed times and starting scheduling again. Trigger the job manually using kubectl create job if immediate execution is required.

4. Decide whether missed runs matter for your specific job. If missed runs could lead to important issues, such as missed invoices or backups, consider adding monitoring to detect when a job hasn't run. This could involve implementing a heartbeat mechanism or checking for a timestamp of the last processed data in your own system. Additionally, you may want to modify your job to determine what it missed out on, rather than relying on the scheduler to replay every slot.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Case study (hypothetical): the provider disabled our webhook endpoint after a weekend of 500s, and nobody noticed for 9 days

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run This is a hypothetical case study, assembled from a pattern that turns up again and again with…

  • Provider disabled webhook endpoint after 9 days of 500 errors
  • Online booking platform experienced unnoticed 9-day downtime
  • Technical contact outdated email address led to disabled endpoint

A one-sample t statistic in Python: what the number actually compares

A two-point difference can be a big deal or ordinary sampling noise. The t statistic compares that difference with the estimated uncertainty in the sample mean.

  • One-sample t statistic compares sample mean to hypothesized value
  • Larger t value with smaller p-value indicates stronger evidence
  • Assumes independent normal observations for valid results

Private Image Originals and Thumbnails: Retention, Resizing, and Signed Downloads

Short answer: use private object storage for originals and generated thumbnails, run resizing in your Node.js worker or an image service, and return short-lived signed download links.

  • Separate private originals and generated thumbnails in distinct prefixes (origins/, thumbs/)
  • Use unique keys for variants (originals/{tenantId}/{imageId}/source.jpg) and record them in DB
  • Generate signed GET URLs for downloads, log request ID, image ID, variant, attempt, and time

How to fix 'Blocked due to access forbidden (403)' in Google Search Console (it's usually your WAF or bot rules)

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com Search Console's Pages report has a reason that looks simple: Blocked due to access forbidden (403) .

  • Confirm issue by selecting URLs and performing URL Inspection and live URL test
  • Identify WAF or bot rules layer causing 403 response through headers and body analysis
  • Verify fix by ensuring Googlebot requests return 200 status in live test

More from Wednesday 18 November →