Github Actions: Re-intentar automáticamente los jobs de CI que un reclamo de spot instance mató
Mi CI corre en un runner spot ARM64 autoalojado. Cuando AWS se lleva la instancia de vuelta a media corrida, GitHub muestra una X roja en cualquier paso que estuviera corriendo, y se ve idéntico a una prueba que falla. Esto es un workflow que lee el log del job, reconoce esa falla específica, y re-corre los jobs fallidos por su cuenta. Las fallas de prueba reales se quedan en rojo. Stack: GitHub…
GitHub Actions jobs of CI that were killed by a spot instance reclaim can be automatically retried, according to the source material. The issue occurs when AWS takes away the instance used for the CI runner mid-task, causing GitHub to display a red X on any running step. This is identical to a test failure. The workflow includes a listener that triggers after any CI or deployment run finishes.
If a failed run is identified, it fetches the logs of the failed jobs and looks for specific lines produced only by the runner's host. If a match is found, it calls the rerun-failed-jobs function. However, there are three cases where this action is not taken - if there's a more recent run of the same workflow and branch, if re-running an older attempt would only exacerbate the concurrency group, or if a third attempt still shows a shutdown signal, indicating the pool has fallen and a human should examine the Auto Scaling Group.
The source material also discusses the importance of checking the ASG for potential outages and the need to adjust maxCapacity to ensure a replacement can be added when needed.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.