Why Your Terraform Cloud Runs Are Stuck

The issue of Terraform Cloud runs getting stuck can be attributed to a combination of factors, including workspaces, queues, locks, and the concurrency ceiling. A workspace is the unit of serialization and state isolation, processing one apply-track run at a time. Once a run reaches a terminal status, it can no longer apply changes and will not block subsequent runs.

However, a common misconception is that a plan with no changes will wait for approval. If the plan contains no changes, it will terminate as planned_and_finished, and TFC will close it out automatically. No-op plans are not a source of dashboard noise.

Queues are per-workspace, not per-org. A queue can have an arbitrary number of stuck runs behind a single stuck run. When diagnosing issues, it is essential to exclude plan-only runs from the queue depth computation, as they do not contribute to the issue at hand.

Workspaces should be grouped by volatility, with volatile workspaces churned through quickly and long-lived resources separated from frequently-changing ones. The anchor workspace acts as a serialization point, and if multiple AWS accounts are assumed from a single anchor workspace, downstream workspaces consuming its outputs are rate-limited by how fast that one queue drains.

In summary, understanding the six layers of Terraform Cloud operations - workspace, run, queue, lock, concurrency ceiling, and trigger - is crucial to identifying and resolving stuck runs. By applying the outlined strategies and best practices, teams can better manage their Terraform Cloud workspaces and reduce the occurrence of stuck runs.