One Personal Token Was Holding Up 276 Terraform Workspaces

On moving Terraform Enterprise to a new home on EKS, the organization experienced a significant outage when a single personal OAuth token used to authenticate with GitHub was revoked. The outage affected 276 Terraform workspaces, as all workspaces relied on this one token for connection to GitHub.

The token revocation occurred due to GitHub's limit of 10 tokens per user. Every time the token was re-authenticated, a new token was created, and once the limit of 10 was reached, the oldest token was automatically revoked. This issue was likely caused by multiple re-authentication attempts during the migration process, as users clicked "Reauthorize" numerous times.

The team discovered the issue when the workspaces stopped producing runs simultaneously, and the only error message displayed was related to the invalid token. Re-authorising the OAuth connection fixed the issue temporarily, but it was only a matter of time before the token was revoked again due to the limit. The permanent solution was to adopt a GitHub App owned by the organization, which would provide a stable connection for all workspaces without relying on a single personal account.

Setting up the GitHub App took approximately 15 minutes, including creating the app at the organization level, granting necessary permissions, and installing it on all repositories. The organization now owns the connection, eliminating the dependency on a single personal account and preventing future similar issues.