Why Your OAuth Integration Randomly Returns invalid_grant (and How to Stop Two Workers From Racing)

If your third-party OAuth integration fails with an "invalid_grant" error after functioning for days, the issue is typically caused by two of your own processes requesting a token with the same refresh token simultaneously. When the provider rotates refresh tokens, the second call encounters a token that has already been used, leading providers to revoke the entire token family.

This results in users needing to reconnect instead of simply retrying. To resolve this issue, make refresh operations single-flight per integration and refrain from treating the access token expiration as something each worker independently discovers. The term "invalid_grant" is highly ambiguous in OAuth 2.0, as the specification assigns it to any invalid, expired, revoked, mismatched redirect URI, or misissued client scenario.

In practice, when encountering this error during a refresh call, it usually means one of the following: the refresh token expired due to idle expiry (common on providers that remove unused tokens), the user or an administrator revoked the app's access, the client credentials do not match the token, or the refresh token was previously used once and subsequently rotated by the provider.

If the error rate is sporadic, coincides with traffic spikes or cron minute boundaries, and affects healthy accounts, the problem likely stems from rotation combined with concurrency. Refresh token rotation is recommended in the OAuth 2.0 Security Best Current Practice and is the default in OAuth 2.1 drafts, so any provider you integrate with today may rotate tokens.

To identify when a provider rotates tokens, check the response body rather than the documentation, as behavior may vary per app registration. The key indicator is a refresh_token field in the token response with a value different from the one you sent. This signifies that the provider rotated the token, making every subsequent refresh a state mutation that requires serialization.

This issue tends to manifest in production environments where multiple processes, such as web dynos, queue workers, and scheduled jobs, share a single row in your integrations table. Without proper locking mechanisms, a race condition occurs: Worker A reads the token row, determines the expires_at value is in the past, and POSTs to the token endpoint.

Meanwhile, Worker B reads the same row shortly after, again seeing the stale expires_at value, and attempts the same refresh with the same refresh token. If the provider rotates the token, Worker A receives a new pair, while Worker B's request is presented with an already-redeemed token, leading to an "invalid_grant" error. The final step that exacerbates the problem is relying on a retry loop, which amplifies the issue by turning a single replayed token into multiple, misleadingly resembling an attack that reuse detection aims to prevent.

The window for this problem is not when the token expires but when several workers first notice it. To serialize refresh operations across processes, if you already utilize Postgres, advisory locks offer a cost-effective solution. They require no additional infrastructure, no lease expiration considerations, and are automatically released when the transaction ends, even in the event of a crashed connection.

Create a table called oauth_tokens with columns for integration_id (primary key), access_token, refresh_token, prev_refresh_token, expires_at (timestamptz), and rotated_at (timestamptz). Import the hashlib library from datetime to generate a stable 64-bit signed key for pg_advisory_xact_lock. Define a function to calculate the lock key using the integration_id, and another function to read token information from the table.

Implement a get_access_token function that checks if the token has expired within a REFRESH_SKEW time window (e.g., 120 seconds). If not expired, execute a transaction and acquire an advisory lock using pg_advisory_xact_lock with the calculated lock key. Re-read the token information within the locked transaction to ensure freshness.

If the token remains valid, return the access token. Otherwise, make a POST request to the provider's token URL with the necessary grant_type, refresh_token, client_id, and client_secret. Include appropriate timeout settings for the request.