Debugging Intermittent Kong 503s When the Logs Showed Nothing but the Status Code

An API gateway failure can sometimes originate within the gateway itself, rather than from an unresponsive upstream service. When using Kong Gateway in a Kubernetes environment, intermittent HTTP 503 errors occurred, despite no apparent issues with the upstream services or Kubernetes DNS. The 503 status codes were logged in the server access logs, but no matching Kong error entries were found.

Through correlation of request status, system logs, and Kong's worker Lua VM memory metric, it was discovered that the root cause was the NGINX worker restarts within Kong. The failure manifested at the peer-selection stage of the gateway's request path, where Kong couldn't select a target for the upstream application. This could be due to all targets being marked unhealthy, incorrect target addresses or ports, DNS or network connectivity problems, health checks, or a crash affecting the gateway process.

The intermittent nature of the issue was explained by the existence of two NGINX worker processes in the Kong pod. If one worker restarted while the other remained active, most traffic could continue normally. However, requests arriving during the worker transition or assigned to the affected worker might fail. The investigation concluded that resource pressure or a crash was not the cause, as the Kong pod remained running and other workers could still serve requests.

The key takeaway is to avoid jumping to conclusions based on a 503 error status code and instead investigate the underlying causes, such as worker restarts, memory pressure, or upstream health issues.