Your Load Test Won't Find These Four Gaps. Learnings from OpenAI's Habitat

OpenAI's Habitat, the storage platform behind ChatGPT, is capable of handling 70M+ requests per second, 500+ petabytes of data, and operates across 40 regions with a 10× growth year-over-year for three years. However, the incidents described in OpenAI's engineering post were not capacity problems, as CPU, memory, disk, or bandwidth were not exhausted.

Instead, three of the four incidents were coordination effects that appeared when scaling out, and two of them traced back to library defaults that were correct on small systems but problematic at scale.

The first incident involved a response sitting ready but unparsed due to thread contention, creating an invisible delay on database dashboards. The second incident was caused by a feature-flag SDK polling every 60 seconds with no jitter, which stalled entire pods in lockstep. The third incident involved a connection pool's LIFO default, creating a self-reinforcing outage that persisted even after removing its trigger.

The root cause of these incidents was the deployment of Habitat as a library, which could not be controlled, rolled back, or updated independently. This lack of control led to critical issues such as database performance while user still waiting, and a synchronized fleet-wide stall. To address these issues, the recommended fixes included measuring event-loop delay, running fewer concurrent requests per process, and scaling out the process count instead of increasing traffic.