Knight Capital: how a reused feature flag lost $460 million

On August 1, 2012, Knight Capital Group experienced a catastrophic software failure that resulted in $460 million in losses. The incident occurred when the company deployed new code to its order-routing servers, but one server still contained older code that was retired in 2003. This older code contained a feature flag, which had been previously used to activate a now-defunct function called Power Peg. The new code reused this flag, assuming it would now control the RLP (Retail Liquidity Program) feature instead.

The mistake was made by a single technician who copied the new code to seven of the eight servers, leaving the eighth server with the outdated code still using Power Peg. On the morning of August 1, emails were sent indicating that Power Peg had been disabled, but these alerts were not detected as actual warnings, and no one took action. Within 45 minutes, the faulty code executed 4 million trades, resulting in significant market disruption and substantial financial losses.

The primary issue stemmed from the reused feature flag, which had been reassigned to control a different part of the system. The flaw lay in the fact that the stop condition for Power Peg had been moved in 2005 but never retested, allowing it to continue sending orders without any end. This caused an overwhelming number of child orders to be sent without regard to the number of share executions, leading to a cascade of market disruption.

Despite the severity of the incident, no one intervened to stop the faulty code for 45 minutes. The SEC was the first to respond, issuing a $12 million fine for the first Market Access Rule enforcement. Knight Capital was forced to seek a $400 million rescue and later merged with GETCO. This incident highlights the importance of proper software development, code review procedures, and the potential risks associated with feature flags in financial systems.