The Blind Leading the Blind: Chasing a Production Bug

This is a recounting of the challenges faced by a PHP developer at Racing Post in London around 2016. The developer was part of a team working on a system that processed racing and greyhound data, using PHP 5 and Sybase for database communication. The application relied on a native Sybase extension written in C, which translated function calls, managed resources, and converted responses into PHP values.

Despite efforts to debug the issue, production intermittently failed, returning empty query results with no apparent errors. The problem seemed random, as the same query could fail at different times. Restarting PHP provided temporary relief, but the underlying issue persisted. The developer and a C expert worked together to trace the lifecycle of a query from PHP into the Sybase extension, examining how connections were obtained, commands executed, and statuses interpreted.

They discovered that the extension might be swallowing native errors, converting failures into empty result sets. By modifying the extension to surface native errors properly, the developer improved error handling and gained better visibility into the issue. However, the root cause remained, and the problem continued to occur. The developer emphasized the importance of distinguishing between valid empty results and failed queries, as the latter required reporting, retry logic, or intervention.