The Polyglot's Dilemma: Conformance Testing a Dozen Specs in as Many Languages

MongoDB engineers manage multiple implementations of multiple specifications in a dozen programming languages. The company develops client libraries, or "drivers," in languages such as JavaScript, Rust, and Python, with each driver having its own unique implementation. While some companies opt for a simpler approach using thin wrappers and C, MongoDB prefers to develop native drivers in each language for better integration with runtime models and conventions. This approach, however, raises the question of how to ensure consistency across all drivers.

Early in her tenure as a Python driver engineer at MongoDB, the author joined in the intense debates surrounding driver standardization. They experimented with various methods, including Cucumber tests written in different languages, but eventually settled on a shared test suite using YAML. This decision proved to be a success, allowing the drivers to converge on a shared API and behavior while still maintaining language-specific idioms and conventions.

The Unified Test Format, built on YAML, now encompasses nearly 1,500 test files that cover a wide range of behaviors, including document manipulation, network latency handling, transaction management, and encryption. The test suite ensures that all MongoDB drivers adhere to the same protocols, enabling the company to prove conformance and update specifications collectively. The YAML tests also uncover hidden behaviors that regular integration tests might miss, such as auto-retry logic and connection pool management.

By utilizing this test-driven approach, MongoDB has significantly reduced its codebase size and minimized the risk of bugs, ultimately strengthening the consistency and reliability of its diverse driver ecosystem.