Developing for Debugging (2025)

Oxide’s control plane manages two DNS services: external DNS accessible to customers for locating the system’s APIs and web console, and internal DNS for internal network service discovery. The authoritative source of truth for DNS data is a strongly consistent distributed database called CockroachDB. Nexus, a component within the control plane, updates the DNS configuration in the database when changes occur, such as adding or removing components that provide service internally or externally.

This process involves background tasks in Nexus that propagate the updated configuration to all in-service DNS servers. These tasks run asynchronously after the event triggering the change but usually complete in a few seconds. While designing this system, the team recognized the need for better visibility into the DNS propagation process to debug issues when DNS names are missing. They built omdb, an internal debugger for the control plane, which provides observability into the DNS propagation process.

Omdb tracks key information about DNS configuration, such as what names are part of the configuration, when it was last updated, and by which Nexus instance. For each Nexus instance, it reports the version of the DNS configuration last propagated and the DNS servers it attempted to propagate to. For each DNS server, omdb provides details about the version of DNS data it is operating from.

With these insights, the debugger can generate a comprehensive summary report on DNS propagation, allowing developers to pinpoint where the pipeline may have broken down. This level of detail goes beyond simply reporting "no problems" or "up to date," as it provides concrete evidence that the system is functioning correctly. Such information can help determine if DNS is the issue or if there are other underlying problems.

The omdb debugger not only aids in real-time debugging but also during development. It enables developers to test and play with the system before writing automated tests and understand how components interact. By using omdb, teams can create better onboarding documentation, train new team members, and provide live demos to illustrate how the system operates. Developing with a focus on debuggability ensures that the system is not only functional but also easy to troubleshoot and maintain.