Testing the claim: a degraded-link matrix as a required CI gate

This article details the creation of a required CI gate to test a robotics fleet substrate under degraded network conditions. The system, called Ganglion, runs over various network profiles to ensure that the system can handle network issues that engineers typically don't encounter in a controlled environment. These conditions include packet loss, high latency, asymmetric connections, and no direct route between endpoints.

Initially, the design attempted to use netem to create lossy network conditions with a repeatable seed, but it soon became clear that netem does not support deterministic seeds. This realization led to the rejection of this approach, as a gate that fails randomly is worse than no gate at all. The final solution used deterministic delay, constant delay with zero jitter, and rate caps for bandwidth, which are all reproducible and reliable.

The gate was built using existing network shaping mechanisms and a harness that ran the deploy, invoke, and verify process through a relay. This approach avoided creating a separate network test that would not exercise the actual product path. Each run was recorded in a JSON artifact, detailing the mode, seed, shaping commands, duration, and results. This allowed for replaying failing runs and identifying genuine bugs.

To keep CI costs low, the gate was implemented as a single required-gate job, not five parallel jobs. The profiles ran sequentially inside the job, and only the build was repeated for all five profiles. The gate only ran on main pushes, not every push, and early exits were allowed for documentation-only pushes. The de-flaking pass, which retries the run once and attaches compose logs on failure, was also implemented to prevent flaky gates from being disabled.

After implementing the gate, all five deterministic profiles passed on the main branch. The gate now includes clean, lossy, high-latency, asymmetric, and nat-relay profiles, providing a comprehensive test suite for the robotics fleet substrate.