Building Faultline: A Reusable Chaos-Injection and Linearizability-Checking Harness

Faultline is an open-source Go harness designed to inject faults into distributed systems, record every concurrent operation clients actually observed, and check whether that history is linearizable. The harness was modeled after the methodology used by Kyle Kingsbury, the creator of Jepsen. The reference target for Faultline is a three-node etcd cluster, and another target, NATS JetStream's key-value store, demonstrates the harness's reusability. No consistency violations were found in either target during testing.

After this successful initial run, Faultline caught six bugs in its own development, which is an indication of its credibility as a correctness tool. The architecture of Faultline consists of several components: cmd/faultline (CLI), harness (target-agnostic), workload generator, checker, and client contract. The contract for every target, which includes Connect / Invoke / Close and Init / Apply, remains the same regardless of the target being used.

The harness employs a six-method contract for new targets and independently verifies each applied fault. It utilizes a Wing & Gong style search, memoized on (remaining-operations, sequential-state) with a bounded search budget. The checker was validated against seven hand-built known-good/known-bad histories before being used to test real targets.

During the development of Faultline, the harness found six bugs in its own implementation. For instance, ToyKV's CAS wire response used the wrong field, and containerIP only read Docker's legacy default-bridge field. These bugs were fixed to ensure the tool was reliable and accurate. The findings from one of the tests show that etcd ran 100 successful campaigns with no linearizability violations, inconclusive checks, or invalid attempts.