Harness Engineering

Harness engineering is a technique that safeguards AI-assisted code generation with deterministic tooling, agent-based review, and regular entropy checks. The purpose is to maintain the correctness and coherence of AI-generated code over time. This concept was first introduced by Birgitta Boeckeler in her article on martinfowler.com, within the framework of teams utilizing AI coding assistants for real software development at ThoughtWorks.

Boeckeler observed that AI assistants, while producing code that looks plausible, tend to drift when left unconstrained. They lose track of conventions, repeat mistakes, and gradually erode the codebase's internal consistency, even if the code continues to compile and pass tests. The degradation occurs silently without immediate detection.

Boeckeler drew an analogy to software engineering, specifically the concept of the test harness. Tests do not ensure code correctness by design; they only detect when the code stops being correct. Similarly, a harness for AI coding needs to verify broader aspects such as whether the codebase adheres to architectural decisions, naming conventions, security constraints, and structural rules agreed upon by the team. Functional tests alone are insufficient for this purpose; a distinct type of harness is required.

Three categories of concerns are addressed within this harness. First, the AI coding assistant relies on the knowledge of the project. If it lacks awareness of specific tools (e.g., logging libraries), it may invent its own approach. If it is unaware of certain programming practices (e.g., avoiding mutable global state), it may inadvertently incorporate them.

If it fails to recognize the necessity of using specific abstraction layers for database writes, it could bypass those layers. Context engineering involves maintaining a document (HARNESS.md in this plugin) that captures the project's stack, architectural decisions, naming conventions, constraints, and their rationales. This knowledge base is crucial for the AI agent, requiring accuracy, specificity, and regular updates, unlike a README that serves human readers.

While context engineering helps in informing the AI about what it must and must not do, enforcing these rules remains a separate challenge. Even after thorough context engineering, the AI may still violate the established constraints due to its probabilistic nature, optimizing for plausibility rather than strict rule adherence. Verification slots act as enforcement points, defined moments in the development workflow where checks are performed, either passing or blocking progress.

These can be deterministic tools (e.g., linters, scripts, regex checks, file-structure assertions) for precise constraints or agent-based reviews (language models evaluating code against constraint descriptions) when dealing with intent, semantics, or complex patterns beyond mechanical rules.

The progressive hardening principle advocates for the gradual migration of constraints from agent-based reviews to deterministic tools as the team's understanding of each constraint improves. A codebase is dynamic, with entropy accumulating over time due to dead code, outdated TODO comments, stale dependencies, and evolving conventions.

Harness engineering incorporates garbage collection rules as a periodic process to combat this entropy. These rules are explicit declarations of what constitutes clean code, checked by scheduled agents or scripts to identify and address accumulating problems before they escalate. A well-maintained harness is not static; it evolves with the codebase.

HARNESS.md acts as a self-referential document, documenting the status of each constraint—whether it is unverified, undergoing agent review, or enforced deterministically.