HarnessDev: Self-Evolving Agent Frameworks — How LLMs Build Their Own Infrastructure

ByteDance's Seed team, along with Singapore University of Technology and Design, Georgia Tech, and other institutions, have unveiled HarnessDev, a groundbreaking research project that demonstrates how Large Language Models (LLMs) can create and evolve their own Agent Harnesses based on task feedback. This innovation addresses a fundamental question in AI agent engineering: Can LLMs build and continuously improve their own Agent Harnesses? The answer is affirmative.

An Agent Harness serves as the core control system for an AI agent, encompassing task execution loops, tool selection, context management, state tracking, result verification, failure recovery, and more. HarnessDev splits the agent development process into two phases: Creation and Evolution.

During the Creation phase, starting from a Weak Seed Harness (a basic framework with minimal I/O capabilities), the LLM constructs a comprehensive agent harness. This involves adding control logic for execution, tools, context, state, lifecycle, and verification. The Evolution phase focuses on fine-tuning the initial harness based on feedback from downstream tasks, evaluating performance, and adjusting the harness accordingly.

Key findings from this research include:

1. LLMs can successfully create effective Agent Harnesses. Out of 18 created harnesses, 17,111 lines of code were added, with Gemini requiring the fewest changes (1,006 lines) but achieving the highest score on Terminal-Bench 2.1 (68.8).

2. Not all implemented mechanisms are always utilized. Out of 108 component instances in Code Harnesses, only 72 were observed executing during actual tasks. Components from State and Memory were rarely used.

3. The Verification Gap reveals that even when a mechanism is implemented, it doesn't guarantee it will be effective. For instance, Opus reported success 99 out of 100 runs but only 48 were correct, leading to the addition of a Completion Check mechanism. Similarly, Data tasks showed 441 out of 2,325 executions produced degraded commits, yet the harness failed to detect them.

4. Cross-Model Adaptation Challenges show that harnesses become specific to the executor used. Switching executors can significantly impact performance. For example, Opus's SWE-Pro Harness scored 69.3 with self-evaluation but only 33.0 when using Gemini as the executor. On the other hand, Qwen's Harness improved by 17.6 points on BrowseComp when using Gemini.

5. Evolution Limitations indicate that improvements on visible feedback sets may not translate well to held-out tasks. Only 53.1% of version changes showed consistent direction between feedback and held-out sets, making it challenging to differentiate genuine improvements from noise.

The research emphasizes the importance of cost-aware design, as different execution strategies can lead to significant variations in token consumption. For instance, the GPT-5.5 Harness consumed 29.3 million tokens with a medal rate of 19.1, while the DeepSeek V4 Harness consumed 208.4 million tokens with a score of 19.6, representing a seven-fold difference in token usage despite similar performance.

In practical terms, this research demonstrates the potential for self-evolving AI systems, the value of structured agent design over pure memory, and the need for robust verification mechanisms. It also highlights the challenges in ensuring that AI agents generalize well across different models and environments. For researchers, practitioners, and the industry, HarnessDev represents a significant step forward in the development of self-evolving AI systems.