IDKMesh: What if AI agents had to prove their work?
A few days ago I wrote about one result from IDKMesh: a panel of 25 verifiers that, after measuring their shared errors, was effectively worth only one independent verifier. That experiment is interesting, but it starts in the middle of the story. So this post is the simpler version: What is IDKMesh, why am I building it, and how is it supposed to work? The problem AI agents are getting very good…
IDKMesh is an open-source research project that aims to coordinate humans, AI agents, software tools, verifiers, and various forms of compute around uncertain tasks. The name reflects the project's core concept: acknowledging that both humans and AI agents may lack complete knowledge. The main goal is to separate the task execution from the verification process, allowing for more reliable and trustworthy outcomes.
A practical example illustrates the difference between traditional AI agent workflows and the IDKMesh approach. In a typical scenario, an AI agent would complete a task, such as fixing a bug in a Python project, and declare the work done. However, IDKMesh proposes a more robust workflow that separates the worker's proposal from the verifier's independent evaluation. This separation ensures that the final decision remains under human control, preventing potential mistakes from being blindly accepted by autonomous agents.
The IDKMesh system consists of five key components: workers, verifiers, verification results, provenance evidence, and explicit final decisions. Workers are given limited, bounded tasks to perform, while verifiers independently check the proposed solutions. The system maintains a record of all verification results and evidence, enabling future inspection and replay.
Crucially, the final decision is not automatically made based on the worker's success, ensuring a clear separation between the worker's actions and the final approval.
While IDKMesh is still in the research and engineering phase, it already includes useful tools like the "idkmesh gate-audit" command, which helps assess the value of independent votes from a review panel. The repository contains various experiments, schemas, validators, simulations, and interoperability work, providing a solid foundation for future development.
The long-term vision for IDKMesh is to explore how large groups of humans and AI agents can collaborate on real-world problems while maintaining a strong focus on verification and trust. This includes dividing work into manageable tasks, running multiple independent attempts, utilizing diverse workers and tools, ensuring reproducible evidence, learning from failures, and scaling the system without granting unlimited authority to autonomous agents.
IDKMesh is open-source, encouraging transparency and community involvement. By making the project's code, experiments, and results accessible, the community can scrutinize the system's assumptions and contribute to its improvement. Whether you are interested in AI agents, verification, distributed systems, CI, developer tooling, or human-in-the-loop systems, the IDKMesh repository on GitHub offers opportunities for collaboration and contribution.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
