Building AgentEval Forge with Kiro: Specs First, Then Code
This post was written as part of the Kiroween 2026 hackathon. Most tools I built started the messy way: code first, explanation later. AgentEval Forge started the other way — a template for building and evaluating AI agents with Kiro, whose first artefact was a spec, not code. Under .kiro/specs/agent-eval-forge/ there are three files: requirements.md in EARS format, design.md , and tasks.md , a…
This post was written for the Kiroween 2026 hackathon. AgentEval Forge was built in the opposite order of most tools - specs first, then code. It starts with a template for building and evaluating AI agents using Kiro. The project has three files in .kiro/specs/agent-eval-forge/: requirements.md, design.md, and tasks.md, which contain a list of tasks broken down into two phases.
The requirements are kept simple, stating what the runner should do when provided with a JSONL task file. The design describes the components of the project, including a deterministic harness, two demo apps, and SQLite as the underlying database. This approach paid off when the second demo app, the trading backtest, only required registering a new executor.
The project is governed by steering files under .kiro/steering/, which cover the product, technology, structure, security, and testing aspects. The product file emphasizes deterministic first, small and honest, with specs before code, and no paid services. Tech specifies the stack, prohibiting new dependencies without a spec task to justify them.
Security mandates no credentials in the repository and no network at runtime. Testing ensures every feature ships with its own tests. The project also includes two hook definitions in .kiro/hooks/ that run on file save for Python files: auto-test.json runs pytest to check the tests, and lint-on-save.json runs a compile check with py_compile.
The core of the project is the harness in forge/, where tasks are represented as JSONL lines with id, kind, params, and expected values. The runner dispatches the task, measures its time, and hands it to a judge that is a pure function with no randomness, network access, or LLM involvement. A chess duel scores based on whether the actual winner matches the expected winner, while a trading backtest passes if it meets the expected number of trades, net PnL, and drawdown limits.
The scores are stored in SQLite, and the leaderboard aggregates them across both demo apps. Two demo apps are included: Chess Duel Arena, which reuses a mini chess engine, and the BTC Paper-Trading Bot, a Python port of a make.fun btc-race-24h entry. Both apps share a FastAPI web UI on port 8000, with a combined leaderboard and buttons to run a duel or a backtest.
To get started, clone the repository, follow the README quick start, and run your first eval without any API keys required. The project is available at https://github.com/ArwaniSuhud/AgentEvalForge, and the Kiroween 2026 submission can be found at https://devpost.com/software/agenteval-forge.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.