A Versioned Evidence Schema for Agent Skills: Provenance, Permissions, Task Fit, and Test Status
AI-assistance disclosure: I used AI to help draft and edit this article. I checked the technical claims and remain responsible for the final text. I also maintain the open-source repository described here, so this is not an independent review. Agent capabilities are becoming easier to package and discover. But cataloging a Skill, MCP server, or plugin is not the same as proving that it works for…
A new open-source project called Agent Catalog Seed aims to create a standardized way of cataloging AI agent capabilities. Instead of simply cataloging popular skills or plugins, the project focuses on making missing evidence explicit and versioned. The state machine for an agent capability has several states: cataloged, statically checked, installation observed, runtime observed, and task acceptance recorded. Each state change requires new evidence.
The project uses a strict JSON Schema for a minimal versioned record. This record includes information such as the schema version, ID, kind (skill), installability, provenance, requested permissions, compatibility, evaluation status, conflicts, and more. By separating concerns, the schema allows for a more granular view of the evidence needed for each state.
A key principle of the project is that the catalog should store evidence types rather than flattening them into a single score. This allows for a more nuanced understanding of an agent's capabilities across different environments. The project deliberately makes entries non-installable and unevaluated, prioritizing honesty over potentially misleading "success" results.
For each capability, the catalog should capture provenance information, including the source URL, maintainer, release or revision details, artifacts inspected or tested, license, and the relationship between the evaluator and author. Requested permissions are recorded but should not be mistaken for enforced permissions. The catalog should separate requested and granted permissions, and it should include both static findings and runtime observations. Additionally, it should record any unresolved risks or contradictions.
The project emphasizes the importance of task evidence, which requires a clear acceptance contract before testing. This includes defining the scenario, acceptance criteria, host and model versions, input boundaries, synthetic or authorized data, planned attempts and retries, human interventions, timeouts, and every pass, failure, interruption, and indeterminate result.
It is crucial to keep failed runs in the catalog, even if retries succeed, and to maintain the original acceptance criteria without alteration after seeing the output.
The catalog should have an evidence expiration policy, with new releases, host updates, model snapshots, permission changes, or task revisions potentially moving a record from observed back to retest_required. This ensures that the catalog remains current and relevant. The public Agent Catalog Seed currently contains a versioned schema, a synthetic example, a validator, boundary tests, evaluation and governance templates, and a real public narrated-video-review Skill.
However, the skill has not been independently tested or verified across multiple hosts, so it serves as a proposal rather than a trust badge. The project encourages teams to open GitHub Issues with non-sensitive test cases related to recurring failures caused by updates to Skills, MCP servers, plugins, or dependencies. By sharing these cases, the goal is to learn whether version monitoring, customer-task regression evidence, and team allowlists can solve engineering problems related to AI agent capabilities.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
