From SkillCheck to TraceMantle: Tracking Agent Skill Changes and Test Evidence
I’ve renamed SkillCheck to TraceMantle and expanded its scope. The original project focused mainly on validating AI agent skill files. TraceMantle retains that functionality and adds support for tracking the files in a skill package, comparing changes, and checking whether previous test evidence still applies. The rename resolves a conflict with another project using the SkillCheck name. It also…
TraceMantle, the successor to SkillCheck, is a tool that tracks changes in AI agent skills and verifies test evidence. The renamed project retains the original's focus on validating AI agent skill files, but now expands to monitor changes in supporting files, compare evidence, and validate if previous test results remain applicable.
This update resolves naming conflicts with another project and better reflects TraceMantle's broader purpose of aiding developers in assessing both the contents of an agent skill and the evidence used to approve a specific version.
TraceMantle can analyze package changes alongside the evidence provided for its checks, ensuring that developers can identify which earlier results remain relevant and which need to be rerun. The tool introduces bundle manifests to track files in a skill package and their content fingerprints, enabling the detection of changes to supporting resources even when SKILL.md stays the same.
The tool's expanded functionality includes three main capabilities: bundle manifests, evidence imports, and evidence comparison. Bundle manifests record files in a skill package and their fingerprints, allowing TraceMantle to detect changes to supporting resources. Evidence imports can retrieve results from a supported, version-pinned Promptfoo export format, preserving the original export and recording information about evaluated inputs, configuration, checks, and execution context.
Evidence comparison checks supplied evidence against the package and a trusted policy, reporting changed inputs, incompatible results, and checks that require another run.
To approve the imported report, the project owner or trusted automation must validate the evidence through a policy that outlines required checks and their inputs. TraceMantle reads this policy from a specific trusted Git revision. Changes in the candidate package cannot override the selected policy or make a required check optional.
The comparison also distinguishes between a failed check and missing or unsuitable evidence. An unknown result indicates that the available evidence does not establish a pass, preventing incomplete evaluations from being treated as successful releases or confirmed failures.
The tool's release workflow checks built packages through clean installations, alongside tests, linting, and strict type checking. These improvements support a broader workflow while maintaining TraceMantle as a local Python CLI and library. The new features, enhancements, and fixes contribute to TraceMantle's stable and reliable operation, with two remaining defects currently being addressed before the implementation is considered complete.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
