Evidence Grades for Agent Claims: Terms, Branches, and Six Leaves

Agent claims go through a structured process to determine their grade. A composite review ticket often starts with a pre-closed ticket containing multiple agent notes. These notes may state that the suite passed, a flake on the runner caused issues, or a dependency bump was safe. However, these sentences lack important details such as exit codes, commands, or job IDs. This gap is a deliberate safeguard to prevent the automatic grading of work that did not produce an artifact.

A grade in this system is a simplified decision, unlike the complex analysis normally associated with grades. The gate's function is strictly limited: it can only decide if a claim can mutate a ticket's state or remain a comment. It does not estimate costs, rank models, or certify suite quality. The checker mentioned later in this report is merely a proposal and has not been executed. The claims made by agents are straightforward statements, not nuanced evaluations.

An evidence packet is a critical component of this system. It consists of a host-side record containing a job ID, the command that ran, the exit code, and a bounded log excerpt. A mere paragraph of prose, even if it appears specific, does not constitute a packet. The grade assigned to a packet is straightforward - it can be Missing, Asserted, Derived, or Observed. These grades do not reflect the model's confidence level, but simply describe the packet's status.

A leaf in this system represents a terminal branch with a single allowed action. Leaves do not negotiate or prioritize urgency based on ticket titles. Mutation, the act of changing a ticket's status, merge state, or assignees, is not a mutation in this tree. Posting a comment, while useful, also does not trigger a mutation. The host is the storage location for packets and the grader's execution environment.

A model response is not a host, nor is a chat scroll. A model call is an optional labeling step that can restatement a grade but cannot create new fields. Redaction, the removal of sensitive information, is mandatory before any packet leaves the host.

The operator describes MonkeyCode as an open-source project with free model access and a free server option. These claims are operator-supplied and should be confirmed before treating them as durable. The article, prepared as part of MonkeyCode's product outreach, does not name specific models, quotas, hardware, retention policies, or time windows. Readers should verify the current project terms before relying on these options.

To grade an agent claim, follow a numbered walk. Start by storing the claim and searching for a packet on the host. If no packet exists, assign Missing and stop. Do not attempt to create an exit code by calling a model. Compare job IDs to ensure the packet from job A supports a claim about job B. If there's a mismatch, the claim remains Missing.

Next, classify the source of the packet. If the exit code, command, and log excerpt match, it's Observed. A summary pointing to a non-stored log is Derived, while a bare sentence is Asserted.

Once the source is classified, read the requested action. Comments are not mutations, and closing, merging, or reassigning tickets are also not mutations. Land on one leaf and write the grade beside the claim, applying the leaf's action and no more. There are six leaves in this system, each associated with four grades. The first two leaves, L1 and L2, handle Asserted claims.

L1 allows a comment that tags the claim as ungraded, while L2 rejects any mutation. The next two leaves, L4 and L5, deal with Observed claims. L4 permits a comment quoting the exit code and command, while L5 rejects an automatic close, requiring a host re-check before human acceptance.