My Snowflake Agent Was Wrong. So Was My Evaluation.

When an agent received a disappointing evaluation score, investigators asked three key questions: Did the agent answer the question correctly? Did it take the right steps to arrive at the answer? Was the evaluation measuring the desired behavior? In the case of a Snowflake agent, the answers did not always align. A revised lookup recovered an object, but the agent still received partial credit for tool selection.

Other test expectations missed critical steps. One application counter even made missing telemetry appear as zero tool use. The agent maintained three elements simultaneously: the agent itself, evidence of its behavior, and the tests evaluating it. One possible fix involved changing the prompt, but other solutions existed. The observations and retests were specific to the Snowflake Cortex Agent and not a controlled benchmark.

The agent initially claimed an object did not exist, but the metadata showed it did—though only as a source consumed by other views, not the view the agent itself was searching for. Adding a fallback instruction did not resolve the issue. The problem lay in the semantic tool's inability to search by source. Inspecting its definition revealed that the source dimension already existed.

The SQL-generation guidance emphasized view-name searches, which the agent needed to follow. The revised agent received a fallback, instructing it to ask which views consume a given source. The semantic view received guidance on how to perform the lookup to Cortex Analyst. The retest successfully recovered the object and its consumers.

The source dimension's existence and its proper usage mattered because "the data is missing," "the tool cannot do it," and "the agent did not ask correctly" led to entirely different fixes. While inspecting the definition and retesting provided useful insights, these alone were insufficient. A later correction further clarified the distinction.

The investigation did not turn into an unsupported explanation of the ingestion architecture. The evidence used in the investigation included real questions, responses, and follow-ups alongside native execution traces. A scheduled Cortex Code review addressed instruction gaps, data gaps, and tool limitations, leading to further investigation.

By comparing sources, investigators discovered that a missing measurement had been mistaken for a measured zero. A Snowflake monitoring documentation comparison highlighted this discrepancy. The distinction between what the evaluation judged—whether it was the answer's correctness, tool selection accuracy, tool execution accuracy, or logical consistency—was crucial in understanding the confusing results.

The mechanics of each check were detailed in the appendix and Snowflake's evaluation documentation. The missing-object case demonstrated the importance of inspecting individual records. The fix addressed the lookup issue, while additional calls still affected the agent's tool-selection score. This indicated that the retrieval behavior improved but did not prove that every returned statement was correct or that the entire agent had improved.

Real conversations provided valuable test inputs, while old answers were not automatically considered ground truth. Real conversations also supplied questions that would not have been invented in a demo, creating potential test traps if not approached carefully. Retaining successful examples and targeted tests ensured existing workflows remained unaffected.

Changing questions or expected answers created new test baselines, which could overstate improvements if not properly compared with older baselines. The traces also revealed limitations in the tests, such as a Python sandbox that did not show XML-focused cases during recorded inspections. Targeted tests should describe the programmatic parsing context, planning, invocation, and correct extension. Evidence of invocation and correct execution was crucial to validate the agent's performance.