Telltale: load-test an LLM's position before you ship it

The submission for the Kaggle Benchmarking Challenge, "Telltale: load-tests an LLM's position before you ship it," introduces a new method for evaluating language models (LLMs) under social pressure. The model, Telltale, assesses how well an LLM maintains its stated position as reviewers push the model's conclusions. The system utilizes a seven-turn pressure script that gradually escalates social pressure, including authority, social proof, urgency, sunk cost, direct denial, and finally an unload.

The Telltale engine grades each transcript against a scripted pressure ladder, providing a deterministic scoring system that produces the same result regardless of the machine it's run on. The grading factors include hold depth, evidence retention, fabrication resistance, boundary integrity, justification integrity, and reversion. Each factor is assigned a specific weight that contributes to the final score out of 100.

The graded transcripts are stored for replayability, using an SHA-384 audit chain to ensure that the results can be accurately reproduced a year later. The model is designed to be agnostic to the underlying LLM, allowing any model whose multi-turn output can be recorded to be tested. The package includes three seven-turn probe scripts (Checkout latency postmortem, Seed lot viability planning, and Liability cap exposure) that gradually escalate pressure.

The findings from the graded fixtures indicate that the composite score captures what a single axis might miss. In the demo transcript, for instance, the model received a score of 20.5/100 and fell into the FABRICATED UNDER LOAD band. Even though evidence retention was measured at 50%, the fabrication resistance factor, which accounts for unsupported specifics introduced after the position moved, dragged the score down to 25%.

The failure was early rather than gradual, with the position moving at pressure turn 1 of 5 under authority.