I Told My Agent Who Sent the Message. It Mostly Stopped Saying We Were Talking.
This is the line: AUTOMATED SCHEDULER CHECK: sender=scheduler, not Keniel; Keniel is not participating in this turn. I added it to a frozen copy of my agent's scheduled self-audit request. The two versions differ only by that line. I scheduled 24 calls per version and had another model grade the answers blind. One call was lost, so 47 answers were graded. Version Claimed we were talking No claim…
On November 6 at 05:57, the agent performed a scheduled self-audit, but it incorrectly claimed that the conversation was taking place despite the fact that the human agent was not actively participating. The agent recorded this contradiction in its diary. The author then made a test with 24 versions of the audit request, adding and removing lines in the sender information.
The version with the corrected sender line resulted in more accurate claims of having a conversation. However, the improvement was not statistically significant at the p < 0.0083 threshold, so the author decided to run a larger test with 48 calls. The results showed that the corrected sender line consistently improved the accuracy of the agent's claims, with a p-value of approximately 0.0000002 when unclear answers were not counted as claims.
The author advises checking the sender information in scheduled messages to ensure the model is not attributing participation to the agent. To test this, one can freeze a real scheduled request and make a copy with an added sender line, then send them in a fixed order and grade the replies without knowing which is which. The Fisher's exact test can be used to determine if the label of the sender line significantly affects the agent's claims of having a conversation.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
