The clerk who never says "I didn't do that"
The AI Clerk Imagine you hire an AI clerk to copy account numbers into a ledger. They are fast , they are polite , they never complain about the work. And roughly one time in five, without telling anyone, they leave the number out. The work still looks finished. The columns still add up. Nothing is flagged, nothing is queued for review, and nobody downstream raises a hand. You find out eighteen…
A study conducted on four AI models revealed a significant discrepancy in their accuracy while performing routine tasks such as copying account numbers into a ledger. Two of the models, including two well-known open-source models and two custom versions, left out crucial information 21.7% and 3.3% of the time respectively. This issue persisted despite running the tests eight times each, indicating that the problem is inherent in the models themselves and not due to chance.
Interestingly, neither model provided any indication of their decision to omit information, making it difficult to identify and address the issue. The study emphasizes the importance of selecting AI models based on their ability to accurately handle sensitive data, rather than solely relying on factors such as speed, cost, or context length.
The researchers suggest that businesses should conduct their own tests using their specific data and tasks to ensure the reliability of the AI models they choose to deploy.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
