As AI takes on more tasks, what happens when people have to jump in to clean up after it?
Having a human “in the loop” overseeing when an AI makes a decision is important, but what happens to that responsible human over time when the AI gets it right over and over again?
Artificial intelligence systems have been running smoothly for weeks or even months, yet when things go wrong, the human overseeing them must step in and take responsibility for the outcome. However, this reliance on human intervention raises concerns about the individuals' abilities to make independent judgments and handle unusual situations.
Academic experts refer to this as latent risk – the hidden danger that arises when people lose capabilities they are still expected to use. As AI takes on more tasks, employees become less practiced in exercising their independent judgment and less prepared to intervene when needed. This risk is not entirely new, as previous research on automation bias has shown that people can overlook problems or follow automated recommendations even when evidence suggests otherwise.
The shift from humans doing the work themselves to overseeing AI recommendations is already affecting how companies train their employees. For example, accounting giant Deloitte is redesigning its graduate audit training to simulate AI's involvement, ensuring trainees still develop the necessary judgment for more complex work. This change comes as cognitive psychologist Lisanne Bainbridge warned about the "irony of automation" – as systems take over more workloads, employees may face new challenges in monitoring the systems and handling the situations which they cannot.
The paradox here is that while people may be called upon to exercise independent judgment less frequently, the judgments they are expected to make become more difficult. Accountability may also become an issue, as employees remain responsible for the outcomes they have less control over. This mismatch between authority, responsibility, and capability can become a significant organizational design problem.
Organizations typically measure AI-enabled processes based on accuracy and productivity, which can look positive even as the human role changes. However, a decline in the number of overrides by employees may not necessarily mean they have become less capable – it could simply indicate that the AI has become more accurate. To truly understand the evolving relationship between people and AI, organizations must consider how work responsibilities are shifting and ensure their governance structures adapt accordingly.
Written by urgent.news from The Conversation's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
