AI receptionists seem like a dangerous idea after a patient revealed it failed to understand her request

Judith Butterfield, a 71-year-old woman recovering from a stroke, recounted how an AI receptionist at her GP surgery repeatedly failed to understand her fragmented speech after the stroke affected her speech and movement on the right side of her body. After numerous attempts, she decided to switch to another practice. The surgery subsequently deactivated the AI system.

This incident highlights a broader issue: healthcare technology must be designed to accommodate patients' communication needs, including those with speech impairments or regional accents. The AI receptionist, known as EMMA developed by QuantumLoopAI, was introduced to manage telephone demand and reduce waiting times. However, Healthwatch Rotherham reported complaints from patients, including those with regional accents and speech impairments, who struggled to communicate with the system.

The automated service allegedly asked Butterfield not to use speakerphone, but when it failed to recognize her name, no human intervened to help her. This issue extends beyond a single receptionist system, as Healthwatch England collected similar accounts from patients experiencing difficulties with AI while booking appointments, requesting medication, and communicating their needs.

While AI can benefit overburdened GP practices by answering calls and freeing reception staff, it also poses risks if not designed with patient communication in mind. Healthwatch warns that vulnerable patients, such as those with communication needs, disabilities, or lower digital confidence, are particularly at risk of being excluded from healthcare technology.

The incident in South Yorkshire underscores the importance of ensuring AI receptionists make it easier to reach a human, rather than creating additional obstacles between patients and the care they need.