Health AI Should be Assistive, Not Autonomous

Physicians have signed progress notes generated by ambient technology, accepted predictive risk scores embedded in electronic health records, and submitted AI-drafted insurance appeal under their own names. Each case treats AI as assistive, yet the organization fails to clearly define who exercises judgment. The real danger is not a dramatic shift to algorithmic diagnosis but clinicians gradually losing the time, information, authority, and discretion required to make decisions while retaining legal, ethical, and professional responsibility.

Each AI tool changes the clinical encounter differently, raising distinct questions about consent, review, and accountability.

The concern lies in the widening gap between responsibility and control. Clinicians may sign machine-generated notes, accept predictive recommendations without independent review, or click accept to avoid extra documentation. A clinician's signature on a largely machine-produced note appears technically accurate, yet lacking the nuances of the encounter that make it clinically important.

Similarly, accepting a predictive recommendation or submitting an AI-drafted appeal under a clinician's name may lack the clinician's own examination and context.

This arrangement, where formal accountability remains with humans but practical control is with technology and organization, is ethically questionable. A person cannot be truly responsible for a decision unless they have the authority, information, and opportunity to make it. Automation bias arises when organizational conditions like time pressure and fragmented records make independent review seem inefficient and optional.

When AI-generated documentation is usually correct, reading every sentence may feel like wasted effort. When a recommendation is automatically included in the record, it gains an unearned authority. Over time, clinicians risk becoming reviewers rather than originators of professional judgment.

This is not an argument against AI technology. Ambient tools can reduce documentation burdens, allowing physicians to focus on patients. Predictive tools can identify deterioration or adverse drug events. AI-assisted appeals can help clinicians challenge insurance systems. The benefits are real. However, efficiency alone does not ensure proper governance.

Prediction is not the same as judgment. A model, built from past data and defined outcomes, can tell you what has happened before but lacks clinical acumen to reliably predict what will happen next or what should happen for a specific patient.

Two safeguards are crucial: specific consent and attention to what the tool cannot see. Patients should be informed about AI recording, retention, the system's output, who reviews it, and whether it influences diagnosis, treatment, or insurance authorization. Declining such consent should not affect care. Clinicians must also pay attention to what the tool cannot perceive, such as tone, contradiction, family dynamics, and subtle clinical cues that often matter most.