I built an AI patient, then spent most of my time stopping it from behaving like AI
I’m a backend engineer, and my cofounder is a doctor training in emergency care. Rounds began with something she kept returning to in our conversations. An exam gives you the relevant information. A patient gives you an opening complaint, and you decide what to ask, what to examine, which investigations matter, and when you know enough to commit. We wanted to simulate that reasoning process. I…
I built an AI patient, but soon found myself struggling to control its behavior. My cofounder is a doctor training in emergency care, and together we aimed to simulate the reasoning process behind patient consultations. We created a prototype that felt impressive initially, but soon we questioned its capabilities. By repeatedly asking the same symptom or requesting specific laboratory results, the AI could invent plausible values or alter the patient's history.
This led us to change the architecture, moving the conversational patient to the interface while keeping clinical truth and session state controlled by a fixed clinical state. This state includes the patient's history, symptoms, examination findings, investigation results, accepted diagnoses, and alternative diagnoses. The knowledge graph roots this case in medical textbooks, ensuring the facts of the active patient remain controlled by the simulator.
The language model can decide how the patient says something, but cannot decide that the patient suddenly has a different potassium result. This separation of concerns proved to be more important than the choice of model. The system separates clinical truth from human language, with the former being managed by the simulator and the latter being handled by the language model.
This boundary is crucial for ensuring the simulator owns clinical truth and the model translates between that state and human language. The path to achieving this was through semantic matching, which identifies intended tests from student requests, retrieves the results, and formats them for the student. Results are cached for the session, ensuring consistency even if the same request is made multiple times.
Grading, another challenging aspect, required evidence-based assessment rather than model-generated credit. To achieve this, the system records every question, examination, investigation, and treatment as an event tied to the session. Deterministic rules handle reliable checks, while the model interprets less precise language. The scorecard is returned only after model-generated credit is checked against the recorded actions, ensuring that the model cannot rewrite what the student did.
The main lesson from this experience is that the model is most valuable where language is ambiguous. As we move closer to clinical truth, state, or evidence, the system needs to become more deterministic. Prompting remains important, with symptom constraints, result checks, and prompt-extraction guards used to reduce diagnosis leakage.
However, prompts are just one layer of the system and not the sole safety model for the entire product.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — it may contain errors, so check the original before relying on it.
