Our hospital chatbot cheated at its own exams — on purpose

When a hospital installs an HIS such as InterSystems TrakCare, the knowledge is stored in three locations: classroom sessions, a Moodle filled with training videos, and a collection of assessments that must all be passed. After six months, this knowledge becomes outdated and inaccessible as recordings are seldom watched again and PDFs remain unopened.

Our goal was to enable users to interact with a chatbot and receive accurate answers in either English or Arabic. To achieve this, we created a custom solution. To train the chatbot, we employed an unconventional method. The bot operates within a testing environment, utilizing free open-source models that may experience timeouts under heavy load.

The process involves the bot driving a real Chrome tab via Chrome DevTools Protocol and automatically completing all 36 Moodle assessments. Each assessment's PDF of a perfect attempt contains the correct answers, which are then compiled into a comprehensive dataset of 433 verified question-answer pairs. In addition to the written content, the bot also utilizes video material.

After extracting 38 short how-to videos and five hour-long live trainer recordings from the Moodle's BrightCove embeds, we generated timestamped JSON segments of these videos and distilled the information into eleven topic files. Our retrieval engine relies on a hand-coded version of BM25, an information-retrieval algorithm, because our entire corpus consists of approximately 1,100 chunks.

This approach allows for instant retrieval and enables us to assign different trust levels to various sources. Answers from completed assessments receive higher priority compared to random transcript lines. Official workbooks, video distillations, and the glossary are given a 1.2×, 1.1×, and 1.15× boost in ranking, respectively.

The bot is designed to cater to Arabic-speaking hospitals, as the majority of hospitals using TrakCare are located in Arabic-speaking regions. To address the challenges of Arabic language processing, the bot normalizes various Arabic characters, removes definite articles, and consults a clinical synonym map with over 100 entries.

Users can interact with the bot in Arabic, receive the response in English, and understand the Arabic language better through the bot's assistance. The entire system consists of a small Node.js server running on an obscure port behind a Cloudflare tunnel, a lightweight LLM proxy with a low temperature setting, and four bilingual hint chips with inline visuals for frequently asked questions.

The entire solution, including the CDP drivers, answer-key JSON, whisper pipeline, BM25 core, FAQ layer, and test battery, is estimated to be around two thousand lines of actual Node.js code. It does not rely on any vector databases or GPU acceleration, making it a cost-effective and straightforward implementation. For those planning to deploy TrakCare or any healthcare information system and seeking to streamline the training process, we recommend consulting with us prior to a new rollout. The 433 question-answer pairs are already prepared and ready to be utilized.