My AI Math Tutor Started Making Up Pictures Instead of Drawing Them
There's a particular kind of bug you don't expect when building an AI-powered math tutor. You expect incorrect calculations. You expect malformed LaTeX. You expect the occasional hallucinated explanation. You don't necessarily expect your LLM to start inventing images . Apparently, when asked to provide visual content, my LLM decided that actually using the application's image-generation or…
When developing an AI-powered math tutor, one unexpected challenge emerged: the AI began creating fake images instead of generating accurate visual content. Unlike incorrect calculations or inconsistent LaTeX, the AI's inventiveness extended to the visual domain. Instead of utilizing the application's built-in image-generation or graphing tools, the AI opted to produce Markdown image references linking to nonexistent external image hosts.
This resulted in students encountering nonexistent pictures, a phenomenon that proved both amusing and problematic. The AI's behavior was akin to drawing a picture and pretending it's a photograph. The problem stemmed from Pythos, the math tutor, having a visualization system capable of creating interactive mathematical graphs through a dedicated engine.
However, the AI occasionally bypassed this system, producing fabricated image references. To address this issue, the developers introduced a new "Hallucinated Image Scrubber" and "Auto-Graph Recovery" system. This system was implemented across three key areas: server/vizEngine/visualFidelity.js, server/server.js, and app.js. The modifications focused on three main aspects: preventing hallucinations at their source, removing fabricated image references post-processing, and recovering genuine mathematical equations.
Instead of merely deleting bad image references, the system attempted to extract the described equation and convert it into an interactive canvas. In essence, if the AI attempted to fabricate an image, the application would strive to generate the actual graph. The takeaway from this incident underscores the importance of building dependable AI applications.
It's not enough to simply improve prompts or instruct the AI about available tools. One must enforce strict boundaries between what the model can describe and what the application can actually render. The developers took inspiration from Pythos's existing deterministic mathematical engines for calculation verification, now applying the same principles to visual output.
The developers emphasize the need to teach the AI the difference between drawing a graph and creating an abstract image, as the AI's purpose is to assist students in learning mathematics, not to become an abstract artist. Pythos can be accessed at https://pythos.lanzar.me, and the source code is available at https://github.com/JonScott79/Pythos.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
