We put an AI helper in our course and spent weeks teaching it to say I don't know

The project started with creating an AI helper inside a course to answer student questions. It was initially easy to get the model to respond, but a student asked a question outside of the model's knowledge, and it fabricated an answer, which prompted the real project to begin. The primary focus shifted from generating text to preventing the AI from making up false information.

The project tackled three main issues: defining what the AI was allowed to know, teaching it to refuse instead of inventing answers, and determining when to step aside and direct the student to a human.

First, the scope of the AI's knowledge was limited. Initially, the model had access to all course materials, including old threads. However, this led to the model treating irrelevant information as if it were core content, unable to differentiate between important and less important data. The solution was to restrict the AI's knowledge to a predefined set of source material, each chunk tagged with its origin and authority. This change improved the model's ability to distinguish accurate information from noise.

Next, the model needed to be trained to refuse a question when it lacked the necessary information. At first, this seemed counterintuitive because the model's training gradient encourages it to always provide helpful responses. To overcome this, the helper was explicitly instructed to refuse when it didn't have an answer, with examples of appropriate refusals provided in the system prompt. This change significantly improved the AI's behavior when faced with questions outside its knowledge base.

Finally, the helper needed to know when to transfer the conversation to a human. This was crucial because the model could not always provide accurate answers, especially for sensitive topics or when the student was frustrated. The AI was programmed to recognize when it should hand off the conversation to a human, passing along the student's query and the AI's limitations. This allowed the human to pick up the conversation without the student having to repeat themselves, creating a more seamless experience.

In conclusion, the most valuable work in creating an AI helper for educational purposes lies in teaching the model to refuse a question when necessary and to hand off conversations to a human when appropriate. These aspects are critical for building trust with students and ensuring they receive accurate information. The demo of the AI answering questions is just the beginning; the real product is the ability to trust the AI and feel confident that it will not mislead the student.