A support chatbot needs to be designed for what it won't answer, not just what it will

When a chatbot is designed to answer questions about a product's features, pricing, and troubleshooting, it faces a challenge. The chatbot can easily be tricked into revealing sensitive information by simply rephrasing the question. To address this, the chatbot's design should focus on what it cannot answer, rather than just what it can.

One approach is to categorize the information that must remain protected and let the chatbot decide whether a question falls into those categories. This method allows the model to generalize to phrasings that were not anticipated in advance, making it more resilient against variation in wording.

The chatbot's refusal to answer should also be designed carefully. It should be short and matter-of-fact, without providing any explanation about why the answer cannot be given. This approach prevents the explanation from becoming a clue that reveals the boundary, which could help someone find a way to bypass it.

Implementing this kind of defense in the system prompt of the language model, instead of in application code, is crucial. Keyword-based filters are fragile against rephrasing, as different phrasing can convey the same intent. By giving the model a judgment criterion based on language understanding, it can generalize to phrasings that were not explicitly written down, providing better protection against novel wording.

In conclusion, a support chatbot's effectiveness is not just measured by its ability to answer legitimate questions. It also requires correctly identifying which questions it must not answer, regardless of how they are phrased. By building a category-based judgment for what must stay protected and incorporating the refusal into the chatbot's policy, it becomes more resilient against real-world use.