The checklist we go through before shipping AI features to real users
Every AI feature looks good in a demo. You show it to the team on Friday, everyone's impressed, and then on Monday real users start typing things into it that nobody on the team would ever type. We build AI-powered B2C products together with founders at Reliqio , mostly in healthtech, fintech and e-commerce. Somewhere along the way we started keeping a list of things to check before an AI feature…
Before releasing AI features to real users, a comprehensive checklist is followed at Reliqio. This list, originating from a simple doc and continually evolving, gauges the necessity of the AI model, potential user responses, and associated costs.
Firstly, the team assesses whether AI is truly beneficial for the feature. It's deemed useful when dealing with messy inputs like free text, photos, voice notes, or long PDFs. If the input can be neatly organized in a few dropdowns, AI might not be required.
Next, they consider the user's response to AI errors. Instead of merely stating 'if,' they inquire about the worst possible outcome and the subsequent steps. This could mean disclosing the source of the error, allowing the AI to make suggestions while the user finalizes the decision, or even passing the task to a real person. Certain queries, especially in health and financial sectors, are better handled by fixed content or human intervention.
To ensure the prompts are realistic, the team gathers real-life examples - from beta users, support chats, or user interviews - and incorporates them into the prompts before each release. They also run these prompts again whenever there's a change in the model or prompt to catch any unexpected regressions.
Cost is another critical factor. The team calculates the cost per user, taking into account the number of daily active users and the frequency of AI feature usage. They aim to minimize this cost by employing shorter prompts, smaller models for simpler tasks, caching frequently used answers, and setting per-user usage limits.
The team also recommends testing the AI feature on a slow network, such as a mobile phone in a subway, to ensure it doesn't feel unresponsive. Streamlining the response to show the first words quickly and displaying a progress indicator for longer tasks can also enhance user experience.
Lastly, the prompts are printed out and reviewed meticulously to understand what data is fed into them. The team checks if the data retention settings of the AI provider align with the privacy policy and that the purpose of data collection can be explained in a single sentence. An 'off switch' is also implemented for every AI feature, allowing for quick fallback to non-AI operations in case of any failures.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
