5 reasons AI evaluations need designers’ unique perspectives

AI evaluations commonly lack designers' unique insights. These assessments, determining when an AI feature is ready for release, rely heavily on scripts and dashboards filled with percentages. Consequently, companies often entrust them to engineers and product managers. Yet, evaluating an AI's success fundamentally hinges on user experience.

Each evaluation raises questions about how the AI should behave in various situations. For instance, what should the AI do when uncertain? Which mistakes can users tolerate? Designers frequently make these decisions in their day-to-day work. However, in AI evaluations, designers often have little influence, if any. This can result in a functional product that fails to meet user needs.

Here are five reasons designers should be included in AI evaluations from the outset. First, designers possess a keen sense of taste and judgment that helps them discern what the right answer should be. Even user ratings can be misleading, as demonstrated by a Stanford study showing people trust flattering AI responses more than those challenging them.

Designers, who work closely with users, develop this sensitivity by observing how products behave through a human perspective. This ability to evaluate answers based on their impact on users' experiences is invaluable. Second, designers can define the AI's response when it lacks knowledge. AI often exhibits excessive confidence, delivering incorrect information, particularly in sensitive areas like medical advice.

In such cases, AI's responses can damage user trust. Designers, with their extensive experience crafting error messages and handling dead ends, can establish clear criteria for the AI's behavior when it doesn't have an answer. The AI should acknowledge its limitations and guide users toward appropriate resources. Third, designers understand the complexity of real users, unlike test sets constructed by AI evaluation teams.

UX researchers are skilled at recruiting participants with diverse communication styles, moods, and backgrounds. Synthetic users used in AI evaluations rarely mirror the complexity of actual users. A study found that 88% of researchers doubted the accuracy of AI-generated personas. Fourth, designers challenge the assumptions underlying AI evaluations.

Typically, these assumptions stem from business goals or easily measurable metrics. For example, a chatbot designed for customer service may assume users prefer automated responses over human assistance. However, this assumption can lead to poor evaluations if users eventually demand human intervention. Designers, who engage with real users, can identify and document such assumptions before evaluations begin.

Lastly, designers can identify design flaws that lead to evaluation failures. When an evaluation fails, engineers and product managers often attempt to tweak prompts or switch models. However, sometimes the root cause lies within the product itself. For instance, a travel assistant booking the wrong date due to ambiguous user input highlights the need for design intervention.

By assessing such issues through usability testing, designers can suggest design improvements, ensuring the AI's answers are more easily verified. In conclusion, while engineers and product managers play crucial roles in AI evaluation, designers bring valuable perspectives that are often overlooked. By involving designers in the evaluation process from the beginning, companies can ensure that the resulting AI features align with user needs and expectations.