'That,' 'this page,' and 'that one' are different: 1 win and 3 losses by treating pronouns uniformly
📝 Originally published (in Japanese) at forge.workstyle.tech . An avatar that interacts through voice receives the following utterances: 👤 Are there any articles about streaming on YouTube? → Article identified 👤 Can you explain that in more detail? → Only responds with "Yes, I'll explain in detail." 👤 Wait, you're not explaining in detail? → Only responds with "Sorry, just a little more..."…
A conversational AI was tested on its ability to retrieve articles based on user queries. The system performed inconsistently, winning 1 out of 4 tests and losing 3. The issue stemmed from pronouns like "this" and "that" referring to different things depending on the context. When the AI was given the user's previous utterance, along with the current one, it only correctly identified the article 2 out of 4 times.
Adding only the relevant exchange from the conversation improved the accuracy to 3 out of 4 cases. However, including the entire conversation history diluted the search query, reducing the correct answer to 3 out of 4. The study also found that "in detail" was often truncated in user queries, even though its presence indicated a request for more information. This truncation led to a third of cases being missed.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
