Docs-as-Evals: The New Job Technical Writers Didn’t Expect
When I started my career in technical writing, the requirements and expectations were simple: learn the product, talk to some SMEs or engineers, get the job done by completing the documentation work, and work on the collected feedback. But with AI coming in, it doesn’t feel the way it used to. The role is shifting dramatically, and so are expectations. But why? If you maintain a dashboard for…
The role of technical writers has evolved dramatically with the advent of AI in documentation. While they used to simply learn the product, converse with engineers and SMEs, and deliver documentation, the new expectation is that they assess the quality of documentation through AI evaluations. This concept is known as "docs-as-evals."
The shift in the role of technical writers is marked by the increasing reliance on AI agents to read and respond to documentation, which have become a more significant source of information than real humans. However, these AI agents can sometimes provide incorrect or nonsensical answers, and the technical writers are now tasked with evaluating these responses.
One major issue identified is that when AI agents answer incorrectly from documentation, it's the technical writer's responsibility to ensure the accuracy of the response. This leads to a new set of challenges, particularly in multi-product systems where RAG (Retrieval-Augmented Generation) does not always provide correct answers.
Instead, it often generates answers that sound correct but are factually inaccurate. The technical writers have developed a way to test whether an AI-generated answer is correct or not by using real user questions and evaluating parameters such as whether the response stays within the correct product, whether the citations are accurate, and whether the conversation history is kept or deleted.
The writers have compiled a list of real user questions and defined what constitutes a correct answer for each one, creating a set of standards to judge the AI's performance. By using this approach, they aim to improve the quality of documentation and ensure that AI-generated responses are accurate and relevant.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
