Anthropic blog post

In a recent blog post, Anthropic discusses the challenges and best practices for enabling self-service data analytics using large language models (LLMs), specifically their own AI model Claude. The post highlights how traditional methods of making data more accessible to non-technical employees often lead to inconsistencies and bloat. However, LLMs offer a new path for self-service analytics that can avoid these issues.

Anthropic found that 95% of their business analytics queries are automated via Claude, with ~95% accuracy overall. By delegating routine tasks to Claude, the data science team can focus on strategic work such as causal modeling, forecasting, and machine learning. The post outlines several best practices for maximizing Claude's effectiveness in driving self-serve business insights.

One key point is that LLMs' generative abilities can also lead to hallucinations, meaning they may produce erroneous outputs. This is similar to the challenges faced with coding agents, where creativity can sometimes result in incorrect solutions. However, unlike coding, analytics use cases often have a single correct answer, making it more difficult to prove correctness.

The post identifies three main attributes that contribute to inaccurate responses in self-service analytics: ambiguity in mapping user questions to specific data entities, staleness of data, and the lack of clear documentation. To address these issues, Anthropic built an agentic data stack with multiple layers.

The first layer focuses on strong data foundations, including data models, transforms, tests, and tables in the data warehouse. Standard data engineering practices such as dimensional modeling, shift-left testing, freshness and completeness checks are still crucial. However, the end user of the data model is now an agent acting on behalf of various users with different levels of data expertise.

The second layer consists of sources of truth, which are reference surfaces the agent consults to navigate the data warehouse. This layer helps reduce ambiguity and turns vague terms like "weekly active users" into specific, governed entities within the data model. The sources of truth are primarily maintained by humans, who curate and own the documentation.

The third layer deals with the procedural knowledge of the agent, which includes how to consult the sources of truth in the correct order, navigate ambiguous data, and understand what a finished analysis should look like. At Anthropic, they use Claude Code, where skills are folders of markdown that the agent reads on demand. These skills significantly improve Claude's accuracy in answering analytics questions, with accuracy rates consistently above 95% and often reaching 99% in certain domains.