When Should a Service Become an AI Product?

In today's rapidly evolving landscape, many services are being transformed into software products, with artificial intelligence (AI) taking on various roles such as AI accountants, recruiters, support representatives, lawyers, and travel agents. While some of these products may succeed, others may struggle or disappear. However, the process of evaluating which services should become AI products is crucial.

The traditional "demo" approach may not be sufficient as advanced models can now generate plausible answers for numerous white-collar tasks. The key question becomes whether someone can effectively check the AI-generated results.

When a service is transformed into an AI product, there is a natural feedback loop in coding, where tests are run quickly, and the outcome is returned almost instantly. However, when dealing with AI models that produce critical outputs, such as pricing a house, recommending a cancer treatment, or making hiring judgments, the timeline for verification can be significantly longer.

The results may appear confident, but the business may wait months to determine if the AI's recommendation was accurate, or the issue might never be fully addressed.

Research indicates that about 80% of US workers could have at least 10% of their tasks affected by language models, while around 19% could see half or more of their tasks impacted. These figures are substantial, but the mere presence of AI exposure does not guarantee the success of a product or the viability of the business economics. To determine the potential of an AI-assisted service, it is essential to evaluate if the model can produce verifiable results that can be checked quickly and cost-effectively.

The BCG experiment conducted by Dell’Acqua and colleagues highlights this challenge. Consultants using AI performed 40% better on tasks that aligned with the tool's capabilities, but when faced with more complex tasks requiring nuanced judgment, the AI's output was 19 percentage points lower compared to human consultants, often without the consultants even realizing it. This demonstrates the importance of having a practical check in place to ensure the AI-generated output is accurate and reliable.

To categorize services effectively, they can be divided into three buckets based on the feasibility of verification. The first bucket consists of functions within a company that have simple, cheap, and quick checks. These services can be readily productized with minimal supervision. Functions such as software engineering, finance operations, first-line support, data extraction, document processing, scheduling, and transcription fall into this category.

The presence of built-in correctness signals in these areas allows AI to automate verifiable steps while leaving the judgment-heavy tasks to human operators.