Why 88% of AI pilots never make it to production

A recent IDC and Lenovo study reveals that 88% of AI proof-of-concept projects fail to transition to production due to unclear objectives, insufficient data preparation, and a shortage of in-house agentic AI expertise. This leads to a build-vs.-run dichotomy, where the team responsible for designing and deploying AI agents does not have the opportunity to run them through business processes.

When the operating team, often experienced with demo usage, employs the agentic system without external assistance, errors can occur. Operational gaps may result in cascading failures and disastrous consequences if a single agent deviates beyond permissible behavior limits. Serhii Semenchenko, CTO of Dedicatted, a Toronto-based AWS Premier Tier Partner for Agentic AI Specialization, shares his insights with The New Stack.

He explains that demo environments are clean, with prepared data and defined input and output, while production environments are unpredictable. He emphasizes the importance of logging, tracing, monitoring, breakers, and validation to ensure failure preparedness. Semenchenko proposes a production checklist for agentic AI systems, which includes timeouts for every agent and logging for traceability.

He also compares agentic AI systems to microservices architecture, where each agent has its own set of permissions and short-lived access keys. This approach prevents data leaks, unintended use, and prompt injection attacks. Semenchenko further explains the concept of multi-agent orchestration using a router that gathers inputs from users, systems, and platforms, making decisions on what to do with input data.

If the final step fails, the composer does not create missing pieces; it retries or marks the failure. To prevent memory poisoning, Semenchenko discusses the use of "guardrails," deterministic checks on all documents and data. He highlights a successful case where Dedicatted built an agentic AI system for CASSEN Testing Laboratories, a Canadian laboratory facing multi-day turnaround times for reports due to manual regulatory testing.

The team adjusted the flow and agents to produce reports similar to those prepared manually by chemists. When migrating from AWS to Azure for Zivian Health, a U.S.-based compliance infrastructure, Semenchenko emphasizes the importance of mapping, adjusting, and validating permissions and monitoring. He stresses the need for a validation suite to ensure the same success rate, latency, and token cost as before migration.