I Built 50+ AI Products in 4 Years — Here's What I Wish I Knew at the Start

My journey building over 50 AI products for Autor has taught me valuable lessons that I wish I had known at the beginning. Initially, I believed that our first AI products should be ambitious and innovative. However, I've come to understand that boring, reliable products are more important. The most successful product, Loquent, handles healthcare scheduling calls without any exotic architecture.

Instead, it focuses on understanding the caller's needs and booking or rescheduling appointments. This simplicity is what makes it reliable enough to handle thousands of automated calls per month without requiring constant babysitting in production.

Another crucial lesson is that prompt engineering is real engineering. Early on, I treated prompts as mere configuration, but now I recognize that they require version control, testing, rollback plans, and monitoring. A change in model behavior can impact prompts, and therefore, they need to be robust against such variations. Each prompt at Autor goes through a rigorous review process, similar to application code, with extensive test suites to ensure robustness.

It also became apparent that the model is only 20% of your production system. The remaining 80% includes monitoring and observability, integration plumbing, edge case handling, deployment infrastructure, and more. These components are as crucial as the model itself. For instance, monitoring and observability involve logging every call to identify potential issues before they affect users.

Integration plumbing connects the AI system to various external services, each with its own failure modes and quirks. Edge case handling addresses situations where the system encounters unexpected inputs or scenarios, such as callers speaking different languages or the clinic's API being down. Proper deployment infrastructure, like zero-downtime deployments and canary releases, is essential for maintaining system stability.

I also learned that it's vital to ship weekly, not just when your product is perfect. Spending too much time perfecting a product can lead to delays in shipping. The team that ships a mediocre version early often ends up with a solid product by the end, while those who perfect their initial product may remain in the development stage indefinitely.

Charging what production AI actually costs is another critical lesson. Early on, I offered significant discounts to clients, leading to inadequate investment in proper monitoring and eventual churns. Production AI costs are not just about the model API; they include senior engineers maintaining the system, ensuring reliability, and providing excellent user experience.