The Systems Around the Model: What Enterprise Architecture Gets Wrong

In the world of enterprise AI, one topic that continually emerges is the selection of models. Teams invest significant time debating the merits of various models, such as GPT versus Claude, open versus closed source, and the potential benefits of larger context windows. However, these questions often take a backseat to a more critical consideration: whether the assistant can access the necessary information to provide reliable answers.

In enterprise settings, information is typically scattered across multiple sources, including API specifications, internal services, configuration files, runbooks, product documentation, tickets, databases, and even knowledge held by experienced engineers. The challenge lies in the fact that these sources do not evolve at the same pace.

An API specification might be updated, while the associated documentation still reflects outdated behavior. A configuration file may contain the latest information, yet an internal wiki page may still reference an older setup. Two seemingly official sources can still present conflicting information.

While large language models (LLMs) can reason over the provided information, they do not inherently resolve issues related to outdated, incomplete, duplicated, or contradictory data. Even a highly sophisticated model can generate confident yet incorrect answers when faced with such flawed inputs. This realization has prompted a shift in how enterprise AI architectures are evaluated.

Instead of solely focusing on the intelligence of the model, attention is now directed towards other crucial questions. For instance, where did the answer originate? Is the source still actively maintained? Who is responsible for it? How frequently does it change? How can its correctness be verified? What happens when multiple sources provide conflicting information?

These considerations bear a striking resemblance to traditional software engineering concerns. As AI systems become more autonomous, capable of interacting with real enterprise systems, traditional engineering principles such as identity, permissions, auditability, observability, rollback strategies, error handling, and failure recovery regain prominence. The AI component does not eliminate these concerns; in fact, it often amplifies their importance.

Trust has emerged as a critical aspect of enterprise AI. Users not only require accurate answers but also need confidence in how those answers were generated. A robust enterprise assistant should be able to trace its response back to reliable sources and provide a clear understanding of the information that influenced the answer. In sensitive scenarios, there should be a transparent record of the system's actions, the rationale behind them, and the entities responsible for authorizing those actions.

Another often underestimated aspect is the maintenance cost of AI projects. Building a convincing prototype in a short period is feasible, but maintaining the system accurately and effectively over an extended period presents a different challenge altogether. As business rules evolve, teams change, authentication mechanisms shift, new data sources are introduced, and old ones are retired, the system can gradually lose reliability if the information surrounding it is not actively maintained.

Therefore, enterprise AI should be treated as a living software system, with clear ownership of data sources, understanding of authoritative sources, tracking of changes, monitoring of retrieval quality, regular review of permissions, and implementation of observability measures. The long-term success of an enterprise AI system may hinge more on operational discipline than on model sophistication.

When contemplating model upgrades, it is essential to recognize that a superior model alone does not automatically rectify broken context, outdated documentation, poor retrieval, weak permissions, or missing observability. Sometimes, the most significant improvement stems from addressing the information pipeline surrounding the model rather than merely introducing a more advanced language model.

This perspective underscores that the AI model is merely one part of a larger engineering ecosystem responsible for context, security, orchestration, governance, integration, monitoring, and reliability. Many of the most challenging engineering decisions are situated within this surrounding layer. Consequently, when assessing an enterprise AI architecture, it is crucial to consider not only the model being utilized but also the extensive engineering infrastructure that supports it.