The Lakehouse Wasn’t Built for AI Agents

The lakehouse, a pattern for converting raw operational data into trustworthy business intelligence assets, has been the focus of significant development over the past few years. However, recent experiences suggest that the current implementation of lakehouses may not be suitable for autonomous agents capable of planning, executing tasks, and making decisions independently.

Traditional lakehouses were designed with human users in mind, where someone interacts with a BI tool, schedules database transformation jobs, and monitors the outcomes. Even real-time lakehouses that stream data into databases still assume a human to interpret the results. The flow of data in a lakehouse is typically linear, progressing from sources to progressively cleansed data, landing in a location where humans can examine the information.

Governance is primarily concerned with cataloging and access policies, rather than actively reviewing data in real-time.

When it comes to autonomous agents, the existing lakehouse architecture falls short. Simply granting an agent access to the gold layer of data might work for demonstrations, but it fails when dealing with multiple definitions of the same concept, as often encountered in enterprise warehouses. An agent without a shared semantic layer may combine incompatible interpretations of data and provide inaccurate results confidently.

To support autonomous agents, a lakehouse needs additional functionality. A semantic layer that enforces consistent definitions of data is crucial, as current implementations are often merely documented and not enforced. A context store, typically based on vector-based technology, is necessary for agents to retrieve relevant information such as documentation, lineage, and prior decisions.

Additionally, a tool layer that mediates access to data is required to prevent agents from executing arbitrary queries against raw storage, maintaining security and governance. Lastly, a feedback loop is essential, as corrections made by agents must be incorporated back into the platform, rather than being lost in chat logs.

Implementing these features transforms the lakehouse from a pipeline to a more complex system, resembling a small nervous system. This evolution raises essential governance questions, such as who approves schema changes initiated by agents and how conflicts between specialized agents are resolved. These issues are not merely hypothetical and become apparent as the system moves beyond a proof of concept.

The proposed solution involves introducing a human-in-the-loop checkpoint at various stages of the process, ensuring approval for irreversible or high-cost actions. While this approach slows down the system intentionally, it is considered the right trade-off until the industry gains more experience with autonomous agents. However, it is essential to acknowledge the additional costs associated with vector stores, orchestration layers, and always-on agent loops, which may not be necessary for organizations primarily dealing with issues like stale dashboards. Instead, they should focus on improving scheduling and implementing a semantic layer.