Enterprise AI Has a Safety Problem That Better Models Alone Can't Fix

The AI industry has been debating model safety for two years, but enterprises continue to deploy unregulated systems around these models. Model safety is a component of a larger system, and the system ultimately determines what the model can do, what changes it can make, and whether anyone will notice. The debate about model safety does not address the fundamental question of whether the entire system is safe.

A safe model does not guarantee a safe system. The core issue is that the system, not the model, is what gets shipped. Safety should be evaluated at the system level, not just the model level. The debate has focused on the wrong layer. The debate has centered on the model, but the system decides what the model can access, what changes it can make, and whether anyone will ever know what happened.

It's like having a safe engine but installing it in a car with no brakes or steering wheel. The engine is safe, but the car is a hazard. This is happening with enterprise AI systems, where teams can provide extensive details about their model's safety scores but know very little about the system's behavior. The safety conversation has given teams confidence in the wrong layer.

To be considered governable, a system must meet four concrete properties: it must be observable, constrainable, reversible, and accountable. If any of these properties are missing, the system is not governable and should not be shipped. The five-question test can help determine whether a system is governable. These questions focus on observability, constraintability, reversibility, and accountability.

The answers to these questions reveal whether a system is truly safe or merely a well-aligned accident waiting to happen.