Can You Trace an AI Decision Back to the Data That Created It?

The tale of tracing an AI decision back to its data roots reveals the challenges faced by enterprise systems. After a decade in building data and analytics platforms, a healthcare analytics program exposed a critical issue during a readmission-risk model's operation. Instead of the model itself, the problem resided four layers back in a semantic layer metric that had drifted after an unnoticed upstream schema change.

Identifying the issue required tracing the decision path from data to context to reasoning to recommendation to action. Most teams typically only address the model's output, recommendation, and action. However, this approach overlooks the crucial layers that contributed to the final decision. When a model produces a recommendation, the true question should be "where did this come from?" Most teams can provide information about the training data or final output but struggle to trace the complete path.

Tracing entails answering five key questions for a specific decision: data inputs and their versions, surrounding context, logic and rules, the system's recommendation, and the subsequent action. Most postmortals only address reasoning and recommendation, as these are the easiest to log. Data and context are often inferred afterward, leading to potential misremembering. Action rarely gets tied back to its original recommendation.

A case study exemplifies the tracing process, starting with the action: a cohort of patients was incorrectly excluded from post-discharge follow-up due to low-risk scores. To reconstruct the past, they needed to trace back through the reasoning: the model showed a risk score below the threshold, and the reasoning revealed that a rolling comorbidity-burden metric within the model had been flattened.

The context indicated that the issue stemmed from an upstream schema change that resulted in fewer matches in a diagnosis-history table, which was not clearly reflected in the dashboards.

The root cause lay in an upstream source system altering diagnosis codes during a routine schema update, a change that went unnoticed. The problem spanned multiple layers, demonstrating that failures often arise from seams between different components, where ownership is fragmented. Versioning inconsistencies across the stack, ephemeral context, and non-inspectable reasoning further complicate the tracing process.

This story underscores the importance of tracing AI decisions back to their data roots. By understanding the complete path from data to action, teams can identify and address issues that may not be immediately apparent. It is a vital skill for teams shipping AI-driven decisions, particularly those built on enterprise analytics platforms, to ensure accountability and avoid costly mistakes.