Can We Stop Running Recurring Analysis as a Scheduled LLM Prompt?

Scheduled prompts have become commonplace across major assistants, with users scheduling analysis every Monday at a specific time. While this approach works well for news briefings, it raises concerns when applied to more complex analysis tasks. The issue lies not with the scheduler itself, but rather with the position of the Large Language Model (LLM) relative to it.

The core purpose of recurring analysis is to detect changes over time, not to generate a report. Determining whether the state of things is moving and in which direction requires a stable instrument. When the instrument changes, measuring movement becomes challenging, leading to confounding factors that make it difficult to discern whether the world truly changed or the measurement did.

This results in a fundamental problem of consistency, as even a model that generates correct SQL consistently can produce different correct SQL outputs across runs. The failure mode that often goes unnoticed is the subtle shift in query logic, such as switching from LEFT JOIN to INNER JOIN or altering date boundaries, which can subtly alter results without immediately causing errors.

This silent change can corrupt time series data, making it impossible to reconstruct earlier measurements months later. The proposed solution is to move the LLM from the runtime phase to the build phase. This involves exploring the problem interactively with the LLM, then freezing the metric definition as deterministic code in version control, reviewing the metric change with a human, and finally running the deterministic code.

This approach ensures that the LLM remains useful for interpreting numbers and handling unstructured input, while the metric definition remains stable and can be audited. This approach has already been discussed within the data engineering community, where tools like dbt's Semantic Layer aim to codify metric definitions to prevent subtle drift.

However, the conversation around scheduling AI tasks has occurred in a separate context, leading to a disconnect between the two perspectives. Ultimately, the LLM's role should be limited to interpreting numbers and handling unstructured input, while the computation of metric definitions should be frozen and version-controlled to ensure consistency and auditability.