Track AI Prompt Changes with a Run Manifest
An AI evaluation improves after a prompt edit. A week later, a teammate cannot reproduce the comparison. The prompt file is available, but the retrieved note changed, a model alias may point somewhere else, and nobody recorded the adapter revision. There is an answer on disk without enough context to explain its origin. Create a run manifest that identifies the actual request inputs,…
A run manifest is a structured record detailing an AI evaluation attempt, ensuring traceability. It identifies the exact request inputs, configuration, source snapshot, and evaluator revision. However, a prompt filename alone is insufficient, as it does not capture the actual messages sent, potential variable substitutions, earlier message retention, reordered retrieved passages, or schema changes.
To record a comprehensive configuration, one must include the resolved model identifier, provider or endpoint configuration, generation settings, rendered messages, ordered context, tool definitions, adapter revision, and evaluation rubric revision. Immutable references or protected artifacts should be used for large inputs, keeping event metadata separate from configuration identity.
Two attempts can share the same configuration but possess different run IDs, timestamps, outputs, and durations. To ensure a stable byte representation, Python's json module can be used with sorted object keys and explicit separators, while hashlib provides SHA-256 to generate a consistent digest. The provided Python script demonstrates this process, generating a manifest_id from a given manifest and validating JSON values for consistency.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
