{
  "id": 3275833,
  "title": "How I Built a Reliable LLM Pipeline for Ad Creative Evaluation (with Strict Pydantic Contracts)",
  "url": "https://urgent.news/2026/08/25/how-i-built-a-reliable-llm-pipeline-for-ad-creative-evaluation-with",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-25T14:02:13.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/strelok25dev/how-i-built-a-reliable-llm-pipeline-for-ad-creative-evaluation-with-strict-pydantic-contracts-376j"
  },
  "original_language": "en",
  "account": "Art directors typically spend 20–40 minutes per creative to verify brand guidelines, mandatory elements, and prohibited techniques. To automate this first-pass review process, the author developed CreativeAudit, a production-oriented pipeline that evaluates ad creatives against a brief and returns a binary PASS / NEEDS_REVISION / FAIL verdict. The system takes a brief and creatives as input, constructs a precise prompt, obtains a structured evaluation from a local LLM, strictly validates the response using Pydantic contracts, and then returns the verdict along with a weighted score and an explanation. This is not merely an AI chatbot; it represents a production-oriented first-pass review pipeline that catches routine and critical violations so that human evaluators only need to focus on borderline cases.\n\nThe key problem with simply asking an LLM for creative evaluations is that the responses are inconsistent in format, the model frequently forgets rules or hallucinates structure, and the output is free-form text rather than machine-readable data. This makes it unsuitable for any real workflow.\n\nTo address these issues, the design goals were set: strict output contracts to ensure invalid responses never become scores, local inference to keep client data within the machine, prompts treated as code for versioning and independent editing, and binary compliance for critical rules. The architecture consists of four main components: Brief + Creatives are input, processed through Jinja2 prompts, sent to a local LLM (Ollama), then validated with Pydantic, and finally produce a Score, Verdict, and Feedback. The business logic is cleanly separated from the interface, enabling it to be called from various services like Streamlit or CLI.\n\nTo ensure reliability, Pydantic is used as a hard contract where the model's non-deterministic responses are checked against strict schemas. For critical brand-book violations, a binary scoring system is employed, assigning a 0 or 10 score instead of allowing subjective interpretations. Prompts are stored in separate Jinja2 files, making them versionable, reviewable, and easily A/B testable without altering the application code. For local inference, Ollama is used, with the option to switch between different models by simply changing the configuration.\n\nThe pipeline also includes a \"smart input mode\" that handles human-in-the-loop for free-form text input. Managers may provide messy JSON or free-form descriptions, so the pipeline first extracts structured data using another LLM, shows the result to the user for correction, and then proceeds with the evaluation. The scoring model assigns three sub-scores: brand_alignment (1–10), constraint_compliance (0 or 10), and message_clarity (1–10). The final score is a weighted combination, with brand alignment and message clarity each carrying 0.3 weight, and constraint compliance with 0.3 weight. The verdict is derived from the total score and the presence of critical failures.\n\nTo validate the system, 38 unit tests were written, all mocking the external LLM to ensure deterministic and fast tests. They cover various scenarios such as schema boundaries, malformed responses, connection failures, and scoring logic. The results showed significant improvements: first-pass review time was reduced from tens of minutes to seconds, output became consistent and machine-readable, and critical brand violations were much less likely to be missed. The main challenge was designing the contracts and failure modes to maintain reliability in the face of potential model errors. Future plans include multimodal support, model benchmarking, a REST API for integration, and evaluation history with campaign analytics.",
  "summary": "Art directors routinely spend 20–40 minutes per creative just checking brand guidelines, mandatory elements, and forbidden techniques. I wanted to automate the first-pass review — without turning it into another unreliable ChatGPT wrapper. So I built CreativeAudit : a production-oriented pipeline that evaluates ad creatives against a brief and returns a clear PASS / NEEDS_REVISION / FAIL verdict.…",
  "key_points": [
    "CreativeAudit pipeline automates ad creative evaluation, returning PASS/NEEDSREVISION/FAIL verdicts",
    "Pydantic contracts ensure strict output format, preventing invalid responses from becoming scores",
    "Binary compliance for critical brand rules, scoring system assigns 0 or 10 for violations"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}