I built a prompt pipeline that grades itself with an independent AI judge, not its own output

A reporter crafted a prompt pipeline that assesses its own work through an impartial AI judge, rather than relying on the model itself. The challenge involved two common issues with LLM-generated content: forgetting to include necessary sections (compliance language, citations, caveats), and the model providing biased self-evaluations.

The journalist's solution was a sequential pipeline: user input → MCP-grounded research → DSPy reasoning/generation → deterministic validation → independent evaluation → optional closed-loop verification. The MCP Research Module injected live domain knowledge into the model's context before generation, avoiding reliance on outdated training data.

DSPy.ai handled the actual reasoning and generation through declarative, optimizable prompt pipelines. A deterministic QA Linter, a rule-based gate requiring no LLM calls, checked for placeholders, duplicated sections, contradictions, and fabricated citations before the output reached evaluation. An independent evaluation via DeepEval/G-Eval scored the output using an external judge model, avoiding the bias of the model grading its own work.

An Iterative Enhancer generated, critiqued, and refined output within a bounded iteration budget, stopping early when quality targets were met. The pipeline incorporated multi-provider routing to OpenAI, Anthropic, Mistral, Qwen, Cerebras, OpenRouter, Groq, and Gemini with priority fallback, ensuring resilience against provider outages.

The entire system was tested with 215 offline tests, requiring no API keys, and built with GitLab CI/CD, including SAST, secret detection, bandit, and pip-audit scanning. Key design decisions emphasized deterministic gates before the AI judge to catch cheap, fast, zero-cost checks, and avoiding self-scoring to prevent bias. The journalist plans to expand safeguard sections and tighten closed-loop verification conditions.