{
  "id": 4200650,
  "title": "We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model.",
  "url": "https://urgent.news/2026/08/29/we-ran-160-agent-tasks-across-two-frameworks-the-frameworks-tied-then",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-29T13:27:52.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/benchclaw/we-ran-160-agent-tasks-across-two-frameworks-the-frameworks-tied-then-we-changed-the-model-3hip"
  },
  "original_language": "en",
  "account": "160 agent tasks were executed across two frameworks, LangGraph and Pydantic AI, in a controlled benchmark. Surprisingly, the frameworks tied with 100% correctness on all four tasks - order processing with tool calls, shipping quote calculation, refund eligibility with date logic, and inventory reorder decisions. However, the model used, gpt-4o, caused one task, refund-policy-minimal-tools, to fail 20 times in a row. This task required precise date arithmetic for calculating return eligibility. The gpt-4o-mini model produced the same incorrect answer 20 out of 20 times for this task, indicating a systematic failure in its date arithmetic. This failure was not due to the framework but the model itself. The benchmark suggests that the framework choice may not significantly impact correctness for structured tasks, but the model choice can have a significant impact, especially for tasks involving temporal reasoning. The finding highlights the importance of model validation in production, as changing models can drastically alter outcomes.",
  "summary": "We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model — and one task failed 20 times in a row. The LangChain vs LangGraph question comes up every time someone starts a new agent project. We had a slightly different version of it: once you've chosen LangGraph as your runtime, does picking the library around it actually change production outcomes? So we ran a…",
  "key_points": [
    "160 agent tasks executed across LangGraph and Pydantic AI frameworks",
    "Both frameworks tied with 100% correctness on all tasks",
    "gpt-4o model caused refund-policy-minimal-tools task to fail 20 times"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}