{
  "id": 20302588,
  "title": "AIF-C01: ROUGE, BLEU, BERTScore or LLM-as-a-judge? Evaluating foundation models the way the exam asks",
  "url": "https://urgent.news/2026/11/16/aif-c01-rouge-bleu-bertscore-or-llm-as-a-judge-evaluating-foundation",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-16T17:23:18.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/getnivaro/aif-c01-rouge-bleu-bertscore-or-llm-as-a-judge-evaluating-foundation-models-the-way-the-exam-449n"
  },
  "original_language": "en",
  "account": "The AWS Certified AI Practitioner exam Domain 3 focuses on applications of foundation models and accounts for 28% of the scored content. One key task is evaluating the quality and suitability of these models. The exam guide outlines three primary methods for this: automatic metrics, human-in-the-loop evaluation, and using another LLM as a judge. Automatic metrics are cost-effective, quick and repeatable, comparing model outputs to reference answers. Human evaluation is more time-consuming and expensive but is necessary for assessing qualities like tone, brand voice or helpfulness. Another approach is using a second LLM to score outputs based on predefined criteria, which scales better for subjective quality.\n\nStandard public datasets serve as benchmark for comparing models on the same tasks. Common metrics include ROUGE for summarization (recall), BLEU for machine translation (precision), and BERTScore for semantic similarity. F1 or accuracy is used for question answering and classification tasks. Bedrock, AWS's managed service for foundation models, offers automated evaluation jobs using built-in datasets, measuring accuracy, robustness and toxicity across various task types. Bedrock also supports human evaluation for criteria that metrics cannot capture.\n\nWhile choosing the right evaluation method depends on the specific application, it's important not to confuse a good metric with a good business result, or to rely solely on human evaluation when a cheaper automatic method would suffice. Additionally, assessing the performance of applications built on foundation models, like retrieval-augmented generation (RAG) chatbots or agent workflows, is crucial. A sample question might ask a candidate to compare three foundation models for summarizing customer support tickets, requiring a quick and low-cost method to measure how well each model preserves the meaning of a reference summary, even when different words are used. The correct answer in this case would be B. Use automatic evaluation with BERTScore.",
  "summary": "Domain 3 of the AWS Certified AI Practitioner exam (AIF-C01), \"Applications of Foundation Models\", is 28% of the scored content. One of its task statements is all about evaluation: how do you know a foundation model (FM) is actually good enough? The exam guide names the tools explicitly, so they are worth learning properly. Three ways to evaluate Automatic metrics on a dataset : fast, cheap,…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "Scientific Reports",
        "title": "Evaluating transformer-based models for predicting metric-based research impact across scientific and medical literature",
        "url": "https://urgent.news/2026/11/14/evaluating-transformer-based-models-for-predicting-metric-based",
        "published": "2026-11-14T00:00:00.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}