AIF-C01: ROUGE, BLEU, BERTScore or LLM-as-a-judge? Evaluating foundation models the way the exam asks
Domain 3 of the AWS Certified AI Practitioner exam (AIF-C01), "Applications of Foundation Models", is 28% of the scored content. One of its task statements is all about evaluation: how do you know a foundation model (FM) is actually good enough? The exam guide names the tools explicitly, so they are worth learning properly. Three ways to evaluate Automatic metrics on a dataset : fast, cheap,…
The AWS Certified AI Practitioner exam Domain 3 focuses on applications of foundation models and accounts for 28% of the scored content. One key task is evaluating the quality and suitability of these models. The exam guide outlines three primary methods for this: automatic metrics, human-in-the-loop evaluation, and using another LLM as a judge.
Automatic metrics are cost-effective, quick and repeatable, comparing model outputs to reference answers. Human evaluation is more time-consuming and expensive but is necessary for assessing qualities like tone, brand voice or helpfulness. Another approach is using a second LLM to score outputs based on predefined criteria, which scales better for subjective quality.
Standard public datasets serve as benchmark for comparing models on the same tasks. Common metrics include ROUGE for summarization (recall), BLEU for machine translation (precision), and BERTScore for semantic similarity. F1 or accuracy is used for question answering and classification tasks. Bedrock, AWS's managed service for foundation models, offers automated evaluation jobs using built-in datasets, measuring accuracy, robustness and toxicity across various task types. Bedrock also supports human evaluation for criteria that metrics cannot capture.
While choosing the right evaluation method depends on the specific application, it's important not to confuse a good metric with a good business result, or to rely solely on human evaluation when a cheaper automatic method would suffice. Additionally, assessing the performance of applications built on foundation models, like retrieval-augmented generation (RAG) chatbots or agent workflows, is crucial.
A sample question might ask a candidate to compare three foundation models for summarizing customer support tickets, requiring a quick and low-cost method to measure how well each model preserves the meaning of a reference summary, even when different words are used. The correct answer in this case would be B. Use automatic evaluation with BERTScore.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.