Urgent.News

What's breaking now, across thousands of outlets.

AI

Introducing the Amazon SageMaker JumpStart Model Leaderboard

The Amazon SageMaker JumpStart Model Leaderboard compares 70+ foundation models side by side on quality, speed, and cost, directly in SageMaker Studio. Learn what the leaderboard shows, how AWS benchmarks models, and how to go from a shortlist to a deployed model.

Amazon SageMaker JumpStart introduces the Model Leaderboard, a tool for comparing open-weight foundation models side by side based on quality, speed, and cost. This new view of the JumpStart base models page in SageMaker Studio provides rankings and information without requiring setup questions. The leaderboard includes quality scores from three public benchmarks: BIG-Bench Hard, GPQA Diamond, and MMLU-Pro, as well as performance and cost metrics measured by AWS.

At launch, the leaderboard features over 70 models from providers like Meta, Google, Qwen, IBM, and Arcee. The key concepts defined in the leaderboard are quality benchmarks, use case profiles and optimization goals, and performance metrics. Quality benchmarks include three reasoning tests that measure different aspects of an LLM's capabilities.

Use case profiles and optimization goals allow users to customize the metrics shown based on their specific workload and deployment configuration. Performance metrics include time to first token, throughput per request, and instance cost. It's important to note that the estimated cost only reflects SageMaker instance cost and does not include additional factors like data transfer or storage.

Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at aws.amazon.com →

More in AI

More from Wednesday 18 November →