Amazon Web Services · Model Leaderboard

Amazon SageMaker JumpStart Model Leaderboard

AWS has launched a leaderboard view in SageMaker Studio for comparing JumpStart's open-weight LLMs side by side. At launch it covers 70+ models from providers including Meta, Google, Qwen, IBM and Arcee. Quality is scored on BIG-Bench Hard, GPQA Diamond and MMLU-Pro, run by AWS with one shared harness, while speed (time to first token, tokens per second) and instance cost are measured on SageMaker endpoints for each deployment configuration. Cost figures cover SageMaker instance cost only.

Open the source ↗

Similar Developer Tools releases

More from Amazon Web Services

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.