Samaya AI · Finance agent benchmark
FrontierFinance Benchmark Grader
Samaya AI has released a rubrics-based LLM grader for its FrontierFinance benchmark, which measures finance agents. For each query, a panel of judge models decides by majority vote whether each rubric is satisfied by a response you supply, and the tool reports macro and micro qualification rates, including a must-have subset and breakdowns by rubric type and data source type. The recommended judges are Claude Sonnet 4.6, Gemini 3.1 Pro and GPT 5.4, chosen for high agreement with a committee of 9 judges. The rubrics dataset is on Hugging Face under CC-BY-4.0; the grader runs through the official Anthropic, OpenAI and Gemini SDKs.
Open the source ↗Similar Developer Tools releases
- AWS Emulator
AWS · Agent-built AWS emulator · Week 47, 2026
- UAEF (Universal Agent Evaluation Framework)
AWS Machine Learning · Agent Eval Framework · Week 47, 2026
- Token Pooling-Aware Training for Multi-Vector Embedding Models
Hugging Face (Tom Aarsen) · Pooling-aware embeddings · Week 47, 2026
- Gemini Decisions API (Gemini Decision 1)
Google DeepMind · Multimodal Decision API · Week 47, 2026
- WorldFoundry
OpenEnvision (Juanxi Tian, Jiadong Hou, Baiang Li, Xin Lin, Zhengbo Jiao, Weijia Zhang, Yeying Jin, Truong Nguyen) · World model infrastructure · Week 47, 2026
- verifiers
Prime Intellect · LLM RL environments · Week 45, 2026
Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.