Specific Labs · Enterprise Coding Benchmark

Real-SWE

A coding benchmark run against private real-world enterprise codebases and production tasks rather than public or synthetic repositories, using native harnesses meant to mirror how enterprise engineers actually work, scored as pass@1 averaged over eight independent runs per task. Across eight frontier models the top result is Claude Fable 5.1 at 38.8% and the lowest GPT-5.6 Sol at 16.2%, at estimated per-task rollout costs from $2.50 for Gemini 3.8 Flash to $6.96 for Fable 5.1. It also tracks a failure taxonomy covering unverified assumptions, missed requirements, integration errors, regressions and wrong file placement. Self-reported by Specific Labs.