Xinping Lei et al. (Tokenwave.AI) · Org-level coding benchmark
SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
SWE-CoWork is a benchmark of 101 development assignments across 101 real repositories, 11 programming languages and 12 application domains. Each task places an agent in a simulated engineering organisation with 26 colleagues who hold distinct, partly hidden requirements, and is scored with private executable tests. Across eleven evaluated models, the two leaders pass about 63% of fail-to-pass tests but complete fewer than 8% of assignments. The paper identifies requirement tracking from clarification through to handoff as a key failure point for autonomous software engineering.
Open the source ↗Similar Research & Safety releases
- AscendGYM
SLAI-AITP (AI Training Platform Team) · Kernel Optimisation Benchmark · Week 48, 2026
- InclusionMed
The InclusionMed Collaboration · Medical AI Benchmark · Week 48, 2026
- Nemotron RL Litmus-Bench V2.0
NVIDIA · Science Post-Training Dataset · Week 48, 2026
- VibePrevisBench
Kling-Keye (Yuzhe Chen, Muxi Diao, Lele Yang, Wenbin Xu, Xuyang Liu, Qile Su, Changyi Liu, Jiankang Chen) · Agent Benchmark · Week 48, 2026
- NovaAtom-Lite
Geodesic Intelligence · All-atom structure prediction · Week 47, 2026
- XYBench
Ai2 (allenai), with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins · XY problem benchmark · Week 47, 2026
Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.