Xinping Lei et al. (Tokenwave.AI) · Org-level coding benchmark

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

SWE-CoWork is a benchmark of 101 development assignments across 101 real repositories, 11 programming languages and 12 application domains. Each task places an agent in a simulated engineering organisation with 26 colleagues who hold distinct, partly hidden requirements, and is scored with private executable tests. Across eleven evaluated models, the two leaders pass about 63% of fail-to-pass tests but complete fewer than 8% of assignments. The paper identifies requirement tracking from clarification through to handoff as a key failure point for autonomous software engineering.

Open the source ↗

Similar Research & Safety releases

  • AscendGYM

    SLAI-AITP (AI Training Platform Team) · Kernel Optimisation Benchmark · Week 48, 2026

  • InclusionMed

    The InclusionMed Collaboration · Medical AI Benchmark · Week 48, 2026

  • Nemotron RL Litmus-Bench V2.0

    NVIDIA · Science Post-Training Dataset · Week 48, 2026

  • VibePrevisBench

    Kling-Keye (Yuzhe Chen, Muxi Diao, Lele Yang, Wenbin Xu, Xuyang Liu, Qile Su, Changyi Liu, Jiankang Chen) · Agent Benchmark · Week 48, 2026

  • NovaAtom-Lite

    Geodesic Intelligence · All-atom structure prediction · Week 47, 2026

  • XYBench

    Ai2 (allenai), with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins · XY problem benchmark · Week 47, 2026

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.