Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander Golubev (Nebius) · Long-tail coding benchmark

Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

Longtail-SWE is a benchmark of 202 issue-resolution tasks across 13 underrepresented languages, including Clojure, Dart, Elixir, Haskell, Julia, Kotlin and Swift. Its tasks are built with a solver-judge-repairer pipeline that revises tests and descriptions to fix tests that reject valid solutions or accept incorrect ones. The authors evaluate nine model configurations, including frontier models, and release the instances, executable environments, tooling and agent trajectories, plus a set of easier tasks for smaller models.