Apple Machine Learning Research (Natalie Mackraz, Andrew Silva, Aini Putkonen, Brihi Joshi, Joelle Alcaidinho, Arnav Arora, Barry-John Theobald, Katherine Metcalf) · LLM Judge Study

Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment

The paper examines how well LLM judges assess whether generated text matches a user's preferred writing style. The authors generated text with four LLMs across three long-form writing tasks and compared human pairwise labels with judge-LLM annotations. They report that judge quality correlates weakly with general ability on MMLU (Pearson's ρ ≤ 0.2) and varies by task and criteria, and that paired comparisons are more consistent than independent scoring. The paper was accepted at the Can We Trust AI Evaluation? Workshop at NeurIPS 2026.