Back to news
arXiv cs.CL · 2026-10-22 00:00 UTC
research

SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges

arXiv:2610.28886v1 Announce Type: new Abstract: LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion. Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-as

Why it matters

Metamorphic testing for LLM judges helps detect scoring fragility and gaming, improving reliability of automated evaluations used in model selection.

Read the original story

Published to Cognify News · Week 43, 2026