LLM Evaluators Show Systematic Language Bias, Study Finds

Jul 17, 2026

A new preprint reveals that large language model (LLM) evaluators assign systematically different scores to semantically identical content across 23 languages, with lower-resource languages receiving more generous evaluations. This bias is consistent across eight open-weight evaluators and frontier models, and is not detected by standard pairwise accuracy metrics. As a result, harmful content in lower-resource languages is more likely to pass safety filters under global thresholds.

Why it matters: This finding exposes a structural flaw in multilingual AI safety evaluations, potentially allowing harmful content to evade detection in lower-resource languages.

Full story at: arXiv Computation and Language ↗