📌 Highlights
The paper reframes translated-benchmark evaluation as a proxy-validity problem: the key question is whether a translated score predicts native performance after accounting for model size and target-language proficiency.
Across seven pairings, the evidence is heterogeneous. Skolprov has the strongest incremental result with R²=.128 and a 95% CI of [.038, .253], while SwedishFacts has no significant incremental value in the main specification (p=.242).
The central empirical pattern is content overlap rather than cultural sensitivity alone. Curriculum-related benchmarks can transfer well, and some culturally loaded benchmarks remain robust, whereas Sweden-specific factual knowledge is poorly captured.
The principal caveat is that SwedishFacts is the only open-ended benchmark, so the design cannot isolate local content from response format.
- 29 models, seven native-versus-proxy pairings, five target languages.
- Four robust, two marginal, and one null incremental-validity result.
- Ranking agreement ranges from τ_b=.89 for Skolprov to τ_b=.51 for SwedishFacts.
- Native benchmarks remain necessary when the target capability is strongly local.
🎯 Introduction
Multilingual LLM evaluation frequently translates English-origin benchmarks such as MMLU into other languages. The paper argues that translation can carry cultural and linguistic biases, introduce artifacts, alter difficulty and rankings, and correlate less reliably with human judgments than native benchmarks.
The authors distinguish their question from whether machine translation is adequate as a translation channel. They ask whether a translated benchmark measures enough of the same evaluation signal as a native benchmark to substitute for it after controlling for general model strength.
The study contributes an incremental proxy-validity framework, a multilingual annotation pipeline for cultural sensitivity and required knowledge, and an empirical comparison across seven native-versus-proxy pairings. Its objective is to identify when translated evaluation is informative and when locally grounded native tests are required.
🔬 Methodology
The comparison covers Estonian, Swedish, Danish, Norwegian Bokmål, and Lithuanian. Translated proxies come from EU20-MMLU for Estonian, Lithuanian, Swedish, and Danish, and from m_mmlu for Norwegian Bokmål. The native sources are Exam-ET, INCLUDE-ET, Skolprov, SwedishFacts, DanishCitizenTest, NRK-Quiz, and INCLUDE-LT.
The evaluation uses 29 instruction-tuned LLMs across four size categories and runs through the LM Evaluation Harness with custom native-benchmark tasks. Belebele scores in each target language provide a language-proficiency control. The authors also construct G-MMLUX, a unified translated subset aligned with culturally annotated GlobalMMLU items.
For incremental validity, the baseline model predicts native score from model size and Belebele score, while the extended model adds the same-language MMLU-proxy score. Nested F-tests, 5,000 model-level bootstrap resamples, and 17 leave-one-family-out refits determine whether the proxy contribution is positive, stable, and statistically supported.
Cultural analysis uses three independent native-language annotations for 100 items per language, producing the Native-500 reference. GPT-5.2 is calibrated on GlobalMMLU-Lite and evaluated on both the held-out GlobalMMLU-Lite split and Native-500. The binary CS/CA pipeline reaches 90.6% accuracy against the native majority, while fine-grained required-knowledge labels are more difficult.
tab:overview_comparison-pairings
Caption: Dataset overview and native-vs-proxy comparison pairings. Each native benchmark is paired with the DeepL-translated MMLU in the same language (G-MMLUX subset). Proxies sourced from EU20-MMLU for ET, LT, SV, DA and from m_mmlu for NB.
| Native (Gold) | n | MMLU-Proxy | n |
|---|---|---|---|
| Exam-ET | 1,614 | ET | 2,800 |
| INCLUDE-ET | 224 | ET | 2,800 |
| Skolprov | 545 | SV | 2,800 |
| SwedishFacts | 1,292 | SV | 2,800 |
| DanishCitizenTest | 668 | DA | 2,800 |
| NRK-Quiz | 3,600 | NB | 2,701 |
| INCLUDE-LT | 534 | LT | 2,800 |
Why It Matters: This table defines the seven comparison pairings and shows the different native-benchmark sample sizes that affect the evaluation design.
Nested proxy-validity models
Meaning: M1 predicts native benchmark performance y_L from model size s and Belebele language-proficiency performance b_L. M2 adds the translated-proxy score x_L; improvement from M1 to M2 is the basis for the nested F-test and incremental proxy-validity claim.
📊 Experiments
The main comparison includes Exam-ET, INCLUDE-ET, Skolprov, SwedishFacts, DanishCitizenTest, NRK-Quiz, and INCLUDE-LT, each paired with a same-language DeepL-translated MMLU proxy. The broader evaluation also uses G-MMLUX, English G-MMLUX for sensitivity control, Belebele for language proficiency, GlobalMMLU-Lite for annotation calibration, and Native-500 for native-language annotation validation.
Ranking agreement is strongest for Skolprov (τ_b=.89, top-5 overlap 5/5), Exam-ET (.88, 5/5), INCLUDE-LT (.87, 5/5), and NRK-Quiz (.85, 4/5). DanishCitizenTest reaches .82 with 3/5 top-5 overlap, INCLUDE-ET reaches .72, and SwedishFacts reaches only .51 with 3/5 top-5 overlap. All reported ranking p-values are below .001.
The stronger incremental analysis classifies four pairings as robust, two as marginal, and one as null. Skolprov has the largest incremental R² (.128 [.038, .253]); INCLUDE-LT, NRK-Quiz, and Exam-ET are also robust. INCLUDE-ET and DanishCitizenTest are marginal, while SwedishFacts has a non-significant full-sample F-test and a bootstrap coefficient interval containing zero.
The leave-one-family-out analysis preserves significance for all four robust pairings, but DanishCitizenTest and INCLUDE-ET lose nominal significance under some family exclusions. Adding English G-MMLUX as a third control leaves six pairings significant by the nested F-test; SwedishFacts remains non-significant with p_F=.092 in that sensitivity specification.
The annotation meta-evaluation provides supporting context rather than the core validity test. GPT-5.2 reaches 73.3% binary CS/CA accuracy on the held-out GlobalMMLU-Lite test split and 90.6% against the Native-500 explicit majority. On Native-500, four-way required-knowledge accuracy is 83.9% over 477 resolved items, or 80.0% when the 23 unresolved ties are counted as errors.
tab:proxy-rank-b
Caption: Native-vs-proxy ranking consistency. τ_b: Kendall's τ_b on overall accuracy across 29 models. T5: top-5 model overlap. τ_CA/τ_CS: τ_b restricted to culturally agnostic / sensitive items. All p<.001. Full statistics in Appendix sec:app:proxy:blockb.
| Native | L | τ_b [95% CI] | T5 | τ_CA | τ_CS |
|---|---|---|---|---|---|
| Skolprov | SV | .89 [.78, .96] | 5/5 | .88 | .77 |
| Exam-ET | ET | .88 [.78, .96] | 5/5 | .83 | .88 |
| INCLUDE | LT | .87 [.79, .93] | 5/5 | .82 | .85 |
| NRK-Quiz | NB | .85 [.76, .92] | 4/5 | .82 | .88 |
| DanishCT | DA | .82 [.73, .90] | 3/5 | .66 | .84 |
| INCLUDE | ET | .72 [.58, .83] | 3/5 | .68 | .72 |
| SwedishFacts | SV | .51 [.25, .74] | 3/5 | .50 | .52 |
Why It Matters: Ranking agreement is strong for most multiple-choice pairings but falls to .51 for SwedishFacts, illustrating why correlation alone is insufficient to establish proxy validity.
tab:incremental-main
Caption: Incremental proxy validity on all items. Brackets show 95% model-bootstrap CIs. Classes follow the fixed evidence definition in the text. Full details are in Appendix sec:app:proxy:incremental. The R² CI is not used as a null test because in-sample R² ≥ 0 by construction.
| Native | R² [95% CI] | β_proxy [95% CI] | Class |
|---|---|---|---|
| Skolprov | .128 [.038, .253] | 1.06 [.80, 1.19] | robust |
| INCLUDE-LT | .076 [.021, .148] | 0.94 [.67, 1.13] | robust |
| NRK-Quiz | .041 [.009, .087] | 0.46 [.29, .59] | robust |
| Exam-ET | .024 [.010, .044] | 0.52 [.34, .85] | robust |
| INCLUDE-ET | .059 [.000, .199] | 0.84 [-.02, 1.30] | marginal |
| DanishCT | .009 [.000, .027] | 0.31 [.01, .57] | marginal |
| SwedishFacts | .035 [.000, .163] | 0.14 [-.10, .33] | null |
Why It Matters: The main validity result is a four-robust, two-marginal, one-null pattern, with Skolprov providing the strongest incremental signal and SwedishFacts providing none under the main specification.
tab:sensitivity-en
Caption: Sensitivity analysis adding English G-MMLUX as a third control. Brackets are 95% percentile model-bootstrap CIs. The main evidence classes remain those in Table tab:incremental-main. Full R², regression, and bootstrap statistics are in Appendix sec:app:proxy:sensitivity.
| Native | β_proxy [95% CI] | p_F |
|---|---|---|
| Skolprov | 1.17 [.88, 1.28] | <10^-12 |
| INCLUDE-LT | 1.08 [.84, 1.30] | <10^-9 |
| INCLUDE-ET | 1.06 [.36, 1.42] | <10^-5 |
| NRK-Quiz | .54 [.34, .68] | <10^-6 |
| Exam-ET | .59 [.39, .95] | <10^-7 |
| DanishCT | .57 [.34, .92] | <10^-3 |
| SwedishFacts | .25 [.06, .48] | .092 |
Why It Matters: Adding English G-MMLUX as a control preserves the broad ordering of effects, while SwedishFacts remains non-significant by the nested F-test.
fig:proxy-gradient
Caption: Proxy validity as a function of cultural sensitivity. Each point represents one native-vs-proxy pairing. (a)~Partial correlation after residualizing on controls. (b)~Incremental variance ($ R^2$) from the nested $F$-test. The dashed line shows the MC-only linear trend (6~pairings; descriptive and not statistically significant at $n=6$). DanishCitizenTest (95\% CS, MC format) already shows weak incremental validity. SwedishFacts lies close to the descriptive MC extrapolation in panel~(b), which is consistent with a locality/content-overlap interpretation, but the design does not isolate locality from its open-ended response format.
Why It Matters: The figure summarizes the paper's central interpretation: incremental proxy validity tends to weaken as native evaluation becomes more culturally sensitive, while also showing why SwedishFacts cannot cleanly distinguish local content from open-ended response format.
🔮 Conclusion
The paper finds a gradient rather than a binary answer to whether translated benchmarks can substitute for native ones. Translation-based proxies are most useful when their source content overlaps with the native evaluation target, as in curriculum-oriented Skolprov and Exam-ET.
Cultural sensitivity by itself is not a sufficient predictor of proxy failure: INCLUDE-LT and NRK-Quiz remain robust despite substantial culturally sensitive content. Conversely, SwedishFacts shows that a locally anchored target can escape the predictive signal of translated MMLU.
The practical conclusion is conditional use. Translated benchmarks can provide efficient evidence for shared subject matter, but local factual knowledge and other poorly aligned capabilities require native benchmarks.
🛠️ Future Research Improvements
A stronger follow-up would use multiple translation systems, including human and LLM translation, to separate source-content effects from translation-quality effects. The current results are specific to DeepL-based proxies.
The SwedishFacts limitation calls for matched experiments: multiple-choice and generative tests with comparable local content, plus generative tests with broader non-local content. This would distinguish locality from response-format mismatch.
Future studies should expand the model sample and language coverage, especially typologically distant and lower-resource languages. Larger model sets would improve regression precision and make family-level stability checks less dependent on nominal thresholds.
Automated cultural annotations should continue to be calibrated against native-language references, with special care for ambiguous required-knowledge categories, class imbalance, and unresolved annotator ties.
🏭 Potential Industry Use Scenarios
Evaluation teams can use translated MMLU-style tests for early multilingual model screening when the intended capability is broad curricular knowledge that overlaps with the source benchmark. The paper supports treating these scores as provisional proxy signals rather than complete substitutes for native evaluation.
Teams building models for education, public services, search, or regional assistants should add native local-knowledge benchmarks for country-specific people, events, geography, institutions, and current affairs. SwedishFacts and DanishCitizenTest illustrate why local-content coverage should be assessed explicitly.
Benchmark dashboards can report both ranking agreement and incremental validity, while also separating multiple-choice from open-ended results. This would make it easier to identify whether an apparent multilingual gain reflects language proficiency, general model strength, source-content familiarity, or genuine local competence.
💬 Critical Analysis
The strongest aspect of the study is its distinction between correlation and proxy validity. Controlling for model size and Belebele performance, then testing bootstrap and leave-one-family-out stability, directly addresses the possibility that strong models rank highly everywhere. The English G-MMLUX sensitivity analysis further tests whether the effect is merely general MMLU familiarity.
The evidence supports a content-overlap interpretation, but it does not establish that translated and native benchmarks measure an identical latent construct. The robust, marginal, and null labels describe evidence strength within the evaluated model sample and benchmark pairings, not universal benchmark quality.
The main unresolved identification problem is the SwedishFacts pairing: it combines highly local content with the only open-ended response format. The paper acknowledges this limitation, and the figure's descriptive trend should therefore be read as suggestive rather than causal.
Annotation quality is adequate for broad CS/CA analysis but weaker for fine-grained required-knowledge categories. Native-500 binary accuracy is 90.6%, while four-way accuracy is 83.9% on resolved items and 80.0% when unresolved ties count as errors; Norwegian Bokmål and dialect labels require particular caution.