📌 Highlights
The paper's central claim is that multilingual safety must be evaluated across language and culture, not inferred from English refusal behavior. MSAFETEMP operationalizes this claim with shared templates, translated variants, benign controls, and culturally localized fills.
The strongest result is the gap between English and multilingual testing: on HarmBench, Olmo-3.1-32B-Instruct rises from 0.0% attack success on the original English seeds to 13.7% on translated variants, while both Apertus models also become more vulnerable after translation.
Training with MSAFETEMP sharply reduces multilingual attack success across the tested model families, but SFT can increase over-refusal and lower general capability. DPO often reduces that regression without fully eliminating it for every model.
The main caveat is evidence quality: generation and judging are largely automated, human validation is limited in language coverage, and inherited English labels may understate the severity of some culturally localized prompts.
- MSAFETEMP covers 44 training languages and 60 evaluation languages.
- Translated and localized variants expose failures that English-only evaluation misses.
- Matched benign controls make over-refusal measurable alongside attack success.
- Safety gains require monitoring helpfulness and general capability.
🎯 Introduction
The paper addresses the multilingual safety gap in large language models. Existing safety alignment and red-teaming are heavily English-centric, while prior work shows that harmful requests can receive unsafe answers after translation, especially in lower-resource languages. Literal translation also preserves English-centric entities and does not reliably represent local extremist movements, historical atrocities, institutions, or community-specific slurs.
The paper frames safety as an interaction among model, language, and culture rather than as a language-independent capability. Its objective is to answer whether open models generalize safety beyond English, whether template-based generation can scale culturally grounded harmful and benign variants, and whether training on those variants improves multilingual safety without unacceptable over-refusal.
The contribution is therefore both a data-generation method and an evaluation protocol: shared templates hold the request frame constant while different slot values isolate translation, benignity, and cultural localization effects.
🔬 Methodology
The pipeline begins with 200 standard HarmBench behaviors reserved for evaluation and 2,500 harmful WildGuardMix prompts for training. The authors remove contextual and copyright categories from HarmBench because they are not defined solely by the prompt. For each seed, a generator extracts single words or short phrases that make the prompt harmful and replaces them with typed slots. A first judge checks whether the template remains valid and self-contained.
The carrier text is translated with the slots fixed, and two validated paraphrases are added per seed and language. The harmful TR branch fills the slots with generic harmful entities. The OR branch fills the same slots with DIRECT or BOUNDED benign alternatives, preserving a matched linguistic frame. The PH branch uses a per-language cultural card containing local groups, organizations, events, concepts, and institutions; the generator may rephrase around the selected entity to preserve grammaticality and natural flow.
Gemma-4-31B-IT performs generation-related tasks, GPT-OSS-120B is used as a generation judge, GPT-OSS-Safeguard-120B judges target-model responses, and Gemini Flash 3.6 verifies cultural cards. The final training setup applies one epoch of SFT to refusals and benign answers mixed with general instruction data, followed by DPO on preference pairs sampled from the fine-tuned model.
The key source notation is the typed-slot representation: a phrase such as \langle\text{illicit\_substance}\rangle is treated as a controllable semantic field, while the surrounding template remains available for translation, paraphrasing, and matched filling.
Typed harmful slot
Meaning: A placeholder representing a harm-bearing phrase after template extraction; it allows the surrounding carrier text to be translated or paraphrased without directly manipulating a complete harmful request.
Multi-slot template
Meaning: The source-backed template structure controls which semantic fields are filled with harmful, benign, or culturally localized entities while keeping the linguistic frame comparable.
📊 Experiments
The evaluation uses HarmBench and WildGuardMix with GPT-OSS-Safeguard-120B judging greedy target-model responses. Attack success rate (ASR) is the share of harmful prompts receiving unsafe compliance, and over-refusal (OR) is the share of DIRECT benign prompts that are refused; lower is better for both. The main comparison uses original English seeds, translated TR variants, culturally localized PH variants, and benign OR controls. The paper also evaluates jailbreak-wrapped HarmBench prompts with 300 prompts per language.
The leading-model panel includes Qwen3.5-27B, GPT-OSS-20B, Gemma-4-31B-IT, Olmo-3.1-32B-Instruct, Llama-3.3-70B-Instruct, Apertus-8B-Instruct-2509, Apertus-70B-Instruct-2509, and Apertus-v1.5-8B-IT. On HarmBench, Olmo rises from 0.0% ASR on original English seeds to 13.7% on TR and 11.9% on PH. On the held-out WildGuard split, translated variants make Olmo, Gemma, and both Apertus models fail 2.4–4.4 times as often as their original English seeds, while Qwen3.5 and GPT-OSS remain near or below 1% ASR but incur high OR.
Localization is not uniformly a stronger attack. In paired tests, PH lowers ASR for 5 of 7 models on HarmBench and 6 of 7 on WildGuard, but raises it for Apertus-8B and the instruction-tuned Olmo checkpoint. The translated and localized arms agree on fewer than half of the behaviors they break, with Jaccard overlap of 0.40–0.45, and approximately 45% of arm-specific failures occur only with local entities. Figure 3 captures this model-dependent behavior across eight example languages.
The training ablation evaluates Apertus-8B-2509, Apertus-v1.5-8B, Olmo-3-7B, and Llama-3.1-8B using AyaRT, PolyTox, multilingual HarmBench and WildGuardMix, OR-Bench, MMLU, and GSM8K. SFT substantially reduces HB and WG ASR, while DPO generally lowers the over-refusal introduced by SFT. For example, Apertus-v1.5-8B moves from 2.3% HB ASR and 2.7% WG ASR to 1.3% and 0.5% after SFT+DPO; Olmo-3-7B moves from 21.5% and 15.1% to 2.9% and 0.8%. Capability costs are visible in MMLU and especially GSM8K for several SFT runs.
tab:datastats
Caption: statistics. WildGuard is used for training and performance signal; HarmBench remains evaluation-only. Template counts are verified/original; benign counts are direct\,$+$\, bounded. Over-refusal uses direct only.
| Track | Seeds | Languages | Templates | Harmful | Benign | Post-hoc |
|---|---|---|---|---|---|---|
| WildGuard | $2,500$ | $44$ | $109,425/110,000$ | $1,537,628$ | $186,639+20,528$ | $97,120$ |
| HarmBench | $200$ | $60$ | $11,976/12,000$ | $173,743$ | $29,598+3,682$ | $9,463$ |
tab:eval-main
Caption: Leading open models on both evaluation tracks (\%, lower is better). Orig.\ is the untouched English seed; TR and PH are the means over translated and localized arms. HB-JB is jailbreak-wrapped HarmBench ($300$ prompts per language): ASR on wrapped PH requests and OR on wrapped benign prompts. Full results in App.~app:model-grids,~app:wg-eval, and~app:matched-wrappers.
| 4c| HarmBench | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| (200 seeds, 60 langs) | 4c| WildGuard test | |||||||||
| (500 seeds, 44 langs) | HB-JB | |||||||||
| Model | Orig. | TR | PH | OR | Orig. | TR | PH | OR | ASR | OR |
| Qwen3.5-27B | $0.5$ | $0.4$ | $0.1$ | $3.5$ | $0.0$ | $0.3$ | $0.1$ | $17.0$ | $<0.1$ | $31.1$ |
| GPT-OSS-20B | $0.0$ | $0.4$ | $0.3$ | $31.8$ | $0.4$ | $0.5$ | $0.2$ | $45.1$ | $0.2$ | $56.0$ |
| Gemma-4-31B-IT | $6.5$ | $4.2$ | $2.5$ | $3.2$ | $0.8$ | $3.2$ | $1.3$ | $7.5$ | $2.0$ | $13.2$ |
| Olmo-3.1-32B-Instruct | $0.0$ | $13.7$ | $11.9$ | $4.6$ | $2.8$ | $10.7$ | $10.8$ | $6.1$ | $16.6$ | $16.8$ |
| Llama-3.3-70B-Instruct | $13.0$ | $5.3$ | $2.6$ | $7.7$ | $6.7$ | $5.6$ | $4.5$ | $12.8$ | $10.3$ | $26.6$ |
| Apertus-8B-Instruct-2509 | $8.5$ | $10.1$ | $12.2$ | $10.2$ | $4.3$ | $10.5$ | $8.7$ | $12.2$ | $31.8$ | $5.3$ |
| Apertus-70B-Instruct-2509 | $8.5$ | $13.0$ | $6.0$ | $5.3$ | $3.2$ | $14.2$ | $9.3$ | $7.5$ | $30.9$ | $6.0$ |
| Apertus-v1.5-8B-IT | $0.0$ | $1.9$ | $1.8$ | $25.1$ | $0.7$ | $2.8$ | $2.4$ | $29.6$ | $24.6$ | $8.7$ |
tab:alignment
Caption: General evaluation results for original base and finetuned models for Apertus v1 and 1.5 8B, Olmo-3-7B, and Llama-3.1-8B. SFT and DPO use ; green and red mark an improvement or a degradation relative to the untuned checkpoint (Base or IT). HB and WG are the multilingual HarmBench and WildGuardMix tracks of , judged by GPT-OSS-Safeguard-120B, AyaRT and PolyTox have their own default judge setup.
| Safety $ $ | Over-refusal $ $ | Capability $ $ | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | AyaRT | PolyTox | HB ASR | WG ASR | HB OR | WG OR | OR-Bench | MMLU | GSM8K |
| Base | $12.2$ | $45.9$ | $14.2^ $ | $3.1^ $ | $1.6^ $ | $44.3^ $ | $2.1$ | $60.1$ | $49.9$ |
| \; + SFT | 1.4 | 14.4 | 3.3 | 2.6 | 13.5 | 12.7 | 15.0 | 59.1 | 45.2 |
| \; + SFT + DPO | 0.1 | 7.1 | 1.5 | 0.5 | 7.7 | 5.5 | 23.7 | 58.9 | 46.9 |
| IT | $0.7$ | $4.7$ | $10.3$ | $10.5$ | $10.3$ | $12.5$ | $82.6$ | $61.6$ | $63.3$ |
| \; + SFT | 1.1 | 11.7 | 3.0 | 2.3 | 13.4 | 14.2 | 27.4 | 60.1 | 50.3 |
| \; + SFT + DPO | 0.1 | 5.8 | 1.4 | 0.5 | 6.3 | 8.0 | 34.6 | 61.7 | 51.4 |
| IT | $0.5$ | $7.8$ | $2.3$ | $2.7$ | $23.2$ | $29.6$ | $83.2$ | $70.2$ | $80.2$ |
| \; + SFT | $0.5$ | 9.4 | 3.0 | 2.1 | 13.5 | 13.8 | 45.6 | 63.1 | 72.8 |
| \; + SFT + DPO | 0.2 | 6.1 | 1.3 | 0.5 | 6.0 | 6.0 | 38.4 | 64.0 | 72.4 |
| IT | $2.1$ | $7.0$ | $21.5$ | $15.1$ | $2.7$ | $3.1$ | $52.5$ | $67.1$ | $83.0$ |
| \; + SFT | 2.6 | $7.0$ | 4.3 | 1.5 | 28.4 | 20.2 | 26.4 | 60.6 | 75.6 |
| \; + SFT + DPO | 0.5 | 5.7 | 2.9 | 0.8 | 19.8 | 13.7 | 49.8 | 62.3 | 76.2 |
| IT | $0.8$ | $4.4$ | $15.9$ | $10.5$ | $4.5$ | $7.4$ | $4.6$ | $72.0$ | $84.0$ |
| \; + SFT | 0.6 | 6.7 | 1.7 | 0.7 | 14.1 | 6.8 | 35.7 | 65.2 | 67.9 |
| \; + SFT + DPO | 0.5 | 6.1 | 2.0 | 0.8 | 5.3 | 2.6 | 24.8 | 63.6 | 68.5 |
fig:pipeline
Caption: The pipeline. Original prompt goes through extraction of harmful terms, validation, translation, and a targeted judge pass of a template. Three branches then produce generic harmful variants (TR), exact-fill benign controls (OR), and culturally localized variants (PH). English (EN) follows the same path without translation.
Why It Matters: This is the central data-construction design: it shows how one English seed becomes translated, benign, and culturally localized variants while preserving a shared template.
fig:temp_example
Caption: example, showing how different augmentation types are produced from different stages in the pipeline.
Why It Matters: The worked example clarifies the distinction between generic harmful fills and culturally grounded fills, including how the surrounding wording can be rephrased around a local entity.
fig:posthoc
Caption: Translated vs. localized attack success on HarmBench for four leading models in eight example languages. Solid bars are translated (TR) and hatched bars culturally localized (PH) variants of the same templates. English is shown as a TR-only reference, since it has no PH arm. Localization raises ASR for Apertus-8B in every language shown but lowers it in most languages for the other models. Full results over 59 languages are in Tables~tab:frontier-tr and~tab:frontier-ph.
Why It Matters: It visualizes the model-dependent effect of cultural localization and shows why translated-only safety testing is insufficient.
🔮 Conclusion
The paper demonstrates that English-only safety evaluation provides an incomplete view of model behavior. Translation can reveal large multilingual safety gaps, and cultural localization reaches failures that translation alone misses. The effect is model-dependent: some models become more vulnerable with localization, while others show lower attack success.
MSAFETEMP provides a practical way to produce matched multilingual harmful and benign data at scale. Its training experiments show that diverse multilingual supervision improves safety on held-out languages and model families, but the gains must be balanced against over-refusal and capability costs.
The central engineering takeaway is to measure safety and helpfulness jointly across language, culture, and attack format.
🛠️ Future Research Improvements
The most important next step is broader human validation, especially for low-resource languages and culturally localized variants. The current human study supports naturalness and cultural fit, but its limited respondent coverage cannot fully establish local validity.
Future work should compare multiple independent judge models and human adjudication on the same samples, re-label localized prompts whose severity differs from the English seed, and quantify judge disagreement by language and harm category.
The fixed single-turn template design should be extended to multi-turn conversations, iterative jailbreaks, and automated attackers seeded with localized variants. Training research should also compare RL-based alignment with SFT and DPO while isolating the sources of capability degradation.
Evaluation should expand beyond inherited English seeds to locally authored harmful and benign prompts, which would test whether the template method captures real cultural threat models rather than only translated descendants.
🏭 Potential Industry Use Scenarios
Model providers can use the template and cultural-card structure to build multilingual pre-deployment red-team suites that compare English, translated, and locally grounded variants under matched conditions.
Safety teams can add the DIRECT benign branch to regression testing so that reductions in unsafe compliance are checked against over-refusal on ordinary multilingual requests. This is particularly useful for customer-support, search, education, and productivity systems deployed across languages.
Organizations serving specific regions can adapt the cultural-card layer to include locally relevant institutions, groups, events, and concepts, provided that native-speaker review and careful harm labeling are added before deployment.
The corpus can also support alignment fine-tuning for open models, but deployment decisions should track ASR, OR, general capability, judge reliability, and the dual-use risks of distributing harmful prompts and jailbreak wrappers.
💬 Critical Analysis
The strongest aspect of the work is its controlled comparison design. TR and PH share seed, template, language, and judge, while OR preserves the linguistic frame with safer slot values. This makes the observed differences more interpretable than comparisons across unrelated multilingual datasets.
The results are practically significant because they demonstrate that multilingual safety failures are not simply a low-resource-language average effect. Model size and English safety do not determine the direction of the translation or localization gap, and strong refusal behavior can coexist with high over-refusal.
The main threat to validity is automated evaluation. LLM judges may share biases with the generators or fail differently across languages, and the paper's human validation is too small to independently establish all cultural judgments. In addition, inherited seed labels and fixed templates may miss harms introduced by local entities or conversational context.
For builders, the paper is best treated as a strong recipe for broadening coverage and finding failure modes, not as evidence that synthetic data alone solves multilingual alignment. Any production adoption should reproduce the matched comparisons, inspect local examples with native experts, and verify that safety gains do not come from indiscriminate refusal.