đź“Ś Highlights
HoS adds learned, channel-conditioned attention gates independently for each head, aiming to let models read untrusted content without treating embedded instructions as authorized commands.
On held-out InjecAgent goals, the strongest evaluated RL-Hammer attacker reached 97% ASR on GPT-OSS-20B, 85% with SecAlign, and 10% with HoS; for Llama-3.3-70B the reported maxima were 99%, 82%, and 1%, respectively.
On standard multiple-choice evaluations, HoS stayed close to base-model utility in the reported zero-shot and few-shot settings. Agentic results were mixed by benchmark: HoS reduced ASR substantially, while Llama-3.3-70B HoS had lower utility on AgentDyn than the undefended model.
The approach assumes that a system can correctly label trusted and untrusted sources. It does not resolve which sources deserve trust or when external instructions should be followed.
- Mechanism: learned per-head, per-channel attention gates.
- Evaluated models: Llama-3.3-70B-Instruct and GPT-OSS-20B.
- Strongest reported RL-Hammer ASR against HoS: 10% on GPT-OSS-20B and 1% on Llama-3.3-70B.
- Main caveat: trust labeling remains external to the method.
🎯 Introduction
Indirect prompt injection occurs when a model mistakes attacker-planted instructions in external content for authorized commands. This risk grows as models read external data and use tools. The paper focuses on instruction–data separation: external content may be read and analyzed, but should not steer actions when it is intended only as data.
The authors argue that existing defenses remain vulnerable to adaptive attacks and that open-weight models are especially challenging to defend. They compare against SecAlign++, a fine-tuning defense, and ASIDE, an architectural method that rotates embeddings of untrusted content.
The paper's objective is to enforce a supplied trust boundary by changing attention routing, rather than asking the model to infer source trust or acquire new capabilities. It evaluates this proposal on Llama-3.3-70B-Instruct and GPT-OSS-20B.
🔬 Methodology
Each token receives a channel such as trusted, untrusted, or chat-template. In the usual setup, system and user prompts are trusted and tool outputs are untrusted. Each attention head learns a gate logit for each query-channel/key-channel pair, so heads can respond differently to the same untrusted segment.
The gates modify pre-softmax attention logits through the log of a sigmoid gate. This scales the unnormalized weight assigned to each key's channel before softmax renormalizes attention. If every key has the same gate, softmax cancels the common shift; thus, differences across channels determine the effect. The formulation applies directly to Llama-3.3, while GPT-OSS treats its attention sink as an additional key-only channel.
Training updates only gate parameters, using a SecAlign++-style recipe with Direct Preference Optimization and adapted Cleaned-Alpaca preference pairs. The preferred response follows the user's instruction; the rejected response follows the injected instruction. The paper reports 46,080 gate scalars for Llama-3.3-70B and 18,432 for GPT-OSS-20B.
equation 1
Meaning: Defines gated attention for the final token: channel-pair gate logits are passed through sigmoid and log, added to attention logits, then normalized by softmax before weighting values.
đź“Š Experiments
The evaluations use Llama-3.3-70B-Instruct and GPT-OSS-20B. Adaptive attacks include black-box RL-Hammer on held-out InjecAgent goals and a white-box Claudini-derived attack on 100 examples each from AgentDojo and InjecAgent. The paper reports cumulative ASR for the white-box attack under a compute budget; HoS required roughly three times the compute of the undefended and SecAlign models to reach 50% ASR.
For benign utility, the paper evaluates ARC-C, GSM8K, MMLU, and MMLU-Pro on all-trusted prompts, including zero-shot and few-shot settings. HoS remains close to the corresponding base models in the presented table; for example, GPT-OSS-20B's 5-shot MMLU-Pro accuracy is 68.3% for the base model and 67.2% for HoS, compared with 59.4% for SecAlign and 52.1% for ASIDE. Llama-3.3-70B HoS scores 54.8% on that table's 5-shot MMLU-Pro column, against 56.3% for the base model and 16.1% for SecAlign.
The multi-turn utility evaluation uses 100 three-turn AlpacaFarm conversations. HoS remains near the base model across turn placements, while SecAlign's instruction completion degrades after untrusted input. Agentic evaluations cover AgentDojo, AgentDyn, WASP, and LLMail-Inject, reporting utility, utility under attack, and ASR where supplied. On Llama-3.3-70B, HoS lowers ASR to 0.9% on AgentDojo, 1.4% on AgentDyn, 1.4% on WASP, and 0.0% on LLMail; its utility is lower than base on AgentDyn (18.3% versus 33.7%).
tab:gptoss_utility
Caption: Utility on all-trusted prompts. SecAlign degrades sharply in several few-shot settings, while HoS remains close to the corresponding undefended model throughout.
| Model | ARC-C (0-shot) | GSM8K (0-shot) | MMLU (1-shot) | MMLU (8-shot) | MMLU-Pro (0-shot) | MMLU-Pro (5-shot) | ||
|---|---|---|---|---|---|---|---|---|
| GPT-OSS-20B | 96.1\% | 91.4\% | 93.6\% | 95.6\% | 83.0\% | 82.3\% | 72.0\% | 68.3\% |
| / SecAlign | 94.8\% | 93.3\% | 70.9\% | 91.8\% | 82.8\% | 54.4\% | 71.6\% | 59.4\% |
| / ASIDE | 94.7\% | 91.3\% | 93.2\% | 93.6\% | 83.2\% | 74.1\% | 71.7\% | 52.1\% |
| / HoS (ours) | 95.4\% | 90.5\% | 92.3\% | 94.6\% | 83.2\% | 80.0\% | 72.3\% | 67.2\% |
| Llama-3.3-70B | 94.5\% | 93.0\% | 93.7\% | 95.7\% | 84.5\% | 82.8\% | 67.8\% | 56.3\% |
| / SecAlign | 93.8\% | 92.2\% | 2.0\% | 68.0\% | 84.5\% | 28.4\% | 68.6\% | 16.1\% |
| / HoS (ours) | 93.7\% | 92.9\% | 93.2\% | 95.7\% | 85.0\% | 82.4\% | 67.9\% | 54.8\% |
Why It Matters: The recovered table has eight numeric columns but only six benchmark subheaders are legible in the extracted evidence. The values are retained, but the two ambiguous column mappings are left unlabeled rather than inferred.
tab:main_results
Caption: Utility and ASR on Agentic Benchmarks for Llama-3.3-70B variants
| Benchmark | Metric | Llama-3.3-70B | w/ SecAlign | w/ HoS (ours) |
|---|---|---|---|---|
| AgentDojo | Utility | 53.7\% | 67.0\% (+13.3) | 58.5\% (+4.8) |
| AgentDojo | Utility attacked | 36.2\% | 63.9\% (+27.7) | 52.4\% (+16.2) |
| AgentDojo | ASR | 23.1\% | 2.1\% (-21.0) | 0.9\% (-22.2) |
| AgentDyn | Utility | 33.7\% | 41.7\% (+8.0) | 18.3\% (-15.4) |
| AgentDyn | Utility attacked | 30.9\% | 38.3\% (+7.4) | 18.4\% (-12.5) |
| AgentDyn | ASR | 18.5\% | 5.4\% (-13.1) | 1.4\% (-17.1) |
| WASP | Utility | 66.5\% | 62.7\% (-3.8) | 62.2\% (-4.3) |
| WASP | Utility attacked | 62.4\% | 79.8\% (+17.4) | 81.4\% (+19.0) |
| WASP | ASR | 7.6\% | 2.4\% (-5.2) | 1.4\% (-6.2) |
| LLMail | Utility attacked | 80.2\% | 84.6\% (+4.4) | 82.4\% (+2.2) |
| LLMail | ASR | 2.1\% | 12.7\% (+10.6) | 0.0\% (-2.1) |
Why It Matters: Reports the Llama-3.3-70B security–utility results, including the AgentDyn utility cost and the differing LLMail ASR outcomes.
Table 1
Caption: RL-Hammer on InjecAgent
| Attacker training target | Eval target: GPT-OSS | Eval target: w/ SecAlign | Eval target: w/ HoS | Eval target: GPT-5.6 Luna |
|---|---|---|---|---|
| InjecAgent (static) | 0\% | 0\% | 0\% | 0\% |
| GPT-OSS-20B | 95\% | 27\% | 5\% | 1\% |
| / w/ SecAlign | 97\% | 85\% | 10\% | 6\% |
| / w/ HoS | 89\% | 46\% | 3\% | 6\% |
| GPT-5.6 Luna | 87\% | 31\% | 2\% | 36\% |
| Highest ASR | 97\% | 85\% | 10\% | 36\% |
| Attacker training target | Eval target: Llama | Eval target: w/ SecAlign | Eval target: w/ HoS |
|---|---|---|---|
| InjecAgent (static) | 16\% | 5\% | 0\% |
| Llama-3.3-70B | 99\% | 4\% | 0\% |
| / w/ SecAlign | 99\% | 82\% | 1\% |
| / w/ HoS | 72\% | 6\% | 0\% |
| GPT-5.6 Luna | 76\% | 0\% | 0\% |
| Highest ASR | 99\% | 82\% | 1\% |
Why It Matters: Shows attack transfer across evaluation targets and reports the highest observed ASR for each target column.
fig:method
Caption: (Top) Heads of Security (HoS) learns to redistribute attention between trusted and untrusted tokens. Green, red, and blue denote trusted, untrusted, and chat template tokens. Here, a head initially attends strongly to both the trusted request (``Count R's'') and an injected instruction (``Send money''). Learned gates preserve trusted and chat template tokens ($1\! $ and $0.9\! $) while suppressing untrusted tokens ($0.1\! $), shifting attention toward the trusted request. Gates are learned independently per head, so other heads may retain or increase attention to untrusted tokens. (Bottom) HoS improves security and utility. Compared to SecAlign and ASIDE, HoS is more robust against adaptive...
Why It Matters: Illustrates the per-head gating mechanism and summarizes the paper's reported security and utility findings.
fig:whitebox_and_heatmap
Caption: White-box robustness and learned attention gates. Left: Cumulative ASR against GPT-OSS-20B and its defended variants on 100 examples each from AgentDojo and InjecAgent, using an attack developed specifically against HoS. An example counts as successful if any suffix found within the compute budget succeeds. Right: Learned shifts in the log attention ratio between trusted and untrusted keys for trusted queries in GPT-OSS-20B (sec:attention_gates_analysis).
Why It Matters: Shows the white-box attack comparison and the distribution of learned attention shifts.
fig:multi_turn_alpacafarm
Caption: SecAlign's utility drops sharply after it encounters untrusted input in the multi-turn evaluation, while HoS remains stable. Per-turn instruction-completion rate for Llama-3.3-70B and its defended variants on 100 three-turn AlpacaFarm conversations. Each turn contains an independent instruction that should be answered separately. Exactly one turn also contains task data as untrusted input, placed at turn 1, 2, or 3 (indicated by the gray region) in the left, middle, or right panel, respectively. Undefended Llama and HoS remain near 100\% across turns and placements. SecAlign drops most sharply on the input turn and remains degraded on later fully trusted turns.
Why It Matters: Compares multi-turn instruction completion when untrusted input appears at different turns.
fig:gpt_oss_tradeoff
Caption: Security--utility trade-off for GPT-OSS-20B. HoS Pareto-dominates SecAlign on (a) AgentDojo and (d) WASP, and matches it on (b) AgentDyn and (c) LLMail. Each point is a separately trained model with a different learning rate. The y-axis reports utility under attack (higher is better), and the x-axis reports attack success rate (lower is better). The undefended GPT-OSS-20B is Pareto-dominated by both HoS- and SecAlign-defended models. ASIDE consistently fails to perform tool calls and achieves near-zero utility on all four benchmarks.
Why It Matters: Presents the security–utility frontier across four agentic benchmarks for GPT-OSS-20B.
fig:role_probes
Caption: HoS improves role perception for GPT-OSS-20B. (a) On AgentDojo benign user tasks, GPT-OSS-20B does not correctly perceive tool outputs, while HoS strongly improves the tool role perception across layers. (b) HoS decreases the perceived Userness of the injection tokens inside the tool outputs compared to GPT-OSS-20B.
Why It Matters: Connects the defense to role attribution, a proposed mechanism related to resistance against injected instructions.
đź”® Conclusion
The authors conclude that instruction–data separation can be implemented by constraining attention rather than broadly fine-tuning model behavior. In their tested open-weight models, HoS improves robustness to the evaluated adaptive attacks and generally preserves utility outside the training format, while matching or improving SecAlign's security–utility trade-off on the presented agentic benchmarks.
The result is a defense primitive with a clear operational dependency: external policy must supply trust labels, and some tasks require treating instructions in external content as actionable.
🛠️ Future Research Improvements
Evaluate more diverse training data, including multi-turn conversations, multiple instructions, and varied templates, as the authors identify the single-turn training distribution as a limitation.
Test additional adaptive attacks and budgets; the paper's white-box and black-box evaluations provide evidence for increased attack difficulty but cannot establish security against all attackers.
Develop a compatible trust-gating mechanism for linear attention architectures, which do not expose explicit per-key attention coefficients in the form used here.
Study how to combine supplied trust boundaries with instruction hierarchy and contextual authorization for tasks where external content legitimately contains instructions.
🏠Potential Industry Use Scenarios
A plausible application is a tool-using assistant that reads web pages, emails, or documents as data while treating tool outputs as untrusted. The paper evaluates agentic tasks including AgentDojo, AgentDyn, WASP, and LLMail-Inject, providing evidence in tool-use settings, though benchmark results alone do not establish production readiness.
For deployment, teams would need an explicit trust-labeling policy and tests for workflows where external text is sometimes intended to guide actions. The paper frames HoS as complementary to instruction hierarchy and contextual integrity rather than a complete authorization system.
đź’¬ Critical Analysis
A strength is the targeted parameterization: each head learns channel-dependent routing, and the reported results pair adaptive-attack evaluations with standard utility, multi-turn, and agentic tasks. The gate analysis and role probes offer evidence about possible mechanisms, though they do not by themselves prove a complete causal account of robustness.
The utility evidence is not uniformly positive. In the Llama-3.3-70B AgentDyn results, HoS utility is 18.3% versus 33.7% for the base model, despite lower ASR. Agentic security and utility should therefore be assessed together for the intended workload.
The extracted multiple-choice table has eight numeric columns while only six benchmark subheaders are legible. Its ambiguous columns are not given inferred names here; verify the original table before relying on those cells. The reported adaptive-attack findings are also specific to the tested models, attack recipes, and budgets.