SecPO: Principled Adversarial Training for Prompt Injection Security

📌 Highlights

SecPO combines a clean-input reference distribution with adaptive adversarial training to address two weaknesses of DPO-based prompt-injection defenses: insecure reference anchoring and weak training attacks.

The strongest reported Qwen3.6-27B results reduce AgentDojo adaptive ASR@800 from 78.8% for Meta-SecAlign to 8.8%, reduce DTaP ASR from 45.9% to 7.6%, and preserve AgentDyn utility at 71.7%.

The main caveat is that residual successful attacks remain, especially social-engineering injections that expand a task's scope while appearing helpful.

• SecPO changes the reference conditioning from xinj to xclean.
• Online AT uses 240 candidate-injection attempts per sample in the reported setup.
• Offline optimized injections transfer to larger models without additional attack generation.
• Text-trained robustness also reaches the reported 0.0% ASR on three visual attack goals.

🎯 Introduction

The paper studies indirect prompt injection, where a benign user asks an agent to perform a task while untrusted data contains instructions intended to control the model. The target behavior is strict separation: follow trusted instructions, use untrusted content as data, and retain benign utility when no attack is present.

Prior model-level defenses such as Meta-SecAlign use DPO over weak simulated injections. The paper argues that DPO's KL-equivalent regularization keeps the learned model close to an undefended and vulnerable reference on injected inputs, while weak attacks fail to expose the behaviors needed for robust training.

SecPO's paper-level objective is therefore to approximate an ideal injection-invariant oracle using quantities already available during training, and to pair this objective with strong adaptive attacks generated through an efficient concurrent search pipeline.

🔬 Methodology

The training data begins with Cleaned-Alpaca instruction-following examples containing a trusted prompt and data field. The authors select 19,157 preference pairs, use a dedicated input role for untrusted data, and generate desirable and undesirable responses with the undefended model. Llama-3.1-8B-Instruct is trained with LoRA; Qwen3.6-27B additionally fine-tunes k_proj and o_proj. Both use one epoch, β = 0.1, and learning rate 1.6×10−4.

SecPO replaces the DPO reference likelihoods evaluated on xinj with undefended-model likelihoods evaluated on xclean. This is justified by the oracle conditions: secure behavior should be invariant to the injection, while clean-input behavior should match the undefended model. The resulting loss preserves the utility target without constraining the model to reproduce the vulnerable reference's injected-input behavior.

The adversarial-training loop uses PAIR-style search. For each sample, an attacker proposes n = 8 candidates, the victim checkpoint responds, and a judge scores injection-following behavior from 1 to 10. With T = 30 iterations, the reported online process performs 240 attempts per sample. The stages are batched across examples and run concurrently; the online Llama-3.1-8B run required 18 hours on 16 H200 GPUs, while offline fine-tuning Qwen3.6-27B on the fixed optimized set required 7 hours on 8 H200s.

📊 Experiments

The evaluation covers instruction following through SEP and AlpacaEval2; agentic tool calling through AgentDojo, AgentDyn, and DTaP-Bench-Indirect; science through MMLU-Pro and GPQA Diamond; mathematics through GSM8K and Minerva_Math; and visual prompt injections through DocVQA-based evaluations. Training uses Cleaned-Alpaca and PAIR search, while evaluation uses held-out Genetic and TAP search attacks plus the unseen RL-based PISmith attack. SEP adaptive evaluations use 1,024 examples and report Pass@800 for Genetic and TAP and pass@10 for PISmith.

On Llama-3.1-8B-Instruct, the complete recipe reduces Genetic ASR@800 from 89.1% for Meta-SecAlign to 12.3%, TAP ASR@800 from 84.6% to 12.2%, and PISmith ASR@10 from 95.1% to 7.0%. The paper reports mean adaptive ASR falling from 89.6% to 10.5%, an 8.5× reduction, while mean utility across six instruction-following, science, and math benchmarks changes from 46.9% for the undefended model to 47.5% for the proposed recipe.

On Qwen3.6-27B with offline AT, Genetic, TAP, and PISmith ASRs become 13.3%, 12.5%, and 1.8%, respectively, versus 93.7%, 90.1%, and 94.0% for Meta-SecAlign. AgentDojo adaptive ASR@800 is 8.8% for the proposed model versus 78.8% for Meta-SecAlign; static AgentDojo and AgentDyn ASRs are both 0.0%, and DTaP ASR is 7.6% versus 45.9%. AgentDyn utility remains 71.7%, matching the undefended model.

The ablations support complementary roles for AT and SecPO. On Llama-3.1-8B, Genetic ASR@800 falls from 89.1% to 29.2% with AT+DPO and then to 12.3% with AT+SecPO. On Qwen3.6-27B, it falls from 93.7% to 60.9% and then to 13.3%. Text-trained robustness also transfers to visual attacks: Qwen3.6-27B records 0.0% ASR@1600 on Steal-PII, Call-Tool, and OpenClaw while retaining 95.2% DocVQA utility.

🔮 Conclusion

The paper's evidence supports the claim that prompt-injection robustness improves when training attacks are strong and the preference objective does not anchor the model to insecure behavior on injected inputs. SecPO and adversarial training contribute separately: AT supplies harder security supervision, while SecPO changes the regularization target.

The resulting Qwen3.6-27B model preserves broad benign capability while substantially reducing attack success on SEP, AgentDojo, AgentDyn, DTaP-Bench-Indirect, and visual prompt-injection evaluations. The method also supports offline reuse of optimized attacks, quantized inference, and transfer to another security-training recipe.

🛠️ Future Research Improvements

The most direct extension is to secure reasoning-enabled models. The main Qwen3.6-27B experiments disable thinking, and the paper explicitly leaves reasoning-model security for future work.

Future evaluations should stress the residual social-engineering failure mode, where an injected action is framed as necessary for completing the trusted task. Measuring task-scope expansion, side effects, and tool authorization—not only whether the attack replaces the original answer—would test this weakness more directly.

The threat model and benchmark suite should be broadened to cases without explicit trust labels, stronger multimodal and tool-specific attacks, and attacks that exploit application-level state across longer trajectories. These directions follow from the paper's stated assumptions and remaining successful examples.

🏭 Potential Industry Use Scenarios

SecPO is relevant to coding agents that read repositories, issue trackers, READMEs, and build logs, where untrusted text can attempt to add unauthorized code changes or commands. The reported DTaP and AgentDojo results provide evidence for this scenario, but deployment still requires tool-level permissions and monitoring.

Email, document, browsing, and enterprise search agents could use the trusted/input role separation plus SecPO-style fine-tuning to reduce instruction-following from retrieved content. The method's ability to preserve benign data-dependent replanning is important for workflows where tool outputs legitimately determine the next action.

The released optimized training set and the reported Q8_0 results suggest a path toward smaller or quantized deployments in security-sensitive agentic systems. The paper reports 0.0% observed static ASR on AgentDojo and AgentDyn for the defended model under both BF16 and Q8_0, with and without speculative decoding, but production validation would still need application-specific adaptive attacks.

💬 Critical Analysis

The strongest part of the paper is the alignment between the proposed mechanism and the ablations: stronger attacks improve robustness, and replacing DPO with SecPO produces another large reduction under the same adversarial-training setup. The clean-input reference also gives a simple implementation change rather than requiring an unavailable oracle checkpoint.

The evidence is broad across static, adaptive, agentic, visual, and quantized settings, but the evaluation remains conditional on explicit trusted-versus-untrusted roles and on the selected attack and judge protocols. ASR reductions should therefore be interpreted as robustness against the tested attack families, not as a proof of general prompt-injection immunity.

The practical cost is another concern. Online AT requires repeated attacker, victim, and judge inference, and the paper's strongest training setup uses substantial H200 resources. Offline reuse reduces this cost but introduces a distribution question: optimized injections generated for one model and attack process may not cover a new application's tools, modalities, or social-engineering patterns.

Finally, the residual failures are operationally meaningful. An agent can complete the user's task and still perform an attacker-requested extra action, so builders should pair SecPO with least-privilege tools, action confirmation, output validation, and audit logs.