Multimodal-SEA: Benchmarking AI Agent Security under Multimodal Social Engineering Attacks

📌 Highlights

Multimodal-SEA evaluates personal agents in interactive episodes where social engineering is supported by forged multimodal evidence and culminates in real simulated tool execution. Its central finding is that multimodal content can make agents more likely to act harmfully even while they verify more often.

The benchmark is broad enough to expose several distinct failure modes: harmful execution, failure to recognize deceptive content, unsupported claims about actions, and refusal of legitimate requests.

• 608 tasks: 152 scenarios × four difficulty levels, spanning 14 domains, 316 Chinese instances, and 292 English instances.
• Seven evaluated multimodal models all incur successful attacks in the reported task sets.
• Adding multimodal content raises pooled D4 from 19.9% to 29.2% while pooled D2 rises from 54.7% to 65.1%.
• GPT-5.6-Sol's Chinese D4 falls from 34.8% to 10.8% with the safe prompt, but D6 rises from 50.4% to 63.5%.
• Low attack success is not enough: Claude Opus 5 has 7.1% attack success in the combined attack results but 32.1% over-refusal on legitimate requests.

🎯 Introduction

Personal agents increasingly inspect images, documents, and websites while acting on users' behalf. The paper argues that existing safety evaluations centered on jailbreaks, prompt injection, or text-based social engineering do not fully test attacks in which an adversary combines impersonation, trust exploitation, situational pressure, and forged multimodal evidence.

The paper's objective is to evaluate intrinsic agent security under this setting with consequences grounded in actual tool execution. Its scenarios cover payments, shopping, work, social interactions, and other domains, and require the agent to inspect content, verify claims, and decide whether to execute or reject requests.

• Prior limitation: text-centric evaluations underrepresent cross-modal inconsistencies and attacker-controlled evidence.
• Paper-level objective: measure whether agents recognize deception, seek independent verification, and avoid unsafe actions while still handling legitimate requests.

🔬 Methodology

The task formulation treats a personal agent \(\pi\) as a tool-calling system acting for user U in environment \(\mathcal E\). An attacker A uses permitted information K, verification conditions C, and permitted artifact types \(\mathcal M\) to produce a trajectory. Success is tied to monetary loss or information leakage, while recognition of forged content is scored independently.

Scenario construction starts from 152 fraud-grounded cases across 14 domains, split into 79 Chinese and 73 English scenarios. Each case specifies the impersonated identity, relationship to the user, fabricated event, and harmful goal, then receives a multi-stage script and four difficulty instances. The environment provides 50 simulated applications, shared user profiles, transaction history, medical records, photo albums, files, browsing, search, contacts, file management, payments, and submission or permission tools.

The harness coordinates task context, attack progression, dialogue history, content records, and adaptive responses. It can select prepared content or request task-consistent images, documents, websites, app interfaces, and videos. The evaluation records six dimensions: D1 loss or leakage severity, D2 verification coverage, D3 first sustained recognition turn, D4 attack success, D5 ungrounded action claims, and D6 missed recognition of deceptive content.

📊 Experiments

The main evaluation uses Multimodal-SEA's 608 task instances: 316 Chinese and 292 English, covering 152 scenarios and four difficulty levels. The target models are Claude Opus 5, Gemini 3.8 Flash, MiniMax-M3, Kimi K3, Qwen 3.8 Flash, GPT-5.6-Sol, and GLM 5.3 Flash; DeepSeek-v4-Pro acts as attacker and accomplice model. Safe-prompt variants are reported for MiniMax-M3 and GPT-5.6-Sol. The reported content setup uses Gemini-3.1-image-flash and Kling 2.5 Turbo.

In the main results, Chinese-task attack success ranges from 8.9% to 36.7% and English-task attack success from 5.1% to 34.2%. GPT-5.6-Sol incurs approximately RMB 554,000 and USD 76,600 in cumulative simulated losses across the Chinese and English task sets. The table also shows that high verification coverage does not ensure low deception miss rate: GLM 5.3 Flash reaches 77.2% D2 on Chinese tasks while retaining 38.3% D6.

The multimodal-content ablation covers 86 Level-3 scenarios across the seven evaluated models. Adding multimodal content raises pooled D4 from 19.9% to 29.2% and pooled D2 from 54.7% to 65.1%; GLM 5.3 Flash's D4 increases by 17.4 points and Qwen 3.8 Flash's by 15.1 points. The pooled result supports the paper's claim that verification activity can rise while the quality or independence of verification remains inadequate.

Difficulty analysis shows pooled D4 increasing from 16.3% at L1 to 18.2% at L2, 19.9% at L3, and 27.4% at L4. The paper reports domain variation as well: Charity reaches 43.6% Chinese and 35.7% English D4, whereas English Shopping and Government reach 5.7% and 7.9%. On 42 Chinese and 42 English benign tasks, average CR is 41.7% and average SCR is 34.4%, while Claude Opus 5 and MiniMax-M3 show 32.1% and 51.2% over-refusal despite low attack success.

fig:task-execution

Caption: An illustrative multimodal social engineering attack (left) and the attack harness (right), which supports attack progression and the use of prepared or dynamically constructed content.

Why It Matters: Shows how multi-turn trust-building and multimodal content feed into the attack harness.

fig:d4_difficulty

Caption: Attack success rates (D4, \%) across four difficulty levels. Chinese tasks appear on the left and English tasks on the right, separated by a dashed line and sharing one vertical axis. Icons follow the model order in Table~tab:main, with four bars per model ordered L1 to L4. Each bar summarizes 79 Chinese or 73 English task evaluations.

Why It Matters: Shows how attacker knowledge and verification conditions change D4 across L1-L4 and language sets.

Figure ure

Caption: Figure 2: Overview of Multimodal-SEA, in- cluding domains, applications, impersonated roles, and task difficulty levels.

Why It Matters: Summarizes benchmark coverage across domains, applications, impersonated roles, and difficulty levels.

🔮 Conclusion

The paper concludes that multimodal perception plus autonomous action creates a security gap: agents can connect information across images, documents, apps, and websites without reliably determining whether those sources are independent or attacker-controlled. Stronger safety instructions may prevent some harmful actions, but they do not automatically improve deception recognition.

For builders, the most important result is methodological as well as empirical: security evaluation should connect agent claims to actual tool execution and environment state, measure recognition separately from refusal, and test benign utility alongside attack resistance.

🛠️ Future Research Improvements

The clearest next research direction is stronger provenance-aware verification. Agents should distinguish independent evidence from mutually consistent artifacts produced by one attacker, expose the provenance of each observation, and require confirmation through channels outside the attacker's control before high-impact actions.

The benchmark also motivates broader causal studies: vary attacker knowledge, verification access, accomplices, content modality, language, and domain independently; test additional attacker and content-construction models; and evaluate whether explicit action receipts, uncertainty calibration, or human escalation reduce both harmful execution and over-refusal.

• Separate the L1-L4 factors experimentally instead of treating each level as one combined condition.
• Measure whether reductions in D4 come from genuine recognition, generic refusal, or failure to complete any task.
• Extend beyond simulated applications to controlled user studies and real agent products where permitted.

🏭 Potential Industry Use Scenarios

The benchmark maps naturally to personal assistants that can pay bills, manage travel, interact with contacts, browse websites, submit forms, or handle work and shopping tasks. Product teams could use its episode structure as a pre-release red-team suite for high-impact tool calls, with execution logs serving as an audit trail for whether an agent actually verified a request or merely stated that it had.

Financial services, enterprise copilots, device assistants, and agent platforms could adapt the same pattern to test impersonated family members, vendors, colleagues, government services, or support staff. The evidence supports using it as a risk-discovery and regression framework, but the simulated setting means deployment decisions should wait for validation on product-specific data, permissions, and escalation policies.

💬 Critical Analysis

The strongest design choice is the outcome definition: success depends on observed monetary loss or information leakage rather than persuasive language or a claimed action. The six-dimensional view is also valuable because it reveals contradictions that a single attack-success metric would hide, such as higher verification coverage alongside higher attack success and lower harmful execution alongside higher deception miss rate.

The main interpretive risk is attribution. Chinese and English tasks differ in cultural context and narrative design, and the difficulty levels combine information access, verification constraints, and accomplices. The paper acknowledges these issues, so cross-language and level comparisons should be treated as risk profiles rather than clean causal estimates.

Reproducibility still depends on details that can materially affect results: the selected model versions, attacker policy, generated artifact quality, turn limits, stopping conditions, and simulated application behavior. A builder reproducing the benchmark should preserve those controls, report benign-task utility, and inspect raw execution traces rather than relying only on aggregate D4 or the agent's prose.

• Strength: execution-grounded harm labels reduce the gap between conversational safety and real tool behavior.
• Strength: ablations show that more verification activity can coexist with worse security.
• Caveat: simulated losses and task-authored fraud cases do not establish real-world incident rates.
• Open question: what verification policy improves both D6 and legitimate-request completion without simply increasing refusal?