CoSA: The Conflict of Skills Attack in Real-World LLM Agent Ecosystems

📌 Highlights

CoSA identifies a compositional security gap in LLM agent ecosystems: Skills can pass individual checks while their outputs, decisions, or target bindings become unsafe when reused by other Skills. The strongest result is a pooled policy-violation increase from 35.02% to 58.45% under additions capped at 5% of each original pool. The main limitation is scope: the benchmark uses fixed tasks, bounded augmentation, selected domains, and a limited set of models and harnesses.

• 288 CoSA-Bench tasks span recruiting, finance, IT, and business.
• The benchmark covers data leakage, unauthorized state modification, and unsafe execution.
• ASR increases in all 15 model–harness combinations.
• Random benign injection lowers OpenClaw ASR to 26.39%, while full CoSA reaches 54.24%.
• CoSA-Guard reduces OpenClaw recruiting violations from 65.28% to 6.67%, with PCTC changing from 66.67% to 65.00%.

🎯 Introduction

Modern LLM agents compose reusable Skills containing instructions, code, and resources. Existing security work largely treats the individual Skill package as the unit of analysis, checking for prompt injection, data exfiltration, credential theft, unsafe code, or behavior manipulation. CoSA argues that this leaves a workflow-level gap: a Skill may produce an artifact or assessment that is safe for its original purpose but unsafe when another Skill reuses it downstream.

The paper formalizes this failure as Conflict of Skills and introduces CoSA, an attack in which locally admissible Skills are added without modifying existing Skills, trusted instructions, models, tools, or runtime permissions. The benchmark fixes requests and initial states before candidate generation and evaluates whether the augmented pool changes concrete policy outcomes under autonomous Skill selection.

• The task policy distinguishes concrete data leakage, unauthorized persistent mutation, and unauthorized executable workloads from merely reporting or preparing an action.
• A valid conflict requires evidence of cross-Skill reuse contributing to the violation; temporal ordering alone is insufficient.

🔬 Methodology

The method represents each task as a user request, initial environment, and finite constraint set. An execution trajectory records Skill reads, tool calls, results, and intermediate artifacts. CoSA assigns operations functional and consequential roles, then checks whether outputs or decisions from one Skill are reused by another in a way that contributes to a violated constraint. The formal condition requires at least two participating Skills, a dependency witness, and individual admission of every Skill.

Controlled conflict discovery examines Skill instructions, scripts, interfaces, permissions, side effects, assessment signals, authorization conditions, and operation roles. Four recurring mechanisms guide candidate construction: security-context attenuation, semantic interface bridging, semantic authority promotion, and target/action binding drift. Capability supplementation adds a missing business operation; relationship reshaping changes artifact exchange, decision propagation, or action-target binding. Candidates are retained when their proposed handoffs are supported by domain contracts and they pass local admission, then selected under a budgeted maximum-coverage rule.

CoSA-Bench compares original pools $V_D$ with augmented pools $V_D\cup I_D$ using paired runs with identical requests, initial states, policies, original Skills, permissions, tools, and execution budgets. CoSA-Guard is implemented as a pre-action harness hook: before externalization, persistent mutation, or execution, it checks information flow and authorization and can permit, block, request confirmation, or report an error.

📊 Experiments

The central benchmark is CoSA-Bench: 288 tasks, with 72 each in recruiting, finance, IT, and business, evenly divided among data leakage, unauthorized state modification, and unsafe execution. The evaluation uses five LLMs—Qwen3.6, MiniMax-M3, DeepSeek-V4, GPT-5.6, and GLM-5.1—and three harnesses: OpenClaw, OpenHands, and OpenCode. Every model–harness–domain–risk combination is evaluated under original and augmented pools, producing 180 groups. ASR is the empirical percentage of runs with a concrete policy violation; OTR and ITR measure explicit reads of original and added Skills.

Across all settings, violations increase from 1,513/4,320 (35.02%) to 2,525/4,320 (58.45%), a 23.43 percentage-point increase. The harness-level changes are +22.29 pp for OpenClaw, +26.04 pp for OpenHands, and +21.94 pp for OpenCode. ASR increases in all 15 model–harness combinations, ranging from +8.33 pp for GPT-5.6 in OpenCode to +34.72 pp for MiniMax-M3 in OpenHands. Domain–harness increases range from 18.89 to 28.61 pp, and data leakage is the largest increase in every harness: 25.00 pp in OpenClaw, 31.04 pp in OpenHands, and 25.21 pp in OpenCode.

The retrieval results weaken a simple exposure explanation. Original-Skill retrieval changes by only -1.04, -1.25, and +0.97 pp across OpenClaw, OpenHands, and OpenCode, while ITR correlates moderately with augmented ASR but only weakly with ΔASR. In the OpenClaw ablation, random benign injection lowers ASR to 26.39%, random selection from checked CoSA candidates reaches 43.40%, and full CoSA reaches 54.24%. The defense evaluation on 72 OpenClaw recruiting tasks across five models reduces ASR from 235/360 (65.28%) to 24/360 (6.67%), with Risk Coverage of 115/139 (82.73%). On matched benign controls, PCTC changes from 240/360 (66.67%) to 234/360 (65.00%), while FBR is 12/360 (3.33%).

fig:intro

Caption: Motivating example. Although each Skill passes the individual security checks, their composition leads to unauthorized execution of code from a submitted package.

Why It Matters: This illustrates the central compositional failure: individually benign Skills can jointly enable an unauthorized consequential action.

fig:cosa-overview

Caption: Overview of CoSA. CoSA constructs a composition surface from the original Skill pool, identifies mechanism-guided augmentation opportunities, and generates, admits, aligns, and selects a budgeted set of locally admissible Skills. Original and augmented pools are then compared under matched execution conditions with autonomous Skill selection. An optional runtime guard checks information flow and authorization before consequential actions.

Why It Matters: The figure connects conflict discovery, candidate construction, budgeted selection, paired evaluation, and runtime defense into one end-to-end framework.

fig:exp-models

Caption: Original and CoSA-augmented ASR across model–harness combinations. Labels show ΔASR in pp; lines connect condition-level rates. ASR increases across domains and risk categories. Columns correspond to OpenClaw, OpenHands, and OpenCode; values are in pp.

Why It Matters: It visualizes that augmentation increases attack success across all model–harness combinations, while the magnitude varies by system.

fig:exp-defense

Caption: The CoSA-Guard on OpenClaw recruiting tasks. (a) ASR under attack conditions; (b) PCTC on matched benign controls. Detailed counts are in Appendix~sec:exp-defense-detail.

Why It Matters: It summarizes the security–utility trade-off of the proposed guard on the defense evaluation.

🔮 Conclusion

The paper's central conclusion is that securing Skills independently is insufficient for securing the workflows they form. CoSA demonstrates that a small set of locally admitted additions can change how existing capabilities exchange artifacts, reuse decisions, and bind actions to targets without expanding runtime permissions. The benchmark results and trace analysis support evaluating composition and downstream effects rather than relying only on package admission or retrieval counts.

CoSA-Guard provides initial evidence that a harness-level check for information flow and authorization can sharply reduce observed violations. However, residual data leakage and the limited defense setting show that runtime gating is a partial mitigation; stronger preservation of provenance and permitted-use constraints across transformations remains necessary.

🛠️ Future Research Improvements

The clearest next step is to extend evaluation beyond the current fixed, bounded setting: longer-horizon tasks, more complex dependency chains, dynamic Skill installation, additional domains, and more harness implementations would test whether the observed compositional gap persists. Future studies should also isolate capability supplementation from relationship reshaping and evaluate how candidate-selection policies affect risk under different budgets.

The defense should be tested across all three risk categories and additional agent environments with richer authorization systems. Because the residual ASR for data leakage is higher than for unauthorized state modification and unsafe execution in the reported defense setting, future work should strengthen provenance propagation and release-scope checks rather than relying only on terminal action blocking.

• Separate the four conflict mechanisms through controlled ablations.
• Evaluate repeated runs and longer trajectories to distinguish stochastic execution effects from stable composition effects.
• Test whether authorization labels and provenance survive transformations across multiple intermediate Skills.
• Report both risky-action production and conditional violation rates for every defense setting.

🏭 Potential Industry Use Scenarios

The paper directly exercises recruiting, finance, IT, and business workflows, making enterprise agent platforms a natural application area. A Skill registry could combine package-level admission with composition-aware validation whenever new Skills are added to a domain pool. Runtime systems could attach provenance and permitted-use metadata to documents, assessments, approvals, and generated commands, then enforce checks before sharing data, changing persistent records, or starting executable workloads.

A practical deployment pattern is a harness-level policy gate around consequential tools. Such a gate could support recruiting systems handling candidate data, finance agents modifying records, IT agents executing operational commands, and business agents sending restricted information. These scenarios are plausible extensions of the evaluated task types, but the paper does not establish production reliability, so they require domain-specific authorization policies, audit logging, false-block measurement, and broader testing.

• Enterprise Skill registries with composition-aware admission.
• Provenance-aware document and decision handoffs.
• Pre-action authorization gates for data release, record mutation, and code execution.
• Security regression tests for every Skill-pool update.

💬 Critical Analysis

The study is strongest in its controlled causal framing. It fixes the task, environment, original Skills, permissions, and execution budgets while varying the Skill pool, and it distinguishes retrieval from realized policy outcomes. The paired benchmark, cross-harness results, trace-supported attribution, and ablation controls make a credible case that selected additions can amplify risk through composition rather than simple package-level maliciousness or pool-size expansion.

The main uncertainty is external validity. The experiments cover a bounded set of domains, models, harnesses, policies, and augmentation sizes, and each task condition is executed once on the prespecified grid. Aggregate ASR does not itself establish a Conflict of Skills witness, although the paper supplements it with trajectory analysis. The random ablations also use one sampled realization and do not independently isolate the two augmentation types. Finally, CoSA-Guard's large ASR reduction is promising but was measured in one harness and domain; its residual leakage, false blocks, and model-dependent PCTC changes show why broader authorization and provenance evaluations are still needed.