MDSKILL: Learning to Internalize Skill Memory through Multi-View Distillation for Efficient Dynamic Agent Self-Evolution

📌 Highlights

MDSKILL targets a specific weakness of memory-augmented agent learning: successful and failed experiences are compressed into textual skills, but those skills may be unreliable, redundant, or conflict because they serve different decision functions. The framework partitions them into state grounding, action planning, and execution control views, then adds a combined branch. Its central proposal is to explore with these views and transfer knowledge among branches only when paired trajectory outcomes and a knowledge-gain gate support doing so.

The authors report overall ALFWorld success improvements over UCOB of 7.1, 3.1, and 2.3 percentage points for Qwen3-1.7B, Qwen2.5-3B-Instruct, and Qwen2.5-7B-Instruct, respectively. WebShop success improvements are 8.1, 3.2, and 3.9 points for those backbones. On Search-QA, results are competitive rather than uniformly higher. The paper’s ablations report that removing distillation causes the greatest degradation, and removing multi-view rollouts has the largest effect among variants retaining distillation. A key caveat is that the evidence is bounded to the reported benchmarks, backbones, and training setups.

• Dynamic retrieval combines query relevance with sampled reliability and advantage estimates whose evidence decays over time.
• Distillation includes forward, reverse, and cross-view directions, selected through outcome comparisons and knowledge-gain gates.
• Interactive-task gains are stronger and more consistent than the Search-QA average accuracy results.

🎯 Introduction

The paper frames agentic reinforcement learning as difficult in multi-turn tasks because rewards may be sparse or delayed. Textual skill memories summarize interaction experience and can guide later decisions, but the stated objective for a self-evolving agent is to internalize that experience into model parameters so that useful capabilities persist without explicit memory retrieval. On-policy self-distillation offers token-level supervision from skill-conditioned trajectories to a skill-free branch, complementing trajectory-level RL.

The limitation motivating MDSKILL is that online skill memories are self-generated and may encode ineffective, harmful, or hallucinated strategies. The authors’ analysis also argues that jointly applying skills with different decision functions can entangle their roles, while redundant or conflicting instructions obscure individual contributions. Direct distillation can therefore propagate erroneous behavior or deliver inconsistent supervision; a skill-conditioned teacher is not guaranteed to be a better teacher than its skill-free counterpart.

The paper’s objective is to improve the quality and direction of knowledge transfer. It separates skill guidance by function, dynamically favors memories with evidence of current value, and uses trajectory outcomes to select transfer pairs. The intended loop is reciprocal: the policy internalizes useful skill knowledge, then generates better trajectories and updated skills for subsequent learning. The figure below gives the system-level overview.

🔬 Methodology

The agent receives a task instruction and interaction history. It can act without a retrieved skill or condition its action on a memory; both branches share policy parameters. The paper uses GiGPO as the underlying RL method, combining episode-level and step-level advantages for sparse or delayed reward credit assignment. MDSKILL generates functional skill pools through view-specific reflection: grounding extracts relevant evidence and unknowns, planning specifies subgoals and action order, and control checks preconditions, verifies outcomes, and supports recovery. A combined view integrates all three.

For retrieval, candidate skills are first filtered by cosine similarity between the current query and each memory’s associated context. Reliability evidence is task success; advantage evidence is measured relative to average skill-free success on the same task. A uniform-prior Beta distribution tracks each estimate, with temporal decay so recent evidence weighs more. Thompson sampling draws reliability and advantage values for candidate skills, whose sampled values are summed to rank memories. This both uses evidence and preserves exploration among uncertain candidates.

For distillation, decision records are grouped by task and anchor state. The method compares counts of successful skill-free and skill-view trajectories and applies strict or relaxed knowledge-gain gates. Valid teacher-student pairs must have a better trajectory outcome for the teacher. Depending on branch identity and gate, transfer is forward from skill-view to skill-free, reverse from skill-free to skill-view, or cross-view between distinct skill views. Token-level KL distillation uses the teacher’s normalized token distribution with teacher gradients stopped. The full training objective adds this term to RL and reference-policy KL regularization. The formulas capture retrieval evidence, pair selection, token loss, and combined optimization.

📊 Experiments

The authors evaluate on ALFWorld, WebShop, and Search-QA with Qwen3-1.7B, Qwen2.5-3B-Instruct, and Qwen2.5-7B-Instruct backbones. ALFWorld reports success across Pick, Look, Clean, Heat, Cool, and Pick2 and an overall success rate. WebShop uses an established 128-task evaluation split and reports task score and success rate. Search-QA covers three single-hop and four multi-hop datasets and reports answer accuracy, including per-dataset and average values. Baseline families include skill-free RL, skill-augmented agents, and self-distillation methods; the paper says MDSKILL follows the corresponding baseline settings.

Authors’ reported main comparison: MDSKILL has higher ALFWorld and WebShop success than UCOB for all three backbones, with gains of 7.1, 3.1, and 2.3 points on ALFWorld and 8.1, 3.2, and 3.9 points on WebShop as backbone size increases. The table also shows Search-QA averages of 42.1, 43.4, and 48.4 for MDSKILL, compared with UCOB’s 41.9, 43.5, and 48.0. Thus the interactive success improvements do not translate into uniform Search-QA wins. In the paper’s interpretation, reusable skills are more directly helpful when tasks revisit interactive environment states.

The authors’ ablation analysis reports that removing distillation produces the greatest degradation, and removing multi-view rollouts has the largest impact among variants that retain distillation. Removing forward distillation also reduces performance within MDSKILL, while cross-view distillation contributes further gains. Distributional analyses report higher token overlap and a smaller, more stable entropy gap for MDSKILL than without multi-view rollouts; cross-view distillation has the lowest entropy gap and highest overlap in that comparison. Skill-evolution analysis reports that the final instance-specific share among high-advantage skills is 26% for MDSKILL versus 42% for UCOB. These findings support the proposed mechanism in the tested setup, but do not independently establish compute parity beyond the paper’s claim of no additional computational overhead.

fig:mdskill_overview_main

Caption: Detailed overview of MDSkill, comprising dynamic skill retrieval, multi-view rollout exploration, and knowledge-gain-driven multi-view distillation.

Why It Matters: Shows the three main components and how skill retrieval, rollout branches, trajectory pairing, and shared-policy updates fit together.

fig:main_ablation

Caption: Main ablation across training.

Why It Matters: The reported ablation curves compare full MDSKILL with removals of distillation components and multi-view rollouts across training.

Figure ure

Caption: Figure 6: Ablation study of MDSKILL. 5.3 ABLATION STUDIES

Why It Matters: The main-body figure includes distributional stability and skill-evolution analyses; its caption extraction includes a neighboring section heading, so inspect the image when interpreting details.

🔮 Conclusion

The authors conclude that multi-view skill organization, dynamic memory selection, and outcome-conditioned distillation can reduce interference from unreliable guidance and help consolidate useful experience in a shared policy. Their ALFWorld and WebShop evaluations show higher overall success than the strongest compared baseline across the tested backbones; ablations and skill-evolution analyses are presented as evidence for the proposed components and internalization process.

For builders, the practical takeaway is to make memory transfer conditional on measured outcomes and to distinguish kinds of guidance before treating retrieved text as a teacher. The paper’s results support that design on its evaluated interactive tasks, while its Search-QA averages and limited benchmark scope counsel against assuming the same benefit across agent workloads.

🛠️ Future Research Improvements

The main-body evidence motivates several follow-up questions without claiming that the paper has already answered them. First, evaluate the approach on additional interactive environments and tasks where skill memories have different levels of reuse, since the current results cover ALFWorld, WebShop, and Search-QA and the authors associate the strongest utility with recurring environment states. Such work should report whether grounding, planning, and control remain useful partitions across domains.

Second, measure training compute, inference overhead, retrieval cost, and wall-clock latency under matched settings. The paper claims no additional computational overhead, but the main-body evidence supplied here does not give a direct accounting of those quantities. Third, probe sensitivity to the temporal half-life, Top-K retrieval, strict versus relaxed gates, and the balance among RL, distillation, and reference-policy regularization; the relevant detailed configurations are referenced outside the evidence boundary.

Finally, test robustness when memories contain stale or adversarially misleading advice and examine whether the reliability and advantage estimates respond quickly enough to policy changes. These are suggested validation directions derived from the paper’s stated risks of noisy and harmful self-generated skills, rather than reported experimental findings.

🏭 Potential Industry Use Scenarios

The method is most directly relevant to products built around repeated, multi-step interactions where an agent accumulates textual procedures from past attempts. Examples consistent with the evaluated task types include household-task agents that navigate and manipulate objects, and shopping agents that search for and select products under natural-language constraints. In these settings, grounding memories could help interpret observations, planning memories could sequence subgoals, and control memories could verify progress or prompt recovery.

A deployment could store memories by functional role, retrieve candidates using task context and current evidence about past usefulness, and periodically train a shared policy from successful and unsuccessful trajectory pairs. If the reported internalization behavior transfers, this could reduce reliance on repeatedly injecting the same memory into prompts. The paper’s findings are evidence from benchmark settings, however; they do not establish benefits in live commerce, physical devices, or other production environments.

Before operational use, builders would need to validate memory freshness, task-specific success definitions, failure recovery, retrieval quality, and resource costs. Search-QA’s less uniform gains are a reminder that the method may be less valuable when performance depends more on retrieval and answer synthesis than on reusable interactive decision patterns. Treat these as candidate application areas and validation requirements, not demonstrated deployments.

💬 Critical Analysis

A strength of MDSKILL is that it makes the supervision problem explicit: a skill-conditioned branch can be wrong, and a dense distillation loss can amplify rather than correct that error. The method’s separation of guidance into functional views is paired with an outcome-based transfer gate, giving the proposal a testable connection between architecture and training signal. The reported ablations are directionally consistent with that account: removing distillation hurts most, multi-view rollouts matter among distillation variants, and cross-view transfer contributes further improvement.

The evidence also has limits. The full Table 1 includes substantial per-task variation, and MDSKILL does not lead every listed Search-QA average or every individual benchmark column. The summary’s headline gains concern overall ALFWorld and WebShop success against UCOB, not every metric or baseline. WebShop task score and success rate must be kept separate; for example, MDSKILL’s success improves while its score is below UCOB for the three displayed backbone comparisons. Search-QA averages are close and mixed.

Reproducibility requires careful attention to the distinction between author-reported experiments and comparison entries attributed to original papers or reported by other methods, as indicated in the Table 1 caption. The paper says baseline settings are followed, but detailed configuration is outside the supplied main-body evidence. The reported stability and skill-evolution figures offer useful mechanism-level evidence, although they do not by themselves prove why gains occur or establish generality. In particular, the claimed absence of extra compute overhead should be checked with an explicit resource comparison. Overall, the paper presents a coherent and practically motivated method with promising interactive benchmark results; deployment confidence depends on reproducing the setup, auditing compute and memory behavior, and testing beyond the reported environments.