Each section below summarizes one paper's reported method, experiments, and limits from its full review. The studies use different tasks and metrics; their results are not directly comparable.
arXiv:2611.08478
Scientist World: Grow a Research World for AI-Scientists
Scientist World turns published empirical research papers into executable environments where AI scientists can improve a reproduced method or devise a new one under task-specific constraints. Its builder workflow reproduces the source result, defines a scored task with visible and held-out evaluation, packages the environment, and has an independent verifier audit it. The release contains 1,000 environments; on a fixed cohort of ten lower-compute environments, models beat the reproduced method in 84% of improve attempts and 74% of innovate attempts. In the reported leaderboard, Fable 5 leads improve mode at mean FNI 118.5, while Fable 5.1 leads innovate mode at 76.3.
How it works
The environment-building workflow uses four roles that work in sequence on a shared workspace. The reproducer implements the paper’s method and evaluation, using released code where available or implementing from scratch otherwise, and scales up runs until the main result is recovered. The designer turns that reproduction into a task by specifying a problem statement, target metric, permitted directions, and constraints. It creates visible and held-out evaluation sets and uses the source result as the baseline.
What the paper evaluates
The model benchmark uses a fixed cohort of ten lower-compute environments because running every model on all 1,000 would be too costly to repeat. Every model attempts each environment under the same compute budget in two settings: improve mode starts from the reproduced source method, while innovate mode gives the problem without the source code and is intended to prevent leakage of the original solution. Scores are evaluated on both visible and held-out splits and averaged across the two. FNI is oriented so zero matches the source method and higher is better. The central reported rates are 84% of improve attempts and 74% of innovate attempts beating the reproduced method.
Limit to keep in view
The model benchmark uses a fixed cohort of ten lower-compute environments rather than all 1,000, so leaderboard conclusions may not generalize to the full collection or its range of fields.
arXiv:2611.03424
IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?
IdeaTaste evaluates whether LLM-generated research proposals resemble the work human research communities publish after a frozen literature snapshot. Across six language models, four areas, and more than 90,000 proposals, agents reach about 26% of the human level on citation matching and 18% on idea matching. They can cover community tasks and methods and often cite diversely, but rarely reproduce the concrete approaches and research plans later pursued by humans. Blinded reviews show the largest quality gap in expected success, indicating that the central weakness is translating a recognized research direction into a viable mechanism and execution plan.
How it works
For each of four areas—machine learning systems, code generation, world models, and automated research—the corpus U contains papers published through 2025 and the reference H contains screened 2026 papers. Both are built around 20 to 22 area landmarks: the corpus follows landmark citations and adds papers cited by at least two citing papers, while the reference combines landmark-citing papers with topic-search results and removes surveys.
What the paper evaluates
The evaluation covers machine learning systems, code generation, world models, and automated research, using six models: Llama 4 Scout, Claude Haiku 4.5, Gemini 3.5 Flash-Lite, DeepSeek-V4-Flash, Qwen3.8-27B, and K2 Horizon 375B A23B. Four models were released in 2026 and may overlap the reference period; Llama 4 Scout and DeepSeek-V4-Flash were trained before 2026 and cannot have seen 2026 papers. Populations are scored on citation matching, citation diversity, idea matching, and idea diversity, where higher values mean closer agreement or greater variety.
Limit to keep in view
The human reference covers January through August 2026, and the final month is incomplete, so the observed future record is partial and may change as additional papers appear.
arXiv:2611.01747
Evolving Memory, Learning to Meta-Control: Test-Time Adaptation for Experience-augmented LLM Agents
Online Meta-Control of Experience (OMCE) equips an LLM agent with episodic and guidance memories plus a lightweight contextual-bandit controller. For each incoming task, the controller uses compact retrieval features to choose among DIRECT, EPISODE, GUIDE, and DISTILL: bypass memory, inject a prior case, inject reusable guidance, or distill guidance from retrieved episodes before execution. Task feedback updates both the memory and the routing policy. Across MultiChallenge, BBEH, BigCodeBench, and SWE-bench Verified, using four backbone models, OMCE reports the highest first-attempt success on 11 of 16 benchmark–backbone configurations.
How it works
For each query in a sequential stream, OMCE retrieves candidate episodic insights and guidance entries by cosine similarity between the query embedding and each entry's applicable-scenario description. Episodic records contain the task trajectory, immediate reward, and case-specific insights. Guidance entries encode principles intended to transfer across related cases. The controller summarizes retrieval relevance, quality, and episodic outcome diversity in a compact state rather than reading raw memory text.
What the paper evaluates
The evaluation uses four online-adaptation benchmarks: MultiChallenge for multi-turn conversation, BBEH for advanced general reasoning, BigCodeBench for code generation, and SWE-bench Verified for repository-level software engineering. Four backbone models are evaluated: GPT-5.2, DeepSeek-V3.2, DeepSeek-V4-pro, and Kimi-K2.5. The comparison includes a base LLM, Reflexion, MemP, ReasoningBank, ACE, and OMCE variants. One-shot success rate (OSR) is the primary metric and measures success on the first attempt; cumulative success rate (CSR) measures success within P = 4 attempts. Reflexion is reported with CSR only because its first attempt matches the base model under the protocol.
Limit to keep in view
The controller depends on timely task feedback; sparse or delayed rewards may make its updates noisy.
arXiv:2611.05313
MDSKILL: Learning to Internalize Skill Memory through Multi-View Distillation for Efficient Dynamic Agent Self-Evolution
MDSKILL addresses a problem in agent self-evolution: textual skill memories can improve interaction, but noisy or conflicting guidance can make it hard to transfer useful behavior into a model’s policy. The method separates skills into state grounding, action planning, and execution control views, selects candidate memories using relevance plus dynamically updated reliability and advantage estimates, and trains a shared policy with reinforcement learning and outcome-filtered distillation across skill-free, skill-guided, and cross-view trajectories. On ALFWorld and WebShop, the authors report higher overall success rates than the strongest compared baseline across three model backbones; Search-QA results are competitive.
How it works
The agent receives a task instruction and interaction history. It can act without a retrieved skill or condition its action on a memory; both branches share policy parameters. The paper uses GiGPO as the underlying RL method, combining episode-level and step-level advantages for sparse or delayed reward credit assignment.
What the paper evaluates
The authors evaluate on ALFWorld, WebShop, and Search-QA with Qwen3-1.7B, Qwen2.5-3B-Instruct, and Qwen2.5-7B-Instruct backbones. ALFWorld reports success across Pick, Look, Clean, Heat, Cool, and Pick2 and an overall success rate. WebShop uses an established 128-task evaluation split and reports task score and success rate. Search-QA covers three single-hop and four multi-hop datasets and reports answer accuracy, including per-dataset and average values. Baseline families include skill-free RL, skill-augmented agents, and self-distillation methods; the paper says MDSKILL follows the corresponding baseline settings.
Limit to keep in view
The reported evaluation covers three benchmarks and the specified Qwen backbones; the supplied main-body evidence does not establish performance across other agent environments or model families.
Read the full reviews
- Scientist World: Grow a Research World for AI-Scientists · arXiv:2611.08478
- IdeaTaste: Do LLMs Share the Research Taste of Human Researchers? · arXiv:2611.03424
- Evolving Memory, Learning to Meta-Control: Test-Time Adaptation for Experience-augmented LLM Agents · arXiv:2611.01747
- MDSKILL: Learning to Internalize Skill Memory through Multi-View Distillation for Efficient Dynamic Agent Self-Evolution · arXiv:2611.05313
Source note: findings and limitations above come from the linked full paper reviews. Follow each original arXiv paper for the authors' complete methods and results.