Each section below summarizes one paper's reported method, experiments, and limits from its full review. The studies use different tasks and metrics; their results are not directly comparable.
arXiv:2610.39417
Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent
EXLABEL is a self-evolving LLM assistant for identifying underspecified tasks and inadequately tested tasks in SWE-bench-style datasets. It builds an experience bank from historical annotations, retrieves and reranks relevant experience, reflects on prediction errors, and updates its judgment knowledge through insertion, modification, and deletion. On the SWE-TQ benchmark, EXLABEL achieved the best F1 across all tested task/subset/backbone combinations, and filtering SWE-rebench with EXLABEL produced a coding model that resolved 66.2% of SWE-bench Verified issues.
How it works
EXLABEL has an experience accumulation phase and an experience reuse phase. Accumulation starts with an empty bank and processes labeled instances through rollout, reflection, and experience update. Rollout summarizes the task, golden patch, and test patch along task type, task explanation, and test explanation; retrieves candidate experience using BM25 and dense embeddings; reranks candidates; and predicts a label and explanatory notes. Reflection compares the prediction with the golden annotation and generates a structured experience item.
What the paper evaluates
The main evaluation uses SWE-TQ-PYTHON and SWE-TQ-MULTILINGUAL with Qwen3.8-27B, DeepSeek-V4-Flash, and GPT-5.6 Luna. Baselines are Vanilla, SimpleRAG, SPICE, Codex, and Claude Code. F1 is the primary metric because it balances precision and recall for detecting unsolvable or unverifiable instances; FPR measures the proportion of valid instances mistakenly filtered. Token usage and monetary cost are also reported. EXLABEL achieved the best F1 in all 12 task/subset/backbone combinations. Its average relative improvement over the best baseline was 44.2% and 49.9% for task clarity on Python and Multilingual, and 24.0% and 18.4% for test coverage.
Limit to keep in view
SWE-TQ retains only instances with complete annotator agreement and applies additional manual cleaning, so its labels may represent a cleaner and less ambiguous distribution than the broader SWE-bench-style dataset population.
arXiv:2610.32813
HarnessMM: Recursive Self-Improvement for Harness Token Efficiency
HarnessMM is a controller-level method for reducing repeated reasoning in tool-using agents. It stores already-resolved actions in Executable Decision Graphs (EDGs), uses public tool receipts and guards to determine when those actions are eligible, and performs Certified Callback Substitution (CCS) through the original native interface instead of invoking the reasoning model again. The actor weights, tools, task prompts, and scorers remain fixed within each comparison. On HumanEval, HarnessMM preserves Codex success at 90.82% while reducing Tokens/task from 66.72k to 47.02k; for DeepSeek Harness, success increases from 71.43% to 85.71% while Tokens/task falls from 96.28k to 11.62k.
How it works
The actor parameters are fixed while HarnessMM changes the harness route from public tool state to the next native action. At a slow decision, the actor emits the current action and a bounded EDG. Each decision node stores a native action, complete literal arguments, a predecessor, and a public guard. The graph is acyclic, and missing or rejecting receipts prevent a node from being admitted. GDR retains the graph across tool transitions.
What the paper evaluates
The evaluation uses the same tasks, backbone, original tools, task limits, and scorer within each comparison. Success is measured at the benchmark endpoint, while token cost includes input and output for all paid calls, including failures. The central coding populations are HumanEval with 98 test tasks and MBPP-sanitized with 257 test tasks. Companion configuration studies cover ALFWorld, SearchQA, LiveMath, AIME, Terminal-Bench 2, and GAIA2; the harness-evolution study uses MiniWoB++ with 32 tasks per arm. The main mechanism table reports the HumanEval and MBPP-sanitized pairs.
Limit to keep in view
Savings depend on the workload exposing actions whose semantics and literal arguments are already resolved; genuinely new semantic decisions still require the slow actor.
arXiv:2610.33799
MTA-Zero: Self-Improving Multimodal Deep Search via Online QA-Solver Co-training
MTA-Zero co-trains a multimodal deep-search Solver and a separate QA Generator, letting the training questions adapt as the Solver improves. Starting from image-grounded one-hop VQA seeds, the Generator searches for evidence and constructs verified multi-hop questions; the Solver answers each question through independent tool-use rollouts. The Solver’s group success rate rewards questions that are challenging but still solvable, and each policy is updated independently with GRPO. In the authors’ reported evaluation, the Qwen3-VL-8B agent’s six-benchmark mean accuracy rose from 36.91% to 51.58%. The approach offers builders a way to create a policy-specific curriculum online, while depending on a verification harness, web access, and substantial repeated rollouts.
How it works
At each iteration, the system starts with a seed context C = (I, q1, a1): an image, an initial question, and its answer. The trainable QA Generator extends that context by searching for information about the preceding answer and proposing a new question, answer, evidence, and source. The fixed harness filters retrieved pages into evidence chunks, checks whether each hop is anchored to and requires the current entity, and tests answer uniqueness, temporal stability, and self-contained wording.
What the paper evaluates
The evaluation covers six multimodal deep-search benchmarks: MMSearch-Plus (MMSrch-Plus), HR-MMSearch-VL (HR-MMSrch), BrowseComp-VL (BC-VL), MMSearch-VL (MMSrch), the testmini split of VDR-Bench (VDR), and the official FVQA test split. Training seeds come from InfoSeek, the FVQA training split, InfoVQA, and OK-VQA; these provide image-grounded one-hop contexts rather than pre-generated multi-hop questions. The paper reports mean accuracy over three generation seeds and an unweighted six-benchmark average when all six scores are available. Higher accuracy is better.
Limit to keep in view
The comparisons with specialist systems reported from prior papers are not cost-matched and may use different tool stacks, training pipelines, budgets, or evaluation settings.
arXiv:2611.01736
RSI-Env: An Open-Ended Environment for Recursive Self-Improvement of Agents
RSI-Env is a persistent environment for studying whether agents can improve both model weights and the harness around a model, then have an accepted successor continue the work. Its Host Controller manages training and evaluation services, validates candidate systems, and preserves version history while keeping private evaluation criteria outside the editable sandbox. Across Open-General and Capability-Directed runs, Self improves selected benchmark results without benchmark feedback; in the Knowledge run, SimpleQA avg@3 rises from 82.0% to 85.0% and then 88.0% across successive system versions. Results are selective: some gains accompany regressions on other benchmarks, and fixed external Teachers do not consistently outperform Self.
How it works
An RSI system is represented as a target model, executable harness, persistent research state, and resource pool. The state retains experimental code, records, and intermediate artifacts; resources include libraries, inference and training services, evaluation services, network access, and computing infrastructure. At each iteration, the Improver inspects the system and retained records, chooses whether to change the model parameters, harness, or both, and proposes an update.
What the paper evaluates
The evaluation begins from Qwen3.8-27B with a minimal harness and compares Self with two fixed external Improvers, GPT-5.6 Sol and GPT-6 Astra, across seven tasks in the three settings. The common allocated budget is ten rounds of three hours, totaling thirty hours, although runs can finish early and the table reports actual time used. Model–harness pairs are evaluated with Harbor on four benchmarks: GPQA Diamond, AIME 2026, Terminal-Bench 2.1, and SimpleQA. The paper reports three evaluations per item, using avg@3 for the first, second, and fourth benchmarks and pass@3 for Terminal-Bench 2.1.
Limit to keep in view
The reported experiments start from Qwen3.8-27B with a minimal harness and cover seven tasks; the evidence does not establish that the same behavior generalizes to other starting systems or environments.
Read the full reviews
- Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent · arXiv:2610.39417
- HarnessMM: Recursive Self-Improvement for Harness Token Efficiency · arXiv:2610.32813
- MTA-Zero: Self-Improving Multimodal Deep Search via Online QA-Solver Co-training · arXiv:2610.33799
- RSI-Env: An Open-Ended Environment for Recursive Self-Improvement of Agents · arXiv:2611.01736
Source note: findings and limitations above come from the linked full paper reviews. Follow each original arXiv paper for the authors' complete methods and results.