RSI-Env: An Open-Ended Environment for Recursive Self-Improvement of Agents

📌 Highlights

RSI-Env provides a persistent setting for agents to explore changes to model weights and harness code, retain research state, and pass accepted systems into later improvement rounds. Its Host Controller supplies services, checks whether candidates are valid and executable, manages version history, and keeps independent capability evaluation outside the editable environment.

Across the reported runs, Self makes selective gains without benchmark feedback. Open-General Self raises GPQA Diamond avg@3 from 65.2% to 73.2% and AIME 2026 avg@3 from 53.3% to 56.7%. In the Capability-Directed Knowledge run, SimpleQA avg@3 progresses from 82.0% to 85.0% and then 88.0% across checkpoints. These results demonstrate behavior in the studied runs, while declines on other benchmarks show that target gains do not imply broad capability preservation.

• The Improver can modify the model, harness, or both, with persistent records carried across accepted updates.
• The three task settings vary how much goal definition and benchmark feedback the agent receives.
• Self and the fixed external Teachers have mixed results across tasks; neither Teacher configuration consistently leads.
• The evidence is limited to the reported tasks and starting system, with cross-benchmark regressions in some runs.

🎯 Introduction

The paper frames agent performance as depending on both an underlying language model and its surrounding harness: prompts, context management, tool policies, and execution logic. Improving an agent can therefore require decisions about training, harness design, and coordination between them. The authors observe that these decisions remain largely human-directed, motivating the question of whether agents can organize and sustain improvement of their own systems.

Prior work described in the paper has enabled agents to construct training data, refine training algorithms, modify harnesses, or optimize model weights and workflows together. The stated limitation is that objectives, editable components, update procedures, or evaluation feedback are typically specified in advance. That constrains study of how an agent identifies an improvement agenda, coordinates changes across components, and carries research forward through updated systems. RSI-Env is proposed as shared infrastructure for exploring those responsibilities while varying how much goal and evaluation guidance the Improver receives.

🔬 Methodology

An RSI system is represented as a target model, executable harness, persistent research state, and resource pool. The state retains experimental code, records, and intermediate artifacts; resources include libraries, inference and training services, evaluation services, network access, and computing infrastructure. At each iteration, the Improver inspects the system and retained records, chooses whether to change the model parameters, harness, or both, and proposes an update. The transition equation captures how executing that update yields a successor system, with randomness possible in training and tool execution.

RSI-Env separates autonomous exploration from a Host Controller. The agent works in a sandbox where candidate changes can be developed and tested separately from the accepted system. The Controller coordinates services, validity checks, independent evaluation, and version and lineage management through controlled interfaces. The candidate is registered when valid and executable; capability changes are assessed independently. Failed attempts and diagnostics remain in persistent state, and the system can expose task-specific evaluation feedback only where the task permits it.

Under Self, the current accepted model–harness pair also acts as Improver, so an accepted successor conducts the next round. Under Teacher, an external Improver changes the target while its own model and harness remain fixed. The three settings progressively reduce the agent's need to formulate the problem: Open-General leaves direction open; Capability-Directed supplies a broad reasoning, knowledge, or agentic goal; Benchmark-Guided names a benchmark and supplies its evaluation feedback. All runs start from Qwen3.8-27B with a minimal harness. The experiment budget is ten rounds of three hours, and training support uses the Tinker SDK for supervised fine-tuning and custom reinforcement-learning workflows.

📊 Experiments

The evaluation begins from Qwen3.8-27B with a minimal harness and compares Self with two fixed external Improvers, GPT-5.6 Sol and GPT-6 Astra, across seven tasks in the three settings. The common allocated budget is ten rounds of three hours, totaling thirty hours, although runs can finish early and the table reports actual time used. Model–harness pairs are evaluated with Harbor on four benchmarks: GPQA Diamond, AIME 2026, Terminal-Bench 2.1, and SimpleQA. The paper reports three evaluations per item, using avg@3 for the first, second, and fourth benchmarks and pass@3 for Terminal-Bench 2.1. GPQA Diamond uses a 198-question subset, AIME 2026 has 30 problems, Terminal-Bench 2.1 has 89 terminal tasks, and SimpleQA uses a random subset of 500 short factual questions.

In Open-General, Self improves GPQA Diamond from 65.2% to 73.2% and AIME 2026 from 53.3% to 56.7%, while Terminal-Bench 2.1 falls from 12.4% to 9.0% and SimpleQA from 82.0% to 77.2%. Under Capability-Directed goals, Reasoning Self reaches 69.2% on GPQA Diamond, Agentic Self reaches 14.6% pass@3 on Terminal-Bench 2.1, and Knowledge Self reaches 88.0% on SimpleQA. Each run also has changes on other benchmarks, including declines. The table records the separate Teacher outcomes and the Self-versus-Teacher variation by task.

Benchmark-Guided Self scores 60.0% on AIME 2026 and 18.0% pass@3 on Terminal-Bench 2.1, compared with 56.7% and 14.6% under the corresponding Open-General and Capability-Directed runs described by the authors. Its SimpleQA score is 65.6%, below the 77.2% Open-General and 88.0% Knowledge Self results. The authors also report that Knowledge Self advances from 82.0% to 85.0% at H1/M0 and to 88.0% at H1/M1; the updated system constructs 48 prompt–completion pairs and trains M1. This is evidence of a successive update sequence in that case, not evidence that all runs sustain gains. Across Open-General and Capability-Directed, Self uses 24.90–28.95 hours, GPT-5.6 Sol 1.19–1.27 hours, and GPT-6 Astra 3.91–6.97 hours; the authors note that only Reasoning and Knowledge Self accept model updates in those runs.

fig:rsi-env-framework

Caption: RSI-Env architecture. The RSI Environment supports autonomous open-ended exploration in a sandbox and retains working results. The Host Controller provides services through controlled interfaces and maintains private evaluation data and version management.

Why It Matters: Shows the separation between sandboxed exploration and controller-managed services, evaluation, validity checks, and version management.

fig:task-contracts

Caption: Three task settings for assessing progress toward RSI, varying goal specificity and the availability of evaluation feedback. Increasing external guidance reduces the problem-formulation responsibilities assigned to the agent.

Why It Matters: Clarifies how the three experimental settings change the responsibilities assigned to the Improver.

fig:benchmark-guided-outcomes

Caption: Benchmark-Guided results. Gray bars show the initial system; colored labels show score changes from that system.

Why It Matters: Displays benchmark-guided outcomes for AIME 2026, Terminal-Bench 2.1, and SimpleQA, including cases where guidance does not improve the score.

fig:checkpoint-progression

Caption: Knowledge Self: (a) SimpleQA avg@3 across checkpoints; (b) composition and quality audit of the 48 training examples.

Why It Matters: Connects successive system checkpoints with the reported training-data construction and audit.

🔮 Conclusion

The authors conclude that RSI-Env supports persistent exploration of model training and harness modification, with accepted systems able to continue research using inherited state. The experiments show capability gains in open-ended and capability-directed settings, including a Knowledge Self sequence with successive SimpleQA improvement. The findings remain uneven: target gains can coincide with declines elsewhere, and the two fixed external Improvers do not consistently outperform Self.

The paper presents the environment as infrastructure for studying recursive improvement and collaboration between autonomous systems and human guidance. Its main-body evidence supports feasibility in the reported runs; it does not establish broad or durable capability improvement across settings. The authors identify guidance on research goals, experimental design, and budget allocation as possible areas for future work.

🛠️ Future Research Improvements

The authors suggest investigating whether targeted human guidance on research goals, experimental design, and budget allocation helps agents make more effective improvements while preserving broader capabilities. This follows directly from the observed coexistence of gains and regressions, and from differences in how Self and the fixed Teachers used the shared time allocation. A useful follow-up would compare such guidance within the same environment while keeping the initial system, resource pool, and independent evaluation consistent.

Further work could extend the reported trajectory evidence. The Knowledge Self case shows two successive increases across H0/M0, H1/M0, and H1/M1, but the paper does not establish long-run stability or whether later generations retain those gains. Longer sequences could track capability changes at each accepted version and examine whether training updates and harness changes continue to complement each other. Replications from other starting models and harnesses would test how far the reported behavior generalizes; these are proposed research questions, not results reported by the paper.

🏭 Potential Industry Use Scenarios

The paper describes an experimental environment rather than a validated deployment product, so industry applications should be treated as plausible uses of its design, not demonstrated outcomes. A team developing tool-using agents could adapt the separation between a sandbox and a controller to let an agent propose prompt, tool-policy, or runtime changes while retaining experiment records and isolating the accepted system. Independent evaluation and validity checks could help teams distinguish a runnable update from one that actually improves task performance.

The task settings also suggest a way to stage internal agent research: begin with a named benchmark and permitted feedback, then study broader capability goals, and eventually leave problem selection open. Such workflows could be relevant to research engineering, model adaptation, or maintenance of domain-specific agents, where version lineage and evidence from failed experiments matter. The reported cross-benchmark regressions mean that a practical deployment would need to assess a range of capabilities and monitor changes after each update. The paper does not report production use, operational cost, or deployment reliability, so those questions require separate evaluation.

💬 Critical Analysis

A strength of RSI-Env is that it treats recursive improvement as a system-level process: model and harness changes share an exploration space, research state persists, and accepted successors can conduct later rounds. The independent Host Controller also creates a useful distinction between whether an update is executable and whether it changes measured capability. The three settings provide a structured comparison of how goal specificity and benchmark feedback affect the Improver's responsibilities.

The empirical results are mixed in ways that argue for multi-benchmark reporting. Open-General Self improves GPQA Diamond and AIME 2026 but loses ground on Terminal-Bench 2.1 and SimpleQA; Knowledge Self raises SimpleQA while GPQA Diamond declines. Teacher comparisons also depend on task, and the reported time use is not matched: Self uses substantially more of the 30-hour allocation than either Teacher in Open-General and Capability-Directed runs. Only Reasoning and Knowledge Self accept model updates in those runs, which further complicates interpreting differences as an effect of Improver identity alone.

The Knowledge case is useful evidence that a successor can continue work and produce another measured gain, but it is a single reported progression and does not settle whether gains persist over longer chains or transfer to other starting systems. The supplied main-body evidence also does not provide a complete account of the implementation artifacts needed for independent reproduction. Builders should therefore treat the results as a demonstration of the environment's studied process and examine update lineage, resource use, evaluation access, and non-target capability changes when attempting a replication.