📌 Highlights
HarnessMM changes when a harness invokes its reasoning model. It preserves resolved actions as executable commitments, activates them from public receipts, and returns to slow reasoning whenever semantic uncertainty remains.
The strongest integrated result is on HumanEval: Codex maintains 90.82% success while Tokens/task falls from 66.72k to 47.02k. DeepSeek Harness rises from 71.43% to 85.71% success while Tokens/task falls from 96.28k to 11.62k.
The mechanism attribution is unusually direct: the paper records model calls and certified bypasses, then separately ablates persistent EDG reuse and retained callbacks on MBPP-sanitized.
The main caveat is that the method is not a universal replacement for reasoning. It is beneficial only when the next action is already semantically resolved and the complete accounting shows that authorship and control overhead are lower than the avoided reasoning cost.
- Core idea: reuse resolved actions instead of re-deliberating after every tool receipt.
- Mechanism: EDG + GDR + CCS with receipt-bound auditing.
- Evidence: HumanEval, MBPP-sanitized, six companion benchmarks, and MiniWoB++ RSI.
- Primary risk: task-specific guards and overhead can limit transfer.
🎯 Introduction
Agent harnesses connect model reasoning to tool execution through repeated reasoning, action, and observation cycles. The paper identifies a narrower inefficiency than prompt replay or reasoning-depth reduction: after an earlier call has fixed the next action and its literal arguments, a later tool receipt may only determine whether that action is ready, not what the action should be.
HarnessMM frames this distinction as semantic uncertainty versus execution uncertainty. An EDG stores the resolved semantic commitment, while public guards and predecessor dependencies determine whether observed feedback activates it. This complements programmatic execution and skill reuse by making the stored object episode-specific and sufficiently concrete to replace a future reasoning callback.
The paper also defines the Substitution Paradox: adding a fast selector or execution layer does not save computation unless it removes enough downstream reasoning to offset authorship, routing, context, fallback, and recovery costs. The objective is therefore policy-level cost reduction subject to benchmark-quality constraints, evaluated on the same task populations and with the original tools and scorers.
equation 2
Meaning: The policy objective minimizes computation cost while constraining capability regret on each declared benchmark endpoint.
🔬 Methodology
The actor parameters are fixed while HarnessMM changes the harness route from public tool state to the next native action. At a slow decision, the actor emits the current action and a bounded EDG. Each decision node stores a native action, complete literal arguments, a predecessor, and a public guard. The graph is acyclic, and missing or rejecting receipts prevent a node from being admitted.
GDR retains the graph across tool transitions. The controller updates predecessor completion and guard truth after each receipt, then forms an eligible set. A typed selector chooses a candidate ID or abstains without modifying the stored arguments; an empty eligible set, abstention, unsupported evidence, or a new semantic decision invokes the original actor.
CCS performs the actual substitution at the registered reasoning boundary. The stored action is sent through the original tool interface, and a certificate binds graph identity, decision identity, predecessor receipt, exact arguments, tool receipt, and bypassed callback identity. This makes omission of the slow callback auditable rather than inferred from a short output or a predicted response.
Recursive self-improvement operates around this runtime mechanism: training feedback refines companion harness configurations, disjoint validation selects a retained configuration, and the selected configuration is frozen before test evaluation. Complete-task accounting includes graph authorship, selector activity, remaining slow calls, fallback, recovery, finalization, and failed paid calls.
equation 4
Meaning: Each EDG decision stores the native action, literal arguments, predecessor dependency, and public execution guard.
equation 5
Meaning: This determines whether a stored decision may execute from the current public state without new semantic reasoning.
equation 6
Meaning: The certificate binds the reused action, its causal predecessor, exact executed arguments, resulting tool receipt, and omitted callback for audit.
📊 Experiments
The evaluation uses the same tasks, backbone, original tools, task limits, and scorer within each comparison. Success is measured at the benchmark endpoint, while token cost includes input and output for all paid calls, including failures. The central coding populations are HumanEval with 98 test tasks and MBPP-sanitized with 257 test tasks. Companion configuration studies cover ALFWorld, SearchQA, LiveMath, AIME, Terminal-Bench 2, and GAIA2; the harness-evolution study uses MiniWoB++ with 32 tasks per arm.
The main mechanism table reports the HumanEval and MBPP-sanitized pairs. On HumanEval, HarnessMM preserves Codex success and reduces actor tokens, while DeepSeek Harness both improves success and sharply reduces tokens. On MBPP-sanitized, the Codex pair reduces tokens with a small success decrease, while the DeepSeek pair improves success and reduces tokens substantially. The original harnesses record zero certified bypasses, whereas HarnessMM records executed substitutions.
The execution ablation on MBPP-sanitized isolates persistent reuse and callback removal. Full HarnessMM reaches 93.77% success at 13.17 Tokens/task (k), compared with 93.00% at 18.03 without GDR; retaining callbacks reaches 93.39% at 16.37. Disabling EDG execution reaches 95.33% at 16.19, showing that lower cost can trade against a small number of solved tasks.
The broader studies show that the method's effect is workload-dependent. On ALFWorld, the selected HarnessMM instruction condition is reported as the highest-success and lowest-token instruction policy. On SearchQA, HarnessMM reaches 76.76% exact match at 0.99 Tokens/task (k). On AIME, its 87.14% accuracy is close to the compared instruction policies but its 5.90 Tokens/task (k) is above the lowest-cost alternatives. On GAIA2, HarnessMM reaches 43.96% success at 639.64 Tokens/task (k), improving over No Skill but remaining below SkillOpt in success. On MiniWoB++, eight RSI iterations raise HarnessMM success from 34.38% to 90.63% while reducing Tokens/task from 3.26 to 1.06.
- Primary quality metric: benchmark success, pass rate, exact match, or accuracy depending on the benchmark; higher is better.
- Primary efficiency metric: complete actor Tokens/task (k), including input and output and, where specified, authorship and control costs; lower is better.
- Mechanism evidence: model-call counts and certified bypasses verify whether slow callbacks were actually removed.
- Ablation evidence: W/o GDR tests single-use graphs, W/o CCS retains callbacks, and W/o EDG disables graph execution while retaining authorship.
tab:humaneval-mechanism
Caption: Coding-harness mechanism attribution on HumanEval and MBPP-sanitized with the shared backbone. Calls/task counts model calls; Bypasses counts executed substitutions that skip a model call. Actor costs include input and output tokens.
| Harness | Pass (\%) | Tokens/task (k) | Calls/task | Bypasses |
|---|---|---|---|---|
| Codex | 90.82\% | 66.72 | 5.35 | 0 |
| +HarnessMM | 90.82\% | 47.02 | 5.09 | 18 |
| DeepSeek | 71.43\% | 96.28 | 9.73 | 0 |
| +HarnessMM | 85.71\% | 11.62 | 4.23 | 147 |
| Codex | 91.83\% | 129.55 | 6.93 | 0 |
| +HarnessMM | 90.66\% | 49.22 | 5.40 | 158 |
| DeepSeek | 90.27\% | 53.98 | 6.35 | 0 |
| +HarnessMM | 91.83\% | 10.61 | 4.23 | 472 |
equation 8
Meaning: Complete episode cost sums input and output tokens over every paid call in the episode.
equation 9
Meaning: This separates development cost amortized over deployment tasks from expected complete-task execution cost.
fig:design
Caption: HarnessMM overview. The upper row shows RSI updating a harness from $EDG_i$ to $EDG_i+1$ using task results and token consumption; observations supply execution evidence, including visual signals when supported. Below, GDR introduces guarded reuse of resolved decisions, and CCS replaces a redundant reasoning subgraph with certified native execution. Solid arrows show regular execution, dashed arrows show guarded reuse routes, and dashed boxes mark edited subgraphs. The graphs are schematic.
Why It Matters: This is the clearest architecture view: it separates offline harness refinement from online EDG reuse and callback substitution.
fig:industrial-harness
Caption: Success and actor token costs for Codex, DeepSeek Harness and their HarnessMM variants. (a,b) use DeepSeek-flash; (e,f) use MiMo-v2.6-pro. Each harness pair shares its backbone, public task prompt, tools and scorer. Costs are reported as Tokens/task (k).
Why It Matters: It summarizes the integrated coding-harness comparisons across backbone and harness pairs.
fig:alfworld-ablation
Caption: and execution-mechanism ablations. Top: six instruction conditions on the same 2,133 ALFWorld tasks with Qwen3.5-9B; bars average seeds 9401--9403 and whiskers show their range. All conditions use history length two and a 50-step limit; neutral text has the same token count as the task instructions. Bottom: four GDR/CCS execution conditions on the same 257 MBPP-sanitized tasks with the shared backbone and one attempt per task. W/o GDR discards the EDG after one execution; W/o CCS retains the slow callback. W/o EDG execution disables graph use while retaining authorship. Tokens/task (k) include complete actor input and output, graph authorship, control, fallback and unsuccessful executions....
Why It Matters: The bottom panels isolate the two core mechanisms: retaining EDGs and removing the slow callback.
fig:vigrace-evolution
Caption: Eight training-guided harness RSI iterations on MiniWoB++. Each comparison uses the same 32 tasks per arm, Qwen3.5-9B, full screenshots, native rewards, 15 actions and 180 seconds per episode. Calls/task counts physical model requests over all tasks. Tokens/task (k) includes complete episode authorship, control, fallback and unsuccessful calls; optimization and selection costs are separate.
Why It Matters: It shows how recursive harness refinement and persistent reuse evolve together across repeated training-guided iterations.
🔮 Conclusion
The paper presents guarded decision reuse as a way to reduce repeated reasoning in existing tool-using harnesses without changing actor weights, native tools, or task interfaces. EDGs preserve resolved episode-specific actions, GDR activates them from public receipts, and CCS makes the resulting callback omission observable through certificates.
The strongest evidence combines integrated harness results with mechanism-level ablations. HumanEval preserves Codex success while reducing actor-token cost, DeepSeek Harness improves both success and cost on the reported comparisons, and MBPP-sanitized separates persistent reuse from callback removal. The results support the paper's claim that changing the execution policy can be more important than shortening an individual reasoning call.
🛠️ Future Research Improvements
A natural next step is to broaden the public-state adapter and guard language so that the method can handle richer tool receipts, partial failures, state mutations, and uncertainty without weakening the requirement that stored arguments remain closed. The paper's current contract makes this boundary explicit but task-specific.
Future evaluations should report wall-clock latency, cache effects, energy, and monetary cost alongside actor tokens. They should also vary development-set size and measure how often authorship, selection, and recovery overhead erase the savings from bypassed callbacks.
The RSI loop would benefit from larger and more diverse validation populations, stronger uncertainty reporting, and tests in which graph reuse can create stale or unsafe actions. Comparing learned selectors with deterministic eligibility policies could clarify when the selector itself is worth its cost.
🏭 Potential Industry Use Scenarios
Coding agents are the clearest deployment target because file writes, compilation, testing, and other tool actions often have literal arguments resolved before later receipts arrive. A harness could retain these actions and execute them after public checks such as file existence or successful compilation, while routing changed requirements back to the model.
Enterprise workflow agents could use the same pattern for structured tool sequences such as ticket updates, document transformations, data validation, and API calls, provided that public receipts expose reliable guards and the stored arguments can be audited. The certificate design is especially relevant where an operator needs to verify which model callback was bypassed and which native action actually ran.
Browser and GUI agents are another plausible scenario because the MiniWoB++ study uses screenshots, coordinate compilation, guards, fallback, and native rewards. However, visual state drift and ambiguous observations make guard design and stale-action recovery central engineering risks.
Deployment should begin with shadow-mode accounting and complete-task cost measurement. The method is most credible when teams can show that quality stays within an agreed bound and that the total cost of authoring, selection, fallback, and recovery is lower than repeated model reasoning.
💬 Critical Analysis
The paper's strongest design choice is its explicit separation of semantic commitment from execution readiness. EDGs do not ask feedback to rewrite an action's meaning; they allow feedback to activate an action that slow reasoning has already fixed. This makes the proposed savings auditable and enables the bypass count to be tied to a registered callback boundary.
The experimental evidence is broad, but not every result isolates the same mechanism. The integrated Codex and DeepSeek Harness comparisons combine literal-bank reuse, context management, and finalization settings, whereas the MBPP-sanitized ablation more directly tests GDR and CCS. The paper appropriately treats the latter as the mechanism attribution and the broader benchmark studies as companion configuration evidence.
The main open question is robustness under stale or misleading public state. A true guard can establish that a declared condition holds without proving that the stored action is still globally correct. The system therefore depends on conservative guards, reliable receipts, and a safe fallback path to the original actor.
For builders, the method is promising when tool trajectories contain repeated, causally ordered actions and when native callback boundaries are explicit. It is less compelling for tasks dominated by fresh semantic decisions, visual ambiguity, or expensive graph authorship. Reproduction should therefore report complete-task costs and quality jointly rather than treating bypass counts or token reductions as sufficient evidence.