HarnessFlow: MCTS-Guided Harness Optimization for LLM Agents

📌 Highlights

HarnessFlow searches over complete executable harnesses with a fixed model and task environment. A candidate may jointly change context construction, state management, control flow, or tool use, so the optimization target is the system governing agent execution rather than model weights or a single prompt. A tree retains each evaluated harness and its evidence while a coding agent proposes refinements.

In the main comparison, the authors report a 69.48% average test score over six benchmarks and the highest mean score in each benchmark column. This is 10.89 percentage points above the Vanilla starting harness and 3.95 points above Meta-Harness. Those comparisons use five runs and the same backbone, tool access, and per-task limits.

The approach’s practical promise comes with conditions: it uses repeated validation evaluations to guide search, the paper says the formulation requires an automatic grader, and search compute adds a one-time cost. The paper also notes that its tree structure prevents a candidate from directly combining mechanisms discovered on separate branches.

• Search object: complete executable harness codebases.
• Search policy: score-based exploitation mixed with preference for less-expanded nodes.
• Main reported result: 69.48% average test score across six benchmarks after 15 refinement attempts per benchmark.
• Key caveat: validation-guided search requires grading signals and consumes evaluation compute.

🎯 Introduction

The paper frames an agent harness as the executable system that governs model access to information, tool invocation, state maintenance, and execution coordination. It argues that these mechanisms can substantially affect agent behavior, creating an opportunity to improve performance without changing model weights. Yet harness design remains expertise-intensive and largely manual, making effective designs difficult to adapt and reuse.

Prior automated approaches described in the paper include configuration or bounded-edit refinement, full-harness code refinement, and systems that organize durable updates or populations of harnesses. The shared obstacle is that the design space is open-ended while evaluating a candidate requires costly end-to-end agent execution. The paper therefore asks how to search complete executable harnesses systematically under a limited candidate-evaluation budget.

HarnessFlow’s objective is to organize that search: represent a harness as a modifiable codebase, retain candidates and their evaluation histories, and choose which candidate to refine using an MCTS-based policy. The coding agent decides the code changes, while the policy allocates effort among candidate paths. This distinction is central to the paper’s contribution and to interpreting its results.

🔬 Methodology

For a fixed model and environment, the paper defines harness performance as expected task reward and seeks a harness with the highest validation score. Candidate evaluation averages repeated executions; execution traces supplement the scalar score by helping diagnose failures. Search starts from an initial harness, spends a fixed number of refinement attempts, and selects the best observed validation candidate for evaluation on a disjoint test set.

HarnessFlow stores each evaluated candidate as a node containing its implementation, validation score, and execution records. Edges encode parent-to-child refinements. At each attempt, it considers up to K highest-scoring nodes for parent selection. The exploitation distribution is a softmax over their scores, with temperature τ. The exploration distribution weights nodes by the inverse power of one plus their direct-child count, controlled by α. A mixture parameter ε sets the balance, and the sampled parent is refined by a coding agent.

The coding agent can use an experience archive of previous code, edits, feedback, task-level records, and relationships. It can also inspect reusable mechanisms in a Harness Component Library, covering context, state, control flow, and tool interfacing. The library is an optional reference: the paper says adopted code and local dependencies are copied into the candidate. Evaluation and archiving complete the cycle; every candidate remains available, including candidates whose scores do not improve.

In the reported default setup, the shortlist size is K=5, the candidate budget is N=15, and ε=0.35. The fixed backbone is GPT-5.4-mini at medium reasoning effort, and each method is run five times on the relevant split. The method thus couples an explicit search policy with open-ended code refinement rather than prescribing a fixed set of harness edits.

📊 Experiments

The evaluation covers six benchmarks: TAU2-bench and AppWorld for agentic tasks; SWE-bench Verified and BigCodeBench-Hard for code; and WebShop and BrowseComp-Plus for web interaction. The paper uses task success rate for TAU2-bench and AppWorld, task score for WebShop, resolution rate for SWE-bench Verified, Pass@1 for BigCodeBench-Hard, and answer accuracy for BrowseComp-Plus. Each benchmark has a fixed-seed validation/test split, and the test set is not exposed during search. Methods use GPT-5.4-mini at medium reasoning effort in the main setup, five runs, and 15 candidate attempts.

The main comparison reports HarnessFlow’s average at 69.48%, above Vanilla at 58.59% and Meta-Harness at 65.53%. HarnessFlow’s reported means are highest in every displayed benchmark column. The paper highlights gains over Vanilla of 21.70 points on WebShop and 17.85 on TAU2-bench, and gains over Meta-Harness of 7.21 on TAU2-bench and 6.60 on SWE-bench Verified. Figure 1 and the table give the comparison; the five-run standard deviations are preserved in the table.

The validation search curve reaches a mean best-so-far score of 71.5% after three candidate evaluations per benchmark and 74.2% by round 15. On four of six benchmarks, the final best is first reached at round 10 or later. The SWE-bench Verified lineage figure shows cumulative validation score rising from 67.60% to 79.20% along selected parent-child refinements; these are cumulative designs, not isolated edit effects.

Ablations show that removing the component library lowers the three-benchmark average from 65.03 to 63.39, while starting from OpenHands yields 62.61. On the separate selection ablation, ε=0.35 averages 79.33 across TAU2, AppWorld, and SWE-V, compared with 73.93 at ε=0 and 73.65 at ε=1. The stronger-executor comparison uses GPT-5.4 and reports an 83.58 average for HarnessFlow on TAU2, AppWorld, and WebShop; this is an additional setting, not the six-benchmark main result.

fig:main-a

Caption: Performance comparison across benchmarks.

Why It Matters: The paper’s result-focused comparison visualizes performance across the six benchmark settings; the detailed reported values are reproduced in the main comparison table.

fig:framework

Caption: Overview of HarnessFlow. Selection, expansion, validation, and experience backpropagation organize full-harness search. A coding agent uses reusable components and archived code, scores, and traces to refine selected harnesses.

Why It Matters: It shows how parent selection, code refinement, evaluation, and archived experience fit into one search cycle.

fig:curves

Caption: Search efficiency on the validation set.

Why It Matters: It provides the validation-set view of best-so-far performance as candidate evaluations accumulate.

fig:evolution-swe

Caption: Harness evolution on SWE-bench Verified. Code-level refinements along the selected lineage raise the validation score from $67.60\%$ to $79.20\%$. Parent--child labels show actual search edges; scores describe cumulative harness designs, not isolated effects of individual edits.

Why It Matters: It illustrates a concrete retained lineage and makes clear that the reported score changes belong to cumulative harness designs, not isolated causal estimates for each edit.

🔮 Conclusion

The authors conclude that MCTS-guided parent selection paired with coding-agent refinement can improve complete harnesses under a limited evaluation budget. On the six-benchmark comparison, HarnessFlow reports average test scores 10.89 percentage points above Vanilla and 3.95 points above Meta-Harness, alongside lower average execution cost than each hand-engineered harness in the main setup.

They qualify the result: search is guided by validation scores and therefore depends on automatic grading, and evaluations consume compute that constrains practical search budgets. The tree also imposes a single-parent lineage. A graph that allows combining mechanisms from separate branches is identified as a natural future extension.

🛠️ Future Research Improvements

The paper explicitly proposes replacing the refinement tree with a graph so a candidate can combine mechanisms discovered on distinct branches. This would address a structural restriction of the current lineage, though the supplied main-body evidence does not report such a system or its evaluation. A useful follow-up would compare graph-based combination against the existing single-parent search under the same candidate-evaluation budget and grading protocol.

The authors also identify two practical boundaries that motivate further study: reliance on validation scores from an automatic grader and the evaluation compute required by search. Research could investigate how robust candidate selection is when grading signals are incomplete or noisy, and how search budgets trade off against downstream deployment cost. Those are proposed questions, not results established by this paper.

The empirical study uses six benchmarks and a 15-attempt search budget in its main setup. Further evaluations across additional tasks, executor settings, and search budgets would clarify how transferable the reported improvements are. The paper’s validation curve shows later best candidates on four benchmarks, so budget sensitivity deserves explicit measurement before assuming that a short search is sufficient.

🏭 Potential Industry Use Scenarios

A plausible application is improving an internal tool-using agent whose tasks have repeatable, automatically scored outcomes. A team could hold its model, environment, and resource limits fixed, search over a versioned harness, and evaluate the selected candidate on held-out tasks before deployment. The benchmarks in this paper span agentic task execution, coding, and web interaction, so the method is relevant to builders working in those broad settings; the paper does not establish production deployment in any of them.

The archive and tree could also support engineering workflows where failed and successful runs provide actionable traces. Teams could use retained code and execution records to inspect why a design changed and to revisit promising candidates, while the component library offers reusable implementation references for context, state, control flow, and tool interfaces. This is a practical inference from the described mechanism, not a measured operational outcome.

Whether this is economical depends on the workload and search cost. The reported comparisons show lower per-task execution costs than hand-engineered baselines in the stated settings, but those figures exclude one-time harness-search costs. A deployment decision should therefore account for the number of tasks over which search cost can be amortized, the reliability of the grader, and the cost of validating changes safely.

💬 Critical Analysis

A strength is the alignment between the method and the engineering object being optimized: harness mechanisms interact, and the paper searches complete codebases rather than treating each design choice as independent. The retained tree and experience archive preserve failed as well as successful candidates, while the selection ablation gives evidence that a mixture of measured performance and expansion history is useful in the three displayed benchmarks. The main table also uses a shared backbone, tool access, and task limits, supporting a more direct comparison within that setup.

The results do not show that the search policy alone causes all gains. The coding agent, archive, component library, initial harness, and search policy work together; the reported ablations examine some, but not every, interaction. A validation lineage’s score increase is cumulative, and the figure caption explicitly cautions against reading individual edits as isolated effects. The average score also aggregates benchmarks with different task types and official metrics, so it should not be treated as a single universal measure of agent quality.

Reproducibility and deployment interpretation need care. The main evidence is limited to six benchmarks, five runs, and a 15-attempt budget, with a separate three-benchmark GPT-5.4 comparison. Execution-cost comparisons exclude one-time search costs, and validation-guided selection presumes an automatic grader. The paper says implementation and reproduction scripts will be released, but the supplied evidence contains no code URL. Builders should verify the runtime, grading behavior, total optimization spend, and held-out performance in their own setting.