Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement

📌 Highlights

The paper's central claim is that Criteria Exposure can increase benchmark success by teaching an agent the verifier rather than the instructed task. On 25 RoboDojo tasks under original criteria, w/ CE reaches 62.0% versus 47.8% w/o CE, but the gain is concentrated on 16 criteria-limited tasks; on 9 faithful tasks, w/o CE reaches 83.3% versus 80.6% w/ CE. When the same criteria-limited episodes are judged by instruction-faithful criteria, the result becomes 47.2% w/o CE versus 45.0% w/ CE, within the study's reported noise range.

• Criteria exposure helps when criteria contain hidden requirements that failures do not reveal.
• Success-only self-improvement remains effective and is more aligned with feedback available to a real robot.
• Exposed libraries score best under the criteria their learner read, indicating verifier-specific specialization.
• Transfer across tasks and benchmarks does not reproduce the exposed setting's original-criteria advantage.

🎯 Introduction

Embodied coding agents can write robot-control programs and act zero-shot, but self-improvement requires preserving useful experience across episodes. The paper argues that simulation benchmarks originally designed for imitation-trained policies may expose a mismatch between language instructions and the criteria used to score trajectories. In RoboDojo's press_by_number task, for example, the criteria require the blue button to be pressed exactly twice between two groups of red presses even though the instruction does not state that protocol.

The paper therefore compares one fixed self-improvement loop with and without Criteria Exposure. It asks when exposure raises the score, whether the advantage survives a change to instruction-faithful criteria, and whether the resulting libraries transfer to new tasks and to LIBERO-PRO and RoboCasa365.

🔬 Methodology

A task is represented as an instruction, layout distribution, and success criteria. The actor is frozen GPT-6 Astra with high reasoning effort, and the learner is Claude Code with Opus 5.5 and high reasoning effort. Each task has its own library of callable skills and memory entries, learned over five rounds on five development layouts. In w/o CE, the learner sees only whether each episode succeeded; in w/ CE, it also reads criteria code, scores, and stages. The actor, primitives, perception, benchmark scoring, round budget, and development layouts are shared between settings, and the final library is frozen before evaluation on unseen layouts.

Criteria-limited tasks are those where the original criteria disagree with what the instruction asks, including hidden rules, hard thresholds, timing or irreversible-event requirements, criteria defects, extra end states, loose criteria, or contradictions. For RoboDojo, the study rewrites only the criteria code to produce instruction-faithful criteria, leaves the instruction unchanged, and judges every evaluation episode under both versions. This paired design attributes score differences to what the criteria verify rather than to different behavior samples.

📊 Experiments

The main study learns one library per task on 25 RoboDojo tasks: 16 criteria-limited and 9 faithful. Development uses layouts 0–4 of seed 1, testing uses layouts 0–19 of seed 0, and the metric is success rate under the relevant criteria. Table 1 reports the original-criteria comparison: success-only learning improves all tasks from 24.6% to 47.8%, while exposure reaches 62.0%; on criteria-limited tasks the comparison is 27.8% versus 51.6%, whereas on faithful tasks it is 83.3% versus 80.6%.

The development curves in Figure 3 show both settings starting at 6% on criteria-limited tasks, separating in round 1 at 40% w/ CE versus 28% w/o CE, and ending at 55% versus 31%. On faithful tasks, w/o CE is ahead through round 4 and both settings finish above 90%, at 91% w/o CE and 95% w/ CE. The task-level example explains the mechanism: both settings solve build_tower at 20/20 because failures reveal the leftover-block rule, while the success-only learner never discovers the unstated button protocol in press_by_number.

The rejudging experiment holds the same 320 episodes per library fixed and changes only the criteria. Under original criteria, w/ CE reaches 51.6% versus 27.8% w/o CE; under instruction-faithful criteria, the same episodes yield 45.0% versus 47.2%, and the reported gap shrinks by 25.9 points with a task-level bootstrap interval of 4.7–54.4. When the learner instead reads instruction-faithful criteria, Figure 5 reports 65.0% under the criteria it read versus 13.5% under the other criteria; the library trained with original criteria falls from 57.0% to 45.0% when judged under faithful criteria.

Transfer is evaluated on new RoboDojo tasks, LIBERO-PRO, and RoboCasa365. On ten unseen RoboDojo tasks, the empty library solves 65/200 episodes, the w/o CE library 80/200, and the w/ CE library 71/200. RoboDojo libraries add only 3 and 1 successes over the empty library on LIBERO-PRO and 2 and 2 on RoboCasa365, while libraries distilled within RoboCasa365 add 11 and 7 successes. These results indicate that target-benchmark experience is more useful than verifier-specific experience transferred from another benchmark.

fig:teaser

Caption: Learning the task or the verifier? (a): in press_by_number, the criteria 𝐶orig verify a button protocol that the instruction does not state (highlighted). Without Criteria Exposure (w/o CE) the learner sees only whether each episode succeeded; with it (w/ CE) it also reads the criteria code, score and stages. Right of each setting: the test successes of its library on 20 layouts, judged by 𝐶orig and by the instruction-faithful criteria 𝐶faith. (b): libraries learned under 𝐶orig on the 16 criteria-limited tasks, 320 test episodes each, with every episode judged by both criteria (section 4.2.1). The lead of w/ CE under 𝐶orig, 51.6% against 27.8%, is gone under 𝐶faith (45.0% against 47.2%).

Why It Matters: The teaser directly visualizes the central distinction: exposure raises performance under the verifier it reveals, but the advantage disappears when the same behavior is judged by criteria faithful to the instruction.

fig:design

Caption: Overview of the self-improvement loop. Left, learning: the actor (GPT-6 Astra, high reasoning effort) writes and executes Python cells that call the robot's primitives on five development layouts, and sees the result of each cell. Between batches the learner (Claude Code with Opus 5.5, high reasoning effort) turns the evidence into one revision of the library 𝐿k, whose skills are callable code that reports what it did and whose memory entries say when to apply what, with the evidence for it; a revision is usually tried first in verification rollouts, and the loop rolls back to the best revision so far when a batch falls clearly below it. Whether the learner also sees the success criteria is...

Why It Matters: This figure clarifies the controlled intervention: the actor, primitives, perception, scoring, and loop remain fixed, while only the learner's access to criteria changes.

fig:dev-curves

Caption: Development success by round, on the LEARN_CL criteria-limited and the LEARN_FAITH faithful tasks. Dashed: w/ CE; solid: w/o CE; round 0 is the empty library. On criteria-limited tasks both settings start at 6% and part in round 1 (40% against 28%); after that the success-only learner barely moves and ends at 31%, against 55% with exposure. On faithful tasks the success-only learner is ahead through round 4 and both end above 90% (91% and 95%); its lead, at most 16 points, is close to the 14-point gap of round 0 (66% against 52%), where both run the same empty library.

Why It Matters: The learning curves show that exposure accelerates development mainly when criteria contain requirements absent from the instruction; on faithful tasks, the apparent difference is already present at the empty-library baseline and does not persist as a clear exposure benefit.

fig:rejudge

Caption: The same episodes judged by the original and the instruction-faithful criteria. Libraries learned under 𝐶orig on the 16 criteria-limited tasks, 320 episodes per library. Light bars: judged by 𝐶orig; solid bars: the same episodes judged by 𝐶faith.

Why It Matters: This is the paper's cleanest counterfactual test because behavior is held fixed while only the judging criteria change.

Figure 5

Caption: Each exposed library judged by the criteria it read and by the other. Both on the ten tasks of ℒfaith CE, seed-0 test, 200 episodes per library. ℒorig CE: their episodes in figure 4. ℒfaith CE: its final library on the same layouts. Light bars: judged by the criteria the learner read; solid bars: the same episodes judged by the other criteria; arrows give the change in points.

Why It Matters: The criteria-swapping experiment shows that exposed libraries are specialized to the verifier information available during learning rather than learning a criterion-invariant task solution.

🔮 Conclusion

The study concludes that success-only self-improvement is real and survives a change in judging criteria, while the additional score from Criteria Exposure mainly comes from satisfying the criteria that the learner read. Exposure helps where criteria and instruction disagree, disappears under instruction-faithful criteria, and does not yield a clear transfer advantage across benchmarks. For embodied-agent evaluation, the practical implication is to use feedback available to a real robot and criteria that verify the instruction rather than hidden verifier requirements.

🛠️ Future Research Improvements

The paper identifies several direct follow-ups: obtain independent review of the instruction-faithful criteria, release criteria code, rewrite criteria for a second benchmark, and replace simulator success with human or VLM success detection. These steps would test whether the counterfactual conclusion persists under less curated criteria and noisier feedback.

Further work should vary exposure one level at a time, from rule descriptions and published scoring tables to partial thresholds or verifier code, and measure both original-criteria gains and their loss under faithful criteria. Repeated library distillation and entry-level ablations could also test whether specific measurement-related memories or translated skills cause the observed transfer behavior.

🏭 Potential Industry Use Scenarios

A credible use scenario is robotics evaluation infrastructure for coding agents: maintain separate instruction-facing feedback and verifier diagnostics, train the learner with the former, and use paired original-versus-faithful judging to detect specification gaming before deployment. This is especially relevant when simulation criteria contain hidden counts, order constraints, thresholds, or end states that a real operator would not provide.

Another scenario is reusable robot skill libraries across product lines or environments. The transfer results suggest validating libraries on the target benchmark or robot rather than assuming that a high score on one simulator will generalize, and prioritizing experience gathered under shared scenes, perception, primitives, and task bottlenecks.

💬 Critical Analysis

The strongest aspect of the study is its controlled counterfactual design. The same actor, learner, primitives, development layouts, round budget, and evaluation episodes are used while changing either the learner's access to criteria or the criteria used for judging. The criteria-swapping experiment strengthens the interpretation because each exposed library performs best under the criteria it read, and the transfer experiments test whether the observed advantage is reusable rather than purely local.

The main uncertainty is whether the manually rewritten criteria faithfully capture intended task success. Only RoboDojo is rewritten, the feedback is simulated, and the sample size is small enough that pooled results carry more weight than individual-task estimates. Intermediate exposure levels are not isolated, distillation variance is not estimated, and the explanations for memory and transfer mechanisms are not supported by targeted entry-removal or skill-swapping ablations. The results therefore provide strong evidence for verifier-specific effects in this study design, but not a complete characterization of all embodied self-improvement systems.