D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models

📌 Highlights

D&A addresses confirmation bias in zero-shot visual-language reward estimation by separating the description of a robot trajectory from the judgment of its task progress. The Describer records observed behavior without the task instruction; the Assessor defines an observable success criterion and scores the description against it. This is intended to produce partial-progress rewards while penalizing failed runs, rather than treating motion or an assumed successful task as evidence of progress.

The authors evaluate D&A on D&A-Bench, RBM-EVAL-OOD, and RoboRewardBenchmark, and report real-world online and offline reinforcement-learning experiments plus feedback-driven policy refinement. On D&A-Bench, Gemini 3 Flash D&A scores Kendall τα 0.67, compared with 0.56 for Robometer and 0.43 for RoboReward; the paper reports GPT-6 Astra D&A* at 0.790 in its plotted results. These findings are specific to the paper’s evaluations and setups.

The approach’s main practical constraint is dependence on promptable VLMs: inference cost and latency, context limits, API variability, and occasional semantic errors can affect reward quality. The paper also cautions that natural-language descriptions may not capture precise spatial or continuous reward requirements.

• Core idea: describe first, then assess against explicit success criteria.
• Reported scope: reward benchmarks, real-world online and offline learning, and feedback-driven policy refinement.
• Key caveat: language and VLM reliability bound the quality and precision of the resulting rewards.

🎯 Introduction

The paper studies dense reward estimation for robot learning from a video of a robot acting and a natural-language task instruction. Each timestep receives a value in [0, 1], with zero representing no progress and one representing task completion. Such signals can support policy learning and can help identify unsuccessful actions in settings where privileged state is unavailable.

The authors argue that direct zero-shot VLM prompting can exhibit confirmation bias: a model may accept an incorrect task framing and infer progress from motion or apparent task intent. They also identify limitations in prior zero-shot approaches that directly extract scalar progress, for example by ranking shuffled frames or using token logits. Post-trained reward models can improve reliability, but the paper describes them as dependent on substantial robotics datasets and repeated training for individual base models.

D&A’s objective is to use promptable pretrained VLMs for reward estimation without robotics-specific fine-tuning, while reducing success bias and preserving fine-grained progress estimates. Its central design choice is a natural-language interface between an observation-focused description stage and an instruction-aware assessment stage. The paper further explores using these outputs for online and offline RL and for corrective feedback to a code-based planner.

🔬 Methodology

For video o₁:T and instruction l, the goal is a reward sequence r₁:T. D&A factors the mapping into a Describer and an Assessor. The Describer narrates the observed trajectory without access to l, reducing the chance that the task expectation shapes its account. The Assessor receives l and the description, constructs an explicit natural-language criterion for what completed success looks like, and scores each described timestep against that criterion. The core mapping is shown below.

Withholding the instruction can make the description omit a target object or other task-relevant detail. The authors address this with an entity extractor that lists scene entities from the instruction and video, followed by a grounded describer that uses those entities while remaining unaware of which ones are task-relevant. The paper presents criterion generation and scoring as steps within a single assessor call. It also describes augmenting the Assessor with failure analysis and corrective instructions, yielding language feedback as well as scalar rewards.

For efficiency, the implementation linearly downsamples the video to H frames, with H = 8 given as an example, then linearly interpolates scores back to the original sequence length. In online RL, the resulting score serves as a dense reward. For offline learning, the scores are used as a potential function to derive action-chunk advantages; the paper reports using these for dataset filtering or policy conditioning. This design makes textual fidelity, temporal consistency, and scoring calibration operational dependencies of the method.

📊 Experiments

The authors evaluate D&A against zero-shot baselines (Naive VLM, GVL, TOPReward) and post-trained models (SOLE-R1, RoboReward, Robometer). The three reward benchmarks are D&A-Bench, RBM-EVAL-OOD, and RoboRewardBenchmark. D&A-Bench contains 54 trajectories, 27 successes and 27 failures, across 14 YAM tasks, with 10 VLM-generated instruction paraphrases per task. Kendall τα is reported for D&A-Bench and RBM-EVAL-OOD, where higher is better; it measures alignment of final-step reward predictions with the ordering of optimal, suboptimal, and failed trajectories. RoboRewardBenchmark uses discrete end-of-episode progress labels and MAE, where lower is better.

For Gemini 3 Flash, the paper reports D&A-Bench Kendall τα of 0.672 in Figure 5; its prose rounds this result to 0.67. The corresponding plotted results are 0.567 for Robometer, 0.437 for RoboReward, 0.294 for SOLE-R1, 0.246 for TOPReward, and 0.162 for GVL. On RBM-EVAL-OOD, plotted values are 0.576 for D&A, 0.648 for Robometer, 0.536 for RoboReward-4B, and 0.359 for SOLE-R1. On RoboRewardBenchmark, MAE is 0.70 for D&A, 0.75 for Robometer, 0.85 for RoboReward-4B, and 1.08 for SOLE-R1. GPT-6 Astra D&A* is plotted at 0.790 on D&A-Bench, 0.714 on RBM-EVAL-OOD, and MAE 0.63 on RoboRewardBenchmark.

Real-world tests include DROID kitchen online RL for putting fruit in a bowl and bimanual YAM offline tasks involving a book and a box, plus action-chunk filtering for extracting and placing a specified book. The authors report online success increasing from an initial 10% to 100% in approximately 90 minutes; TOPReward reaches 5% in the described comparison. For the book-to-box offline task, D&A improves over behavior cloning by 80%. For chunk filtering, reported mean completion times are 18.8 ± 1.8 seconds with D&A and 28.5 ± 9.2 seconds for the baseline fine-tuned policy. In LIBERO goal-suite Code-as-Policies refinement, success is reported as 11.4% without history, 40.6% with plan history, and 51.4% with D&A feedback.

The ablation table uses Gemini 3.7 Flash on RBM-EVAL-OOD. Success-criterion prompting alone scores 0.561 versus 0.486 for the baseline. Decoupling without grounding scores 0.343; adding grounding in that configuration raises it to 0.546. The full configuration scores 0.626. These results support the authors’ account that the success criterion and entity grounding matter, while also showing that decoupling alone can hurt when descriptions omit relevant objects.

fig:method

Caption: Overview of . Given a video $o_1:T$ and instruction $l$, decomposes reward prediction $R(o_1:T, l) = A (l,\, D(o_1:T) )$ into a Describer $D$ and an Assessor $A$. $D$ narrates the trajectory blind to $l$ to avoid confirmation bias, producing a temporally consistent description $d_1:T$; $A$ then scores each timestep of $d_1:T$ against an explicit success criterion for $l$, yielding dense per-frame rewards $r_1:T [0,1]$ as well as rich language feedback.

Why It Matters: Shows the central pipeline, the separation of description and assessment, and the path from trajectory text to rewards and feedback.

fig:setup_dsrl

Caption: Real-world RL Results. improves real-world online (left) and offline reinforcement learning (middle, right) while other zero-shot and post-trained models struggle. Left: in the online setting, fails to distinguish between success and failure, predicting high reward for both picking up the ice cream cone and the banana. Middle, right:Behavior cloning on suboptimal data hurts performance, while offline RL with leads to a higher success rate and faster completion time. and lack generalization and assign similar reward to optimal and suboptimal action chunks.

Why It Matters: Illustrates the real-world online and offline learning settings and the reward discrimination issues described for comparison methods.

fig:qual_enter_success

Caption: Success: the robot presses enter.

Why It Matters: Pairs with the paper’s qualitative comparison of success and a visually similar failure where completion depends on a subtle on-screen change.

fig:scaling

Caption: scales with model capability across four open- and closed-source model families. scales much worse than , while and show an inverse correlation between performance and model capability.

Why It Matters: Presents the reported relationship between base-model capability, benchmark performance, and token cost.

🔮 Conclusion

The paper presents D&A as a prompt-based way to turn robot videos into dense progress rewards and corrective language feedback. Its experiments report improved reward-model behavior across three benchmarks and show uses in online and offline learning and iterative policy refinement. The authors’ central explanation is that an observation-focused description followed by assessment against explicit success criteria reduces the success bias seen in direct prompting.

The authors also identify important constraints: frontier-model inference can be costly and slow, APIs can be stochastic and difficult to control, and semantic hallucinations can undermine descriptions. Natural language may not capture fine geometric or continuous reward requirements. Thus, the reported evidence supports further evaluation of the method in deployment settings, rather than assuming that prompt-based scoring is reliable for every robotic task.

🛠️ Future Research Improvements

The limitations identified by the authors motivate testing D&A where natural-language descriptions are least precise: tasks involving exact distances, tight spatial tolerances, or continuous control objectives. A useful next evaluation would compare language-based progress estimates with task-specific geometric or state measurements, where those measurements are available, and report how disagreement affects policy learning. The supplied main-body evidence does not establish a solution to this precision problem.

The authors also note latency, token cost, context-window constraints, API stochasticity, limited control, and semantic hallucinations. Follow-up work could quantify these factors across repeated runs and model families, then test whether structured outputs, consistency checks, or lower-cost models preserve reward quality. These are suggested engineering and research directions, not improvements demonstrated by the reported experiments.

The paper reports real-world results for particular manipulation tasks and embodiments. Further work should test broader task distributions, longer horizons, additional embodiments, and failure types, and examine whether extracted entities consistently preserve the details required for scoring. Evaluations should separate reward-model agreement from downstream policy success, since a useful correlation metric alone does not establish safe or reliable deployment.

🏭 Potential Industry Use Scenarios

A plausible use is offline curation for robot-learning datasets that contain successful, suboptimal, and failed demonstrations but lack reward labels. D&A could supply progress scores for filtering or conditioning policy training, as the paper demonstrates on bimanual YAM tasks. A deployment team would need to verify scores against task-specific review data, especially where the wrong object or an incomplete placement can appear superficially successful.

Another possible use is reward generation for online policy improvement, where a model evaluates observed task progress and supplies a dense signal to an RL pipeline. The authors report a DROID kitchen task in which D&A rewards supported improved performance. Such use depends on inference speed, API availability, and the cost of scoring videos at the required frequency; the paper explicitly identifies latency and token costs as limitations.

D&A’s language feedback could also assist iterative robot programming or policy refinement by describing a failure and proposing a correction. The paper evaluates this pattern with Code-as-Policies on LIBERO, where the reported success rate improves over using plan history alone. That result is evidence for a research scenario, not proof of reliability in production workflows. Before operational use, teams would need to check feedback accuracy, keep appropriate human oversight for consequential actions, and test the system on their own tasks and failure modes.

💬 Critical Analysis

A strength of D&A is that it exposes an intermediate description and success criterion, making a reward judgment more inspectable than a scalar obtained directly from video and instruction. Its ablation results provide evidence that design choices interact: the full grounded configuration scores 0.626 on RBM-EVAL-OOD, while decoupling without grounding scores 0.343. This is a useful warning against implementing only the high-level two-stage idea while omitting the entity-grounding component.

The reported experiments are broad in task type but not sufficient to establish universal generalization. Benchmark scores, real-world outcomes, and LIBERO feedback results use different measures and setups; they should not be treated as interchangeable evidence. The comparison also depends on the selected base VLMs and configurations, and several figure captions in the supplied extraction are incomplete. The numerical values included here are those recoverable from the supplied main-body text or table.

A further open question is whether a language description remains faithful enough for reward use under occlusion, ambiguous object identity, or subtle state changes. The paper’s own limitations acknowledge semantic hallucinations and difficulty representing exact geometry. A production evaluation should therefore inspect descriptions and criteria, test repeated-run variability, and measure reward calibration on failed as well as successful trajectories before using scores to update a policy. Downstream gains reported in selected tasks are promising evidence, but do not alone establish robustness or safety across deployment conditions.