CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

📌 Highlights

CaPtain demonstrates that a small open VLM can be post-trained into a standalone low-level code-as-policy robot agent without a frontier model, a VLA skill, or privileged state at test time.

The strongest reported result is 78.5% host success on the three-suite LIBERO-PRO evaluation, with RL improving the SFT model by 19.0 percentage points; on the full four-suite aggregate, the model reaches 68.6% from 49.2%.

RL gains persist across LIBERO-Plus and LIBERO-MAX, while transfer reaches 83.3% on SimplerEnv WidowX and 75.0 on CaP-Bench but remains weak on RoboDojo.

The central limitation is scope: the policy is trained on single-arm LIBERO tasks and the critic depends on privileged simulator state during training.

• 4B open VLM trained with synthetic SFT followed by multi-turn RL.
• 22 low-level perception, geometry, and guarded-motion capabilities.
• 73.8% held-out LIBERO host success after RL versus 55.4% after SFT.
• Simulation-only adaptation enables qualitative real-robot execution.

🎯 Introduction

The paper frames code-as-policy robot manipulation as a language-agent problem: the model observes images, writes programs against perception and control APIs, and reads back execution results. Prior systems either query closed frontier models or delegate manipulation to pretrained VLAs and high-level skills, leaving the model unable to improve the underlying grasp, approach, and placement behavior.

CaPtain targets the missing regime: a small open VLM that makes every manipulation decision through low-level primitives. Its objective is to combine in-harness SFT with multi-turn RL while exploiting simulator-only physical state for credit assignment. The paper argues that this makes post-training applicable to manipulation decisions that unfold over tens of requests and whose consequences may appear several turns later.

🔬 Methodology

CaPtain-Harness presents a stable policy-facing interface across simulation and hardware. The agent receives a natural-language task and API description, then executes a sequence of Python requests. Variables persist in the sandbox, tool calls return structured success or failure receipts, and the harness can attach camera images. The interface deliberately omits semantic skills such as grasp, place, and open-drawer, forcing the policy to choose geometry, approach vectors, jaw commands, and recovery actions.

The policy history is represented by the sequence in Equation 1. SFT minimizes the token-level loss in Equation 2 over synthetic trajectories generated in the harness. During SFT, earlier coordinates are masked with probability 0.2 and the vision tower remains frozen. The RL critic in Equation 3 is an ensemble of five MLPs over 36 privileged physical features, the request index, and a task embedding; it is trained with Equation 4.

Request-level credit is computed with Equation 5, combining predicted-success changes with per-request costs. Equation 6 adds group-relative outcomes to generalized advantage estimates, while Equation 7 distinguishes completion outcomes. The policy objective in Equation 8 uses double-sided ratio masking and KL regularization to the SFT model. The critic is used only during training and is removed at test time.

The paper's RL-round procedure samples 64 start states, produces four rollouts per start, updates the critic on all request states, uses up to eight sampled requests per episode for the policy gradient, and synchronizes policy weights every four rounds.

📊 Experiments

Training uses the 40 original LIBERO tasks. Evaluation covers held-out LIBERO initial states, LIBERO-PRO Spatial/Object/Goal and Long suites, seven LIBERO-Plus perturbation dimensions, LIBERO-MAX Base and Dynamic cases, and zero-shot transfer to RoboTwin 2.0, RoboDojo, SimplerEnv WidowX, and CaP-Bench. Evaluations use temperature 1, thinking enabled, at most 60 requests, 150 physical actions, and 1,080 seconds per episode; host success is the final-state predicate, while strict success additionally requires an explicit finish declaration.

On held-out LIBERO, RL raises host success from 55.4% to 73.8% and strict success from 52.1% to 67.5%. The reported gains are +19.0 points on the three-suite LIBERO-PRO average, +19.4 on all four suites, +17.6 on LIBERO-Plus, +21.2 on LIBERO-MAX Dynamic, and +18.8 on LIBERO-MAX Base. The authors report improvements across all seven LIBERO-Plus perturbation dimensions, with the largest gain for lighting at +23 points and the smallest for sensor noise at +13.

Against published LIBERO-PRO systems, the RL model reaches 78.5% on the three-suite average and 68.6% on all four suites, compared with 59.5% and 49.2% after SFT. The paper reports 83.3% average success on SimplerEnv WidowX and a CaP-Bench score of 75.0, rising from 55.0 for SFT and 5.0 for the untrained Qwen3.5-4B under the same harness. Zero-shot transfer is more uneven: 22.6% on RoboTwin 2.0 overall and 30.0% on its single-arm subset, while RoboDojo reaches 0.8% success overall.

The paper also reports a qualitative real-robot transfer after fine-tuning in a coarse XR1 twin simulation without real-world data. The results support the portability of the interface, but the benchmark evidence is strongest on single-arm LIBERO-derived tasks and weaker on cross-embodiment or bimanual tasks.

fig:method

Caption: Overview of . Top: inference, one episode in : at each request the policy writes a Python cell against a low-level API and receives text receipts and new images, for up to 60 requests. Bottom: training and test: SFT on synthetic agent trajectories, then RL in which a turns privileged physical state into per-request credit; the critic is removed at test time.

Why It Matters: This figure connects the deployment loop, synthetic SFT stage, and privileged-critic RL stage into one reproducible system design.

fig:sftrl

Caption: RL gains survive perturbation. SFT vs. RL host and strict success on held-out LIBERO and on four perturbation suites.

Why It Matters: It is the clearest visual summary of the reported post-training gains and their persistence beyond standard LIBERO.

fig:breakdown

Caption: Breakdown, -SFT vs. . (a) LIBERO strict success by suite. (b) LIBERO-Plus host success by perturbation dimension. (c) LIBERO-PRO host success by suite and perturbation type.

Why It Matters: The breakdown shows whether improvements are broad across perturbation dimensions and benchmark suites rather than concentrated in one aggregate score.

fig:real

Caption: Real-robot episodes. Two tasks with new objects and layouts; ten frames per episode, with time stamps. (b) requires reasoning about the instruction: the agent must infer that the cube with the color of the sky is the light-blue one and that the container that is not round is the rectangular tray.

Why It Matters: The qualitative episodes demonstrate that the same low-level policy interface can support object grounding and indirect language instructions on hardware.

fig:twin

Caption: The twin simulation used to adapt the policy to the XR1 arm (left) and the real scene at test time (right).

Why It Matters: This figure clarifies the simulation-only embodiment adaptation used before real-robot evaluation.

🔮 Conclusion

CaPtain supports the paper's central claim that low-level code-as-policy manipulation can benefit from the standard SFT-then-RL recipe used for language agents. The combination of a stable execution harness, synthetic interaction-format alignment, and privileged physical-state credit produces substantial gains over SFT and preserves those gains across several perturbation suites.

The results also show a clear boundary. The approach is competitive on LIBERO-derived single-arm settings and transfers qualitatively to a real robot, but unfamiliar kinematics, dual-arm coordination, and unsupported contact behaviors remain substantial obstacles.

🛠️ Future Research Improvements

The most direct extensions are the directions identified by the method's failure modes: train across multiple embodiments, add simultaneous bimanual actions, and improve reach-aware planning for unfamiliar arms. These changes would test whether the learned code policy represents reusable manipulation principles or primarily exploits the LIBERO/Panda regime.

The critic should also be made usable beyond exact simulators. A learned or externally estimated physical-state representation could enable real-world RL, but its effect on credit calibration and policy stability needs controlled evaluation. Finally, the action interface should support compliant force and contact behaviors so that tasks such as sustained wiping are expressible.

• Add bimanual and multi-embodiment training tasks.
• Replace exact simulator state with calibrated estimated state for critic inputs.
• Extend guarded motion with force-aware and compliance-oriented primitives.
• Measure performance under longer horizons and more diverse recovery behaviors.

🏭 Potential Industry Use Scenarios

CaPtain's strongest near-term use case is simulation-first deployment of adaptable manipulation agents for structured workcells where the robot can expose reliable RGB-D perception, guarded motion, and execution receipts. The persistent Python interface could support inspection, pick-and-place, sorting, and instruction-grounded manipulation while keeping the policy-facing API stable across robot backends.

The real-robot result suggests a practical workflow in which a coarse embodiment twin teaches kinematic and gripper conventions before hardware deployment. However, the reported evidence does not establish production reliability, safety certification, high-throughput latency, or robust force interaction, so industrial adoption would require extensive closed-loop safety validation and domain-specific data.

💬 Critical Analysis

The paper's strongest technical idea is the fixed-time-index physical critic: it credits changes in predicted success caused by physical state transitions while avoiding direct access to the policy's own action counters. This is well matched to manipulation, where perception and verification requests may be numerous but only some actions change the scene. The reported broad perturbation gains provide evidence that RL is improving closed-loop recovery rather than only memorizing nominal trajectories.

The comparison should nevertheless be interpreted with care. Published baselines use different model sizes, VLA skills, frontier APIs, data regimes, and evaluation protocols. CaPtain's advantage is most meaningful within the open-weight, no-VLA-skill category, while OpenRUA and Harness VLA remain higher on the reported LIBERO-PRO averages. Transfer results also reveal a substantial embodiment gap: 22.6% on RoboTwin 2.0 and 0.8% success on RoboDojo are far below the main LIBERO-family results.

Reproduction depends on details that materially affect outcomes, including the exact harness safeguards, receipt truncation and compaction behavior, synthetic-trajectory distribution, asynchronous rollout staleness, early stopping during RL, and the definition of host versus strict success. Builders should therefore reproduce the interface and evaluation protocol before attributing gains solely to the critic or to RL.

• Strength: the method exposes low-level manipulation decisions to post-training.
• Strength: gains persist across multiple perturbation families.
• Caveat: cross-embodiment and bimanual performance remains limited.
• Caveat: simulator privilege and interface coverage constrain direct hardware-RL conclusions.