ReContact: Reconstruct and Retarget Human Contacts from Video for Contact-Rich Humanoid Manipulation

📌 Highlights

ReContact addresses contact transitions: sequences in which an object moves between hand and body support so the hands can reposition or become free. It encodes the intended support sequence in an object-centric plan that specifies contact pairs, anchors, pressing directions, and sticking or sliding modes. The plan guides object reconstruction, whole-body retargeting, and residual policy learning, so the desired interaction is represented consistently across these stages.

In simulation, the central thigh-supported regrasp result is 83.5 ± 5.2% success at the prescribed two-palm bottom hold, while the strongest adapted baseline reaches 0.16%. This is distinct from completion alone: a controller can finish the motion while holding the box in a side squeeze. A policy trained on a family of 247 references achieves 97.8% success within the generated tube-placement workspace, versus 20.4% for a source-reference policy with local start randomization.

The authors demonstrate learned policies on a Unitree G1 for seven contact-rich manipulation tasks and two locomotion-manipulation tasks. The paper's quantitative comparisons are simulation results; hardware execution is evidence of feasibility, not a reported cross-task success-rate comparison.

• Core idea: encode the intended contact transition explicitly and reuse that representation for reconstruction, retargeting, and control.
• Strongest result: 83.5 ± 5.2% T2 two-palm bottom-hold success versus 0.16% for the best adapted baseline.
• Key caveat: physical demonstrations are reported without an aggregate hardware success metric.

🎯 Introduction

Humanoid manipulation often requires an object to pass between the hands and body surfaces. A box may rest on a raised thigh while the palms slide underneath; a tube may be tucked under an arm to free a hand. The paper calls a change in the contacts supporting an object a contact transition. Transferring such a transition from human video is difficult because body occlusion degrades object-pose estimates, the robot's kinematics and dimensions differ from a person's, and a policy can track the object trajectory while making different contacts.

Prior approaches cover pieces of this problem: some preserve contact locations during retargeting, others supervise binary link contact or compare contact directions, and some learn transitions among predefined holds. The paper argues that these representations do not jointly carry contact locations, pressing directions, and support-change order through reconstruction, retargeting, and control. A side squeeze and a bottom hold can yield similar palm locations while requiring different force directions, making trajectory matching insufficient evidence of the intended skill.

ReContact's objective is to use an object-centric contact plan throughout the pipeline. Exocentric video supplies body and object motion and an editable plan; egocentric video supplies a recovered hand–object segment. Contact priors are intended to stabilize object reconstruction under occlusion, while whole-body fitting and a residual controller are intended to preserve and execute planned contacts. The paper evaluates both transfer of contact sequences and extension of one reference to new object placements.

🔬 Methodology

The contact plan has phase intervals, active contact pairs, a carrier, and a stable subinterval for training assistance. Each pair binds a robot link and a point and normal on that link to an object-frame anchor and outward normal; its mode is sticking or sliding, with a path for a sliding anchor. Contact keyframes mark support changes, while posture keyframes and foot-support intervals provide additional motion constraints. For exocentric clips, GPT-6 Astra drafts the plan and an operator can review it. For egocentric clips, the recovered hand–object segment forms the plan, with the hand as carrier.

Object-pose reconstruction combines visible landmarks, temporal smoothness, object-size information, and plan-derived priors. Support constrains an object resting on a scene surface; a carrier prior maintains a relative pose to a hand, thigh, or forearm; slide constraints relate anchors across a phase. These priors bridge intervals where the body occludes the object. Reconstruction and robot fitting alternate through human review, but each fitting round estimates the object track first and holds it fixed while fitting the robot.

Whole-body retargeting optimizes pelvis, waist, limbs, and available finger joints. Contact point residuals bring the robot link point to the object anchor, and normal residuals oppose the robot and object normals while leaving rotation around the normal available for posture adaptation. The objective combines contact, posture, stance, smoothness, and collision terms. Dense references are checked for contact alignment, orientation, penetration, joint limits, speed, and stance. The resulting reference contains joint positions and velocities, object poses, and the same contact plan used by control.

At run time, a residual policy corrects a frozen motion tracker's body targets and reference hand coordinates; commands are clipped, filtered, and rate-limited. PPO trains the policy with proprioception, reference and phase information, contact masks, and object pose. For each active pair, reward combines distance to the anchor, a matched-contact force-direction term, and persistence. A training-only virtual-object wrench is withdrawn by a gated curriculum, and evaluations use no assistance. For placement variation, the method re-solves the plan's placement-dependent segment, connects it from a shared start, and retains references that pass a kinematic and static-balance audit.

Important equations define the plan, contact residuals, retargeting objective, reward, and pelvis station. The reward's direction component is gated by a matched physical contact near the planned anchor, so a force elsewhere on the object does not receive the same credit. The family procedure preserves object-relative contact anchors as placements change; for size changes, the paper describes re-anchoring face identities and normalized in-face coordinates.

📊 Experiments

The simulation study covers T1 under-arm box carry, T2 thigh-supported regrasp, T2-R reverse regrasp, T3 tissue-pack under-arm transfer, T4 shuttlecock-tube under-arm transfer, T5 a tube-transfer placement family, and T6a spray-bottle and T6b banana grasps. T1–T3 use exocentric video and T6 uses egocentric video. T5 consists of 247 references generated from T4. The G1 is simulated in mjlab; all compared policies train for 3000 updates of 4096 × 24 transitions. Shared-reference comparisons use three adapted baselines. The paper reports success rate, completion, object error, anchor error, adherence, normal-direction error, and mechanical work; arrows in the table headers mark metric direction.

For T2, three policies per method use seeds 42–44 and three batches of 1000 evaluation episodes per policy. ReContact reaches 89.0 ± 8.6% completion and 83.52 ± 5.20% success under the stricter two-palm bottom-hold predicate. The best baseline's success is 0.16 ± 0.24% (CWS adapter); baselines complete 24.3–65.7%. ReContact's left and right palm bottom-contact fractions are 80.8 ± 1.8% and 81.2 ± 7.6%, respectively. The authors interpret this as evidence that contact-point and direction rewards distinguish the intended hold from a side squeeze. Across T1, T3, T4, T6a, and T6b, Table II(b) uses one frozen policy per row and 3000 episodes; ReContact achieves 100.0% SR on T1, T3, and T4, and 99.9% on T6a and T6b. In T4, its anchor error is 1.03 ± 0.11 cm and adherence 83.4 ± 0.7%, while CWS has lower normal error, 8.8 ± 1.5 degrees versus ReContact's 11.8 ± 1.2 degrees.

For T5, the source-reference policy with 2.5 cm start randomization achieves 20.4% success across the workspace; the family policy achieves 97.8%. The reported family-policy success across distance bands is 99.3%, 98.0%, 96.6%, and 98.6%; the Ext. column reports 19.5%. The paper also evaluates T2 across eight payloads, from 0.1 to 2.0 kg, although training randomization spans 0.074–0.135 kg around a 0.1 kg nominal mass. ReContact's completion remains 81–89%, and its two-palm success remains 78.9–88.2%; competing controllers reach at most 0.2% two-palm success. For reconstruction and retargeting, a shared-input comparison reports contact kept at 98% for ReContact, versus 77%, 71%, and 34% for UMR, OmniRetarget, and HOI-Retarget. The authors also demonstrate learned skills on a physical Unitree G1 for seven manipulation and two locomotion-manipulation tasks; the supplied evidence gives no aggregate hardware success rate.

fig:framework

Caption: Overview of ReContact. (a) An exocentric video supplies body and object motion; Fig.~fig:egocentric shows the egocentric route. (b) GPT-6 Astra reviews sparse keyframes against the video, and contact-constrained whole-body optimization turns them into humanoid--object references. (c) The object-centric contact plan, here for the thigh-supported regrasp, lists, per phase, the paired robot--object surfaces with their contact points, contact directions and sticking or sliding modes; adding or removing a pair gives a contact transition. Keyframes of the run shown: contact keyframes where support changes ($ $) and motion keyframes inside phases ($ $). An objective that does not specify the press...

Why It Matters: Shows how the contact plan links video reconstruction, retargeting, residual control, and generation of references for new placements.

fig:t2_support

Caption: Only ReContact moves the T2 box from thigh support to a two-palm bottom hold. Vertical contact force (normal plus friction) exerted on the box by the right thigh and each palm, divided by the box weight $mg$ (dashed line), in $48$ episodes per controller. Lines are medians and bands interquartile ranges over the episodes alive at each reference time ($0.1$~s moving median within each episode); the axis is logarithmic beyond $ mg$. Shading marks the palm slide and the terminal bottom hold. Panel titles give two-palm successes.

Why It Matters: Visualizes the intended force-support sequence and shows why motion completion alone does not establish a successful regrasp.

fig:t3_sampling

Caption: Workspace coverage from one reference (T5 tube-transfer family). Top: success of the task-family policy, smoothed with a $1.5$~cm Gaussian kernel, inside the generated workspace (outline) and in the $6$~cm band outside it; white points mark the $247$ references and the star the source reference's tube position. Bottom: per-position outcomes of ReContact policies of the same configuration trained on the source reference (local, $ 2.5$~cm; workspace-wide; no start randomization) and on the family.

Why It Matters: Compares placement coverage from a single reference with coverage after training on the generated family.

fig:composition

Caption: Hardware deployment of learned contact skills on a Unitree G1. (a) Walk-then-grasp using locomotion and the bottle-family policy. (b) Table-to-floor box transfer combining forward and reverse thigh-supported regrasps with loaded walking.

Why It Matters: Illustrates physical execution and composition of contact skills with locomotion on the G1.

🔮 Conclusion

The paper presents a pipeline in which an object-centric contact plan connects video-based reconstruction, robot retargeting, and residual control. Its main simulation result is that the plan's contact locations and force directions help produce the intended thigh-to-palm support transition, where trajectory completion by itself would permit a side squeeze. Generating and training on placement-specific references also broadens tube-transfer success across the workspace. Physical G1 demonstrations show that learned contact skills can be executed and composed with walking, while the quantitative evidence remains primarily simulation-based.

🛠️ Future Research Improvements

A useful next step is to measure the full human-in-the-loop cost of contact-plan creation: how often reviewers change proposed phases, anchors, directions, and modes, and how those corrections affect downstream success. Since the plan drives reconstruction, reference fitting, and rewards, experiments that perturb each plan field could establish which annotations need precision and which tolerate uncertainty.

The paper's physical demonstrations motivate larger hardware evaluations with repeated trials, explicit task predicates, and reported success and failure breakdowns. The simulation study could also be extended to more object geometries, support sequences, and placements beyond the generated region, while keeping contact fidelity and task completion as separate metrics. Robustness tests with uncertain object dimensions, visual pose errors, and unmodeled contact dynamics would clarify which deployment assumptions are most consequential. These are proposed evaluation directions, not results reported by the paper.

🏭 Potential Industry Use Scenarios

A plausible application is humanoid handling of bulky goods whose dimensions make a hand-only grasp awkward. A robot could pick a package, rest it on a thigh or forearm, and reposition its hands before carrying or placing it. ReContact's plan explicitly represents the support change and pressing directions, features that matter when a visually plausible trajectory could still leave the item unstable. The paper's box-transfer and under-arm tasks provide relevant simulated examples, and its physical G1 demonstration combines forward and reverse thigh-supported regrasps with loaded walking from table to floor.

A second scenario is a robot learning a repeated transfer at multiple tabletop positions from a demonstrated task family. In the reported tube task, training over 247 generated references yielded 97.8% success inside the workspace, which suggests a route to handling placement variation without collecting a separate video for every location. This evidence is task-specific and in simulation. Before warehouse or service deployment, engineers would need to establish performance under perception errors, varied object loads and surfaces, safety constraints around body contact, and recovery from missed or unstable contacts. The paper does not report an industrial deployment or operational throughput study.

💬 Critical Analysis

The paper's strongest methodological point is alignment across stages: one contact representation guides object-pose fitting, robot motion fitting, and policy rewards. The T2 results support the claim that object tracking and completion are insufficient measures of a regrasp; the baselines often complete motion without the required two-palm bottom hold. The T5 comparison also suggests that re-solving the placement-dependent segment can improve workspace coverage over local start randomization. Results on T4 and T6 add evidence that contact fidelity can improve on tasks where success rates are already high.

The evaluation is not uniformly favorable on every metric. On T4, CWS reports lower normal error than ReContact, and the T1 carry has low anchor adherence even though completion is high; the authors attribute that measure to contact occurring away from the planned anchors. These details reinforce that task success, object tracking, contact location, force direction, and effort capture different properties. The reported hardware runs establish execution examples, but the evidence provided does not quantify reliability over repeated physical trials.

Reproduction and deployment also depend on a reviewed contact plan, object geometry and pose, a pretrained tracker, retargeting optimization, simulation training, and task-specific policies. The paper provides detailed components and evaluation settings, but the supplied evidence does not establish sensitivity to annotation errors or broad transfer beyond the tested tasks. A careful reproduction should validate contact predicates and all metric aggregation rules, and should inspect whether generated references remain feasible at placements near workspace boundaries.