TriPhysBench: Evaluating Physical Intelligence for Embodied Agents

📌 Highlights

TriPhysBench targets a practical coordination problem: an agent must recover physical observations from video or retrieve relevant specifications, then use them in quantitative reasoning. Its 410 questions span 200 videos and two complementary subtasks. Across six evaluated models, the strongest full-benchmark overall answer score is 0.5038 for Gemini-3.6-Flash. TriPhys trains complete tool-integrated solutions with supervised fine-tuning and preference optimization; on 110 questions from held-out videos, all three Qwen variants improve over their backbones, and TriPhys-27B reaches 0.5818 overall answer accuracy. The benchmark is weighted toward quantitative estimation, while the reported training comparison is limited to three Qwen configurations and a single partition.

• Benchmark: 228 MPR and 182 KPR questions, from 90 and 110 videos respectively.
• Best full-benchmark overall ACC: Gemini-3.6-Flash, 0.5038.
• TriPhys-27B held-out overall ACC: 0.5818, versus 0.4941 for Qwen3.6-27B.
• The authors report variable identification and conclusion formation as recurring solution weaknesses.

🎯 Introduction

The paper frames physical intelligence as coordination among physical perception, physical knowledge retrieval, and physical reasoning. An agent may need to estimate motion from a video, find a relevant physical property, and combine both using a suitable relation. The authors use Popper’s Three Worlds as a conceptual organization: perception corresponds to the physical world, retrieval to objective knowledge, and reasoning to mental activity. The task is to keep quantities, assumptions, and the observed physical situation consistent.

Prior benchmarks cover physical plausibility, event prediction, physical understanding, quantitative video reasoning, scientific multimodal questions, external knowledge access, and general-purpose tool use. The paper argues that these lines leave insufficiently characterized how reliably an agent can obtain the needed physical quantities, choose knowledge applicable to observed conditions, and use both in quantitative problems. TriPhysBench is intended to evaluate that combination in real-world videos. Its MPR and KPR tasks differ in whether selected observations or physical specifications must be acquired, while sharing answer forms and quantitative demands.

🔬 Methodology

The benchmark contains 410 questions from 200 videos. MPR uses 90 EgoExo4D recordings and 228 questions; pose-based measurements and tracked trajectories support physical quantities, with selected observations withheld so solvers must recover them from video. KPR uses 110 Physics-aware-videos recordings and 182 questions; observations are stated, while selected external specifications such as mass or dimensions must be retrieved and matched to the described object and conditions. Both subtasks include point estimation, interval estimation, physical judgment, and scenario comparison. Physical event graphs link event participants and processes to observations, properties, and physical relations; question generation withholds quantities needed to recover a target, and question review combines model assessment with rule-based checks.

TriPhys trains on complete tool-integrated trajectories. A trajectory records the model’s decisions, actions such as retrieval or computation, and tool observations. In Stage I, verified trajectories are used for SFT; the answer block may be rewritten into stepwise form, subject to solution-score checks. The SFT loss supervises model-generated reasoning, tool calls, and answer tokens, while tool returns remain in context and are masked out of the prediction target. In Stage II, complete candidates sampled from the SFT policy are divided into accepted and rejected groups using answer and solution criteria. DPO compares these trajectories against a frozen SFT reference, with weights to balance MPR and KPR. At inference, one policy makes tool calls and incorporates their returns into subsequent reasoning.

The reported scoring formulas make the evaluation targets explicit. SCS weights observation identification and formula applicability at 0.25 each, derivation at 0.30, and conclusion consistency at 0.20. OGS combines ACC and SCS with weights 0.70 and 0.30. The main training equations specify masked trajectory imitation and preference comparison; they are useful implementation references, although the extraction does not supply enough detail to reconstruct all prompts, thresholds, or data processing from these equations alone.

📊 Experiments

The paper first evaluates six vision-language models on the full 410-question benchmark: GPT-5.5, Gemini-3.6-Flash, InteractiveOmni-4B, Qwen3.5-4B, Qwen3.5-9B, and Qwen3.6-27B. All receive the same instructions and a native tool loop with web search and Python execution for at most six turns. Each returns a solution and final answer, which are scored separately. ACC is higher-is-better answer correctness with partial credit; SCS is higher-is-better solution correctness; OGS is their weighted aggregate. Gemini-3.6-Flash has the highest full-benchmark ACC at 0.5038 and OGS at 0.5585. GPT-5.5 has the highest MPR ACC, 0.4316, while Gemini leads KPR ACC at 0.6004. The subtask results differ, so an aggregate alone conceals distinct performance patterns.

The training evaluation uses a video-level split: 300 training questions from 147 videos and 110 test questions from 53 videos. TriPhys-4B, TriPhys-9B, and TriPhys-27B are compared with their matched Qwen backbones and two closed-source reference models on those same test questions. All three trained models improve overall ACC, SCS, and OGS over their backbones. TriPhys-27B reaches 0.5818 ACC, 0.7156 SCS, and 0.6219 OGS; its backbone has 0.4941, 0.6784, and 0.5494. The authors report a 17.7% relative ACC gain. Gemini-3.6-Flash scores 0.5910 ACC on this test set, so it remains above TriPhys-27B on answer accuracy.

The stage ablation compares base, SFT, and SFT plus DPO checkpoints. For 4B and 9B, SFT provides the larger initial improvement; DPO then adds 0.033 and 0.012 overall ACC respectively. For 27B, SFT leaves ACC nearly unchanged, while DPO raises it by 0.087. The paper also reports declining OGS from Easy to Hard questions and frequent variable-identification failures. A separate cross-judge analysis reports strong SCS agreement between DeepSeek-V4-Flash and GPT-5.5, but those ratings remain model-judged. The source comparison table for other benchmarks has no recoverable numeric values in the supplied evidence, so no cross-benchmark numeric comparison is included.

fig:overview

Caption: Overview of TriPhysBench and TriPhys. Blue, yellow, and pink denote physical perception (W1), knowledge retrieval (W3), and reasoning (W2), respectively. Scenes and workflow are schematic.

Why It Matters: Shows the benchmark’s three coordinated capabilities and the TriPhys workflow at a glance.

fig:construction

Caption: TriPhysBench construction: (a) video processing for MPR (EgoExo4D) and KPR (Physics-aware-videos); (b) physical event graphs; (c) question generation and review. Solid arrows indicate data flow; dashed arrows indicate external inputs. Scenes are schematic.

Why It Matters: Clarifies how source videos, event representations, and review steps produce benchmark questions.

fig:method

Caption: TriPhys training: (a) full-trajectory SFT on verified trajectories, with the answer block rewritten into stepwise form; (b) full-trajectory DPO on chosen and rejected trajectories sampled from the SFT policy. In the loss mask, 1 marks supervised model-generated content and 0 marks gray tool returns kept as context; the shared prefix of each pair is masked. The orange dashed line marks the fallback chosen trajectory from Stage I, and the gray dashed line marks the frozen SFT reference; snowflakes denote frozen weights and flames denote trainable LoRA parameters.

Why It Matters: Makes the two training stages, trajectory masking, and preference reference concrete.

fig:failure

Caption: Failure analysis: (a) performance by construction difficulty, measured by OGS; (b) component failure rates (ratings of at most 0.5). Rates use all questions in each subtask; categories can overlap.

Why It Matters: Highlights the observed degradation with harder questions and the recurring component-level failure patterns.

fig:case

Caption: Case study of TriPhys-4B on an MPR test question. The columns show Qwen3.5-4B, the SFT model, and the TriPhys model. Each column lists the source of the rim diameter, the tool calls, excerpts of the answer chain, a compressed trajectory, and the answer with ACC and SCS. Red highlights mark errors, and faded nodes mark tool calls whose results are not used.

Why It Matters: Illustrates how retrieving a relevant physical specification and making the derivation explicit changes a solution across training stages.

🔮 Conclusion

The authors conclude that TriPhysBench provides a shared evaluation of physical perception, knowledge retrieval, and reasoning through two complementary video-question settings. The six-model comparison shows that coordinated physical problem solving remains difficult, while component and difficulty analyses point to input identification and supported conclusions as recurring challenges. TriPhys trains a policy on complete tool-integrated solutions and improves answer and solution scores over three Qwen backbones on held-out videos. Its largest reported model reaches 0.5818 ACC and 0.6219 OGS. These findings support trajectory-based training as a useful approach within this benchmark, while broader models and repeated evaluations are needed to establish how general the gains are.

🛠️ Future Research Improvements

The paper’s discussion points to several concrete next evaluations. First, repeat training and evaluation runs to quantify the stability of the improvements, and extend comparisons across model families, capacities, and tool-use strategies. Second, construct matched questions for the same physical problem while varying which observations or specifications are supplied; this would better isolate the dependencies among perception, retrieval, and reasoning than comparisons between the current subtasks, which also differ in sources and composition. Third, analyze tool trajectories and validate intermediate quantities with human review alongside model-based ratings. Such checks could reveal whether training improves acquisition, relation selection, numerical derivation, or conclusion formation, and where errors still propagate. These are research directions suggested by the authors; results for them are not reported in the supplied main-body evidence.

🏭 Potential Industry Use Scenarios

The paper evaluates benchmark questions and model-generated tool trajectories; it does not report a deployed product, field trial, or operational performance study. Still, its task design suggests plausible areas for builders to investigate. A maintenance assistant could combine video observations of a repair with retrieved component specifications and calculations, while a sports or training assistant could estimate motion-related quantities from recorded activity. A laboratory or education assistant might connect visible physical processes with material properties and quantitative explanations. In each case, an agent would need to verify that retrieved values match the observed object and conditions, preserve units, and expose enough of its derivation for review. These are potential application scenarios inferred from the paper’s methods and video domains, not demonstrated industry outcomes; deployment reliability, safety, latency, and data coverage remain unestablished.

💬 Critical Analysis

The benchmark’s main strength is that it makes acquisition demands explicit: MPR asks the model to recover selected measurements, whereas KPR supplies observations but requires external specifications. Video-level partitioning also reduces direct question leakage across train and test. Reporting both answer and solution scores, plus a stage ablation, provides more diagnostic evidence than final-answer accuracy alone. The case study further shows a concrete path by which a retrieved constant can replace an incorrect recalled value and a fuller derivation can improve the solution assessment.

The evidence still has limits. The subtasks vary in source videos, physical phenomena, and answer-form distribution, complicating causal interpretation of MPR-versus-KPR score differences. The benchmark’s emphasis on quantitative estimation and its large work-and-energy component may underrepresent other forms of physical competence. The training evidence covers three related Qwen configurations and one 110-question test partition; it does not establish reproducibility across seeds or architectures. SCS agreement between two model judges is useful evidence of consistency, but not a substitute for human verification of every intermediate claim. Finally, the reported results demonstrate improved benchmark scores, not reliable physical understanding or safe performance in real embodied deployments.