PSGAgent: Enhancing MLLMs' Dynamic Visual Perception for Panoptic Scene Graph Generation via Agentic Reasoning

📌 Highlights

PSGAgent addresses two perception bottlenecks in panoptic scene graph generation: saliency bias toward dominant image regions and semantic ambiguity caused by contextual interference or fine-grained predicate confusion. Its central design is to let a multimodal language model decide when additional visual evidence is needed, then obtain that evidence through targeted tools.

The authors report 50.27 Recall@100, 42.88 mean Recall@100, and 46.58 Mean for the 3B model under SGDet, plus 73.04 Recall@100, 67.75 mean Recall@100, and 70.40 Mean under PredCls. The strongest evidence for the proposed mechanism comes from the bottleneck subsets, where PSGAgent reaches 40.06 on saliency-biased samples and 37.45 on semantically ambiguous samples.

The main caveat is evaluation scope: all reported experiments use PSG, while extensions to video scene graphs and embodied agents remain proposed future work.

• Adaptive perception is learned as part of structured relation prediction.
• A 3B PSGAgent exceeds the reported 7B R1-SGG SGDet Mean by 7.98 points.
• Hierarchical rewards and GSPO are important: the full model reaches 49.91 AP@50, 50.27 Recall, and 42.88 mRecall, while the reported GRPO variant reaches 46.86, 47.92, and 41.79.
• The model is strongest when object regions are reliable, as shown by the PredCls result.

🎯 Introduction

Panoptic Scene Graph generation converts an image into a structured graph of thing and stuff entities, panoptic masks, and pairwise predicates. The paper positions PSG as a bridge between visual perception and semantic reasoning for embodied AI, image–text retrieval, and detailed image captioning.

The authors argue that prior MLLM-based systems generally process a fixed global view. This creates saliency bias, where small or peripheral relations are overlooked, and semantic ambiguity, where long-range visual interference or similar actions such as standing on and walking on are difficult to distinguish. Query-driven visual exploration is also insufficient because PSG requires exhaustive discovery of relations rather than answering a single localized question.

PSGAgent therefore makes perception adaptive: the model reasons about uncertainty, invokes a tool, observes the transformed image, and continues until it emits a structured graph. The paper couples this interface with error-aware trajectory construction and reinforcement learning so that tool invocation is learned as a strategy rather than hard-coded as a preprocessing step.

🔬 Methodology

The agent models a graph G = (O, R), where each object contains a category and a non-overlapping binary mask, and each relation is a predicate between a subject and an object. Its two visual operators have complementary purposes: cropping increases resolution for small or boundary relations, while highlighting removes unrelated context around an interacting pair.

Error-aware data mining begins with mispredicted triplets from a vanilla MLLM. Saliency-biased cases are selected when the subject–object union occupies less than 20% of the image or is near an image boundary. Semantic-ambiguity cases use non-overlapping entities whose predicates have high confusion or belong to the low-frequency tail. Each selected sample is synthesized into a multi-turn trajectory of thoughts, actions, and observations, with trajectory length bounded by the maximum interaction count.

The cold-start SFT stage teaches basic structured tool use. The RL stage uses GSPO with group-normalized advantages and a sequence-level importance ratio. The total reward combines final accuracy and valid output structure with causal zoom and strategic highlight signals, encouraging useful, non-redundant, and difficulty-aware perception actions.

fig:framework

Caption: Overview of PSGAgent. We reformulate PSG as a multi-turn, tool-augmented decision-making process. (Left) Trajectory Construction. We perform error-aware data mining to identify challenging cases caused by saliency bias (relations occupying small regions or located near image boundaries) and semantic ambiguity (long-range contextual interference, high predicate confusion, or tail relations). These identified cases are used to synthesize error-aware Multimodal Chain-of-Thought (M-CoT) trajectories through a structured generation pipeline. (Right) Cold-start SFT and RL-based Enhancement. The agent is first initialized via SFT to learn basic tool usage, and then optimized with GSPO for strategi...

Why It Matters: The framework figure connects error mining, trajectory synthesis, SFT, GSPO, tool invocation, and the final scene-graph output.

📊 Experiments

The evaluation uses the PSG dataset, which contains 48,749 images, 133 object categories, and 56 predicate categories with panoptic segmentation masks. SGDet requires object detection, category prediction, and relation prediction from the image; PredCls supplies ground-truth object boxes and labels to isolate predicate reasoning. The paper reports Recall@100, mean Recall@100, their average Mean, AP@50, zero-shot Recall@100, and open-vocabulary relation scores under a 7:3 base–novel split.

Under SGDet, the authors report 49.91 AP@50, 50.27 Recall, 42.88 mRecall, and 46.58 Mean for PSGAgent. With predefined category prompts, PSGAgent* reaches 50.41 AP@50, 51.81 Recall, 44.19 mRecall, and 48.00 Mean. Under PredCls, the model reaches 73.04 Recall, 67.75 mRecall, and 70.40 Mean, improving the reported VCTree + CAFE Mean of 42.85 by 27.55 points.

Generalization results are 8.19 zero-shot Recall and 27.52 open-set Recall, 20.20 open-set mRecall, and 23.86 open-set Mean. On the bottleneck subsets, PSGAgent reports 40.06 Recall for saliency-biased samples, 37.45 for semantically ambiguous samples, 50.49 for head predicates, 34.85 for tail predicates, and 50.27 overall. The authors also report that 69% of M-CoT instances are classified as saliency bias and 31% as semantic ambiguity, while over 72% require four or five tool-call rounds.

Ablations indicate that coordinated tool use requires RL: the reported GSPO model reaches 49.91 AP@50, 50.27 Recall, and 42.88 mRecall, compared with 46.86, 47.92, and 41.79 for the GRPO variant. Removing the M-CoT dataset reduces the GSPO result from 49.91 AP@50, 50.27 Recall, and 42.88 mRecall to 49.27, 48.99, and 38.64. The mask-source study reports 43.99 Mean for zoom only, 45.24 with OneFormer, 48.24 with ground-truth masks, and 46.58 with the default Mask2Former.

fig:R1SGG

Caption: R1-SGG(7B)

Why It Matters: The qualitative comparison is used to illustrate missed subtle relations, predicate contradictions, denser scene graphs, and reduced hallucination.

fig:error_type

Caption: Distribution of error types

Why It Matters: The error distribution supports the construction of separate saliency-bias and semantic-ambiguity training subsets.

fig:train_trewards

Caption: Rewards Curve

Why It Matters: The training-dynamics figure shows reward growth during tool exploration followed by stabilization.

🔮 Conclusion

The paper concludes that PSG benefits from dynamic visual perception rather than a single fixed-resolution observation. PSGAgent combines two targeted operators, an error-aware M-CoT cold-start dataset, and GSPO with hierarchical rewards to produce denser and more accurate scene graphs.

The reported results support the proposed mechanism across standard SGDet, idealized PredCls, open-set, zero-shot, and targeted bottleneck evaluations. In particular, the bottleneck and ablation results indicate that the gains are associated with adaptive tool use and its training signal, not only with the presence of a larger visual-language backbone.

The conclusion identifies Video Scene Graph Generation and embodied agents as future extensions, but those scenarios are not evaluated in the supplied main-body evidence.

🛠️ Future Research Improvements

A direct next step is to evaluate whether the think–act–observe policy transfers from static PSG images to video, where tool decisions must account for temporal consistency, changing object locations, and repeated evidence across frames.

The paper would benefit from broader cross-dataset evaluation and stronger efficiency reporting. The current evidence reports up to five interaction steps, approximately five tool calls per episode after training stabilization, four A6000 GPUs, and roughly seven days of GSPO training, but it does not establish the accuracy–latency trade-off for deployment.

Future work should also test robustness to imperfect detections and segmentation in settings where masks are not available or where object identities change over time. The mask ablation is encouraging, but the default system still uses Mask2Former for the highlight operation.

Because the qualitative discussion emphasizes plausible relations absent from sparse annotations, human evaluation or alternative graph-quality measures could complement exact ground-truth matching and better assess open-world scene-graph usefulness.

🏭 Potential Industry Use Scenarios

In embodied AI, the tool policy could support robotic systems that first inspect a whole scene and then zoom or isolate specific interacting entities before planning actions. The paper motivates this application, but does not report robot experiments.

For image–text retrieval and detailed image captioning, denser relation graphs could improve fine-grained matching and descriptions involving small or peripheral objects. These are plausible downstream uses grounded in the paper's stated motivation rather than evaluated product results.

For visual assistants and inspection systems, the highlight operator offers a way to focus a multimodal model on a specific subject–object interaction while suppressing unrelated context. Practical deployment would require measuring latency, segmentation failures, and behavior under distribution shift.

💬 Critical Analysis

The strongest aspect of PSGAgent is the alignment between its error taxonomy, tool design, and reward structure. Saliency-biased samples map naturally to zooming, while semantic ambiguity maps to pairwise highlighting. The predicate-group results and the tool-specific ablations provide evidence that this alignment improves the targeted cases rather than merely increasing aggregate scores.

The comparisons should be interpreted by evaluation setting. PredCls gives the model ground-truth object regions and therefore demonstrates relational reasoning under favorable localization conditions; SGDet is the more complete task. Likewise, PSGAgent* uses predefined object and relation categories in prompts, so it should be compared separately from the standard PSGAgent setting.

The mask-source results reduce concern that the method is entirely dependent on Mask2Former: zoom only reaches 43.99 Mean, OneFormer reaches 45.24, and dropping 30% of masks retains 44.59 Mean. However, the gap between the default model and the ground-truth-mask upper bound shows that segmentation quality still affects the ceiling.

Reproducibility questions remain around the exact trajectory-generation prompts, the implementation of output-validity and hallucination penalties, the complete tool-call schema, and the compute cost of GSPO. The supplied evidence establishes the reported benchmark results and ablations, but not whether the same policy remains effective with different backbones, datasets, or interaction budgets.

Figure ure

Caption: Figure 5 illustrates the training dynamics of the agent, which consist of two phases. The early tool exploration phase shows a steady reward increase alongside rapid growth in tool calls, indicating that the agent is learning to effectively invoke tools. Subsequently, both curves stabilize, while the number of tool calls converges to approximately five per episode. This stable reward curve and calling frequency demonstrate that PSGAgent successfully masters a coordinated tool-use strategy for complex reasoning.

Why It Matters: This caption-aware crop provides the paper's interpretation of reward and tool-call stabilization during GSPO training.