Computer Vision
DispFlow-GS: Displacement Flow Supervision with Motion Disentangling for Monocular Deformable 3D Gaussian Splatting
Thai Duy Nguyen, Haitian Zhang and Addison Lin Wang
Accurate dynamic scene reconstruction is important for robotic perception, where temporally consistent representations of dynamic environments are essential. Deformable 3D Gaussian Splatting (3DGS) models dynamic scenes through deformation fields, and recent methods incorporate motion supervision by aligning rendered Gaussian flow with optical flow. However, we find that such Gaussian-flow-based supervision provides only limited improvements in motion modeling. We identify a fundamental limitation of this supervision paradigm, namely a domain gap between rendered Gaussian flow and optical flow. To address this limitation, we propose a motion supervision framework built on Displacement Flow, which splats per-Gaussian 3D displacements onto the image plane to provide direct and stable optimization signals. We further disentangle scene motion from camera motion via intermediate-view rendering, enabling more reliable motion priors and targeted constraints on deformation and geometry. We also observe a discrepancy between motion fidelity and image-based evaluation, where improved motion awareness does not necessarily translate into better rendered image quality or higher image-based metric scores. Motivated by this mismatch, we introduce Deformation-Rendering Consistency (DRC), a motion-aware metric that measures the
alignment between predicted deformation and rendering improvement. Experiments on dynamic scene
benchmarks show substantial improvements in motion localization and motion–rendering consistency, reaching up to 39% and 6%, respectively, while image-based metrics change by only about 0.1%. These results confirm the observed mismatch between motion fidelity and image-based evaluation, demonstrating the significance of DRC for motion-aware evaluation.
Read the full abstract ▾
Computer Vision
WeLike2Party! In-Context Motion Transfer for Multi-Human Image Animation
Sangeyl Lee, Seunghyun Shin, Seungho Park et al. (+2 more)
Human image animation aims to transfer motion from a driving
video to subjects in a reference image. Despite remarkable progress in
video generation, achieving high-fidelity animation of multiple interacting subjects remains a challenge. Many existing approaches rely on explicit motion representations such as 2D skeletons or parametric body meshes and struggle to preserve identity-motion binding under inter-person occlusion. To address this limitation, we propose WeLike2Party, a multi-human animation framework built on direct in-context
video conditioning without explicit pose or mesh extraction at inference. We further introduce Reference Asymmetric RoPE Conditioning to preserve fine-grained appearance details, and Identity Binding Supervision to associate each reference identity with its intended motion trajectory. To support cross-identity training, we construct MotionTwin, a large-scale synthetic
dataset comprising 14.4K cross-identity
video pairs with shared subject and camera motions, totaling 84.3 hours of photorealistic
video. We additionally present MotionTwin-Bench, a cross-identity
benchmark specifically designed to evaluate subject-level visual fidelity and identity-motion binding. Extensive experiments on MotionTwin-Bench and real-world
videos demonstrate that WeLike2Party outperforms recent state-of-the-art methods in subject-level visual fidelity, identity-motion binding, and overall perceptual quality, particularly in multi-person interactions with substantial occlusion.
Read the full abstract ▾
All authors Sangeyl Lee, Seunghyun Shin, Seungho Park, Wooseok Jeon, Hae-Gon Jeon
Computer Vision
SFE-VGGT: Source-Free VGGT Distillation for Event-Based Monocular Depth Estimation
Thai Duy Nguyen and Addison Lin Wang
Recent event-based depth estimation methods successfully transfer
geometric priors from
vision foundation models via cross-modal
distillation. However, their reliance on synchronized RGB-event pairs or depth annotations during training severely restricts practical deployment. To overcome this bottleneck, we propose SFE-VGGT, a novel source-free framework that distills the
geometric priors of VGGT to the event domain without any paired RGB observations. Our core idea is to reconstruct surrogate frames directly from the target event stream to act as a frozen
geometric teacher, entirely eliminating the need for genuine source RGB data. Crucially, as these surrogate frames inherently yield imperfect and spatially varying supervision, directly distilling from them propagates artifacts. To resolve this, we introduce a novel reliability-aware
distillation strategy. This includes Density-Aware Feature
Distillation to emphasize informative event regions, and Confidence-Weighted Depth
Distillation to dynamically regulate supervision based on relative teacher-student prediction confidence. Meanwhile, we propose a Cross-Frame Relational Consistency loss that enforces temporal
geometric stability using reliable inter-frame correspondences, bypassing the need for temporally consistent teacher's depth. Extensive experiments demonstrate that, despite source-free, our SFE-VGGT closely matches the accuracy of RGB-dependent baselines under standard conditions and significantly surpasses them in challenging nighttime scenarios. Across MVSEC nighttime sequences, SFE-VGGT reduces the average 10 m depth error by 15.3% compared with EventVGGT. Moreover, our method exhibits robust zero-shot generalization across real-world
datasets, proving that highly effective
geometric priors can be transferred to event cameras using strictly source-free supervision.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Track-and-Complete: Learning Humanoid Skills from a Single Failed Human Video
Sarmad Idrees and Jongeun Choi
Learning humanoid skills from
videos typically requires a successful human demonstration, which often demands custom data collection. Although failures have traditionally been treated only as negative examples in
robot learning, they can still reveal a usable trajectory prefix before the task fails, as well as the intended outcome. To leverage this information from a failed-attempt
video, we propose TRACC, a pipeline that imitates the useful portion of the motion trajectory and then completes the task based on the inferred task outcome. The usable motion prefix serves as prior knowledge until the failure occurs, after which the task-completion reward guides the policy to learn the intended task goal without requiring a successful task trajectory. We evaluate our method on six in-the-wild failed human tasks from the Oops!
dataset. Our experimental results demonstrate the effectiveness of the proposed approach for learning from failed attempts when no successful demonstration is available. Thus, these findings establish failed human
videos as a viable source of supervision for humanoid skill learning.
Read the full abstract ▾
Computer Vision
Seg3DParts: Segmentation-Grounded Controllable Part-Level 3D Generation
Jiantao Lin, Meixi Chen, Yingjie Xu et al. (+5 more)
Part-level 3D assets are essential for editing, reassembly, and interaction, yet recovering such structure from a single image remains challenging due to occlusion, ambiguous boundaries, and the need for coherent multi-part
reasoning. Existing approaches struggle to achieve both controllable part-level generation and coherent multi-part structure, as part identity and spatial allocation are typically inferred implicitly. We present Seg3DParts, a segmentation-grounded framework for controllable part-level 3D generation from a single image. By treating segmentation as an explicit grounding signal, our method defines part identity during generation, enabling each component to be anchored to a corresponding image region. To ensure coherent assemblies, we introduce structured cross-part interaction that allows components to exchange global context throughout the generative process. As a result, Seg3DParts directly generates well-aligned part meshes in a shared canonical space without post-hoc
alignment, supporting flexible and controllable decomposition. We further introduce PartObjectNet, a large-scale
dataset with over 200K objects and 1M annotated parts. Experiments demonstrate that Seg3DParts achieves superior geometry quality, cross-part coherence, and part-level controllability over existing methods.
Read the full abstract ▾
All authors Jiantao Lin, Meixi Chen, Yingjie Xu, Chenbo Fu, Leyi Wu, Hao Chen, Yinchuan Li, Ying-Cong Chen
Computer Vision
Representation Dynamics Reveal Semantic Saliency and Similarity for Visual Token Pruning in MLLMs
Weixuan Li, Zikun Zhou, Xinyi Zhuang et al. (+4 more)
Multimodal large
language models (MLLMs) incur high inference latency from long visual
token sequences. Existing
pruning methods commonly use
attention maps or output features to estimate
token importance or redundancy. Several recent approaches also exploit representation changes, but when and how these changes reflect foreground saliency and semantic consistency remain insufficiently understood. We analyze visual
token representation dynamics across encoder depth and uncover two findings. First, the relationship between
token update magnitudes and foreground saliency is layer-dependent: large
token updates concentrate on foreground regions in two depth intervals, separated by several sink-dominated layers at intermediate depths. Second, similarities between
token update directions better distinguish same-class from different-class
tokens than those between encoder output features. Building on these findings, we propose MSDG-Prune, a training-free method that uses update magnitudes and directions to preserve salient and diverse visual information. Specifically, we group
tokens by update-direction similarity and use query-weighted saliency derived from update magnitudes across a chosen depth window for group-wise
token pruning. Extensive experiments across four MLLMs demonstrate the effectiveness and generalizability of MSDG-Prune. On LLaVA-NeXT, it retains 91.9% of uncompressed performance on average with only 5.6% of visual
tokens, while achieving a 7.8x prefilling speedup. Code is available at https://github.com/liweixuan-hitsz/MSDG-Prune.
Read the full abstract ▾
All authors Weixuan Li, Zikun Zhou, Xinyi Zhuang, Xinyan Guo, Rui Tian, Chuyao Zhang, Lin Gao
Computer Vision
SafeVantage: Vantage-Aware Memory for Reliable Embodied Decisions
Sean Hardesty Lewis, Zuyi Guo, Benwang Chen et al. (+3 more)
Reliable embodied decisions under partial observability require informative observations and sufficient supporting evidence. However, semantic scores alone do not reveal which viewpoints justify a claim or where additional evidence should be acquired. We introduce SafeVantage, a vantage-aware semantic
memory and active acquisition framework that retains each claim's supporting views, camera poses, and estimated target location, keeping positive support distinct from search coverage. A learned candidate-observability model uses claim-grounded geometry to predict target visibility at reachable viewpoints. These predictions guide view selection through expected reduction in terminal decision loss, accounting for travel cost and geometrically distinct corroboration. A calibrated head then combines support, spatial consistency, and coverage to produce Yes, No, or Abstain decisions. We evaluate SafeVantage on a category-presence
benchmark spanning 232 unseen ProcTHOR houses and 7,424 paired episodes per method and action budget. Compared with validation-selected equal-budget baselines, SafeVantage achieves macro-F1 gains of 24.7% and 12.0% at eight and twelve actions, respectively, with lower risk and higher answer rates at both budgets and 31.7% less travel at eight actions. Equal-input HM3D experiments show lower selective risk under fixed observations, while controlled ScanNet interventions show that restoring supporting views improves downstream VLM answers. Ablations further support the contribution of candidate observability to decision quality and acquisition efficiency. Results demonstrate the value of claim-level viewpoint evidence for connecting semantic
memory, active acquisition, and reliable decision-making. Code is available at https://safevantage.github.io
Read the full abstract ▾
All authors Sean Hardesty Lewis, Zuyi Guo, Benwang Chen, Zirui Li, Hongyi Lin, Heye Huang
Computer Vision · Cross-listed from cs.CL
RAEGNet: Relation-Aware Evidence Graph Network for Harm-Aware Multimodal Fake News Detection
Wenbin Shen, Guoxuan Qin, Guangxu Yao et al. (+4 more)
Existing
multimodal fake news detection methods often introduce external information to assist detection. However, most of them rely on entity-level
retrieval and are therefore prone to introducing event-irrelevant noise. Meanwhile, existing methods mainly focus on improving overall performance and do not account for differences in the degree of harm posed by different instances of fake news. To address these limitations, we design an Event-Level Evidence
Retrieval Framework (ELERF) and propose a Relation-Aware Evidence Graph Network (RAEGNet). ELERF retrieves external evidence based on the complete event semantics of a news item. RAEGNet constructs a directed graph that incorporates news-evidence stance relations and evidence-evidence interaction relations, and introduces a conditional-harm branch to jointly model authenticity and potential harm. Experimental results demonstrate that RAEGNet outperforms multiple baseline methods across all evaluated metrics on Weibo-21, Fakeddit, and our self-constructed SSS
dataset.
Read the full abstract ▾
All authors Wenbin Shen, Guoxuan Qin, Guangxu Yao, Baodong Wang, Yuanbo Rui, Zhongjie Ba, Zhichao Lian
Computer Vision
DiffReID: Discriminative Diffusion Model for Object Re-Identification
Yingquan Wang, Pingping Zhang, Dong Wang et al. (+1 more)
As a fundamental image processing task, object Re-Identification (ReID) aims to retrieve objects across non-overlapping cameras. Recently, with the development of deep learning, significant advancements have been made in object ReID. However, most existing methods suffer from generalization due to the limited size and diversity of ReID
datasets. Meanwhile, current models tend to focus on extracting semantic patterns rather than learning identity-aware feature distributions. To address these issues, we propose a novel feature learning framework named DiffReID for object ReID. It leverages a discriminative
diffusion model to gradually learn identity-aware distributions and generate identity-invariant features. More specifically, with the Contrastive
Language-Image Pre-training (CLIP) model, we first obtain identity-aware text features by
prompt tuning. Then, we propose a
Vision-guided Noise Generator (VNG) to initialize probabilistic noises and gradually corrupt identity-aware text features. Afterwards, we take visual features as conditions and propose a Light Weight Denoiser (LWD) to denoise the corrupted text features step-by-step for identity-aware distribution learning. To obtain discriminative features, we further generate identity-invariant guided features from randomly sampling visual-guided noises. Finally, we propose a Mutual Enhancement Constraint (MEC) to facilitate mutual learning between visual features and guided features to enhance the representation robustness and discrimination. Extensive experiments on five object ReID
benchmarks demonstrate that our method shows better results than most state-of-the-art methods. The source code is available at https://github.com/AWangYQ/DiffReID.
Read the full abstract ▾
All authors Yingquan Wang, Pingping Zhang, Dong Wang, Huchuan Lu
Computer Vision
ProGuT: Label-Efficient Panoptic Segmentation for Forest Scenes
Pankaj Deoli and Karsten Berns
Panoptic segmentation in forest environments is bottlenecked not by semantic quality but by instance separation; existing unsupervised panoptic approaches produce usable stuff maps but near-zero thing quality. Depth or flow-based instance discovery methods needs sensors that are not always available. We present ProGuT (Prototype Guided Training), which produces panoptic pseudo-labels without per-image training masks, needing only unlabeled images and one-time cluster-to-class mapping. ProGuT clusters CLIP patch features, then recovers trunk instances through multiscale
geometric prior that falsifies non-trunk structures via structure-tensor. This is cheap compared to depth, flow or class-supervision methods to create pseudo labels. These are then used for downstream tasks which we evaluate against other unsupervised baselines. ProGuT achieves a Panoptic Quality (PQ) of 65.2 on Our-forest
dataset (2.6x improvement over the initial pseudo-label quality) and reaches 65.9 mIoU on Freiburg Forest, outperforming unsupervised baselines like PiCIE (45.3 IoU) and STEGO(57.6IoU). Additionally, ProGuT outperforms existing unsupervised methods for class-agnostic trunk instance
benchmark.
Read the full abstract ▾
Computer Vision
Less Supervision, Better Generalization: Weakly Supervised Fake Region Localization in Diffusion-Edited Images
Junhee Lee, Donghyeon Jeon, Taeoh Kim et al. (+2 more)
Localizing AI-edited regions is essential for interpretable forensic analysis, but remains challenging due to subtle and spatially distributed artifacts that are misaligned with semantic or object boundaries. Existing approaches rely on pixel-level supervision from controlled editing pipelines, which is difficult to scale and can introduce misleading signals: artifacts frequently extend beyond annotated regions, while out-of-mask pixels are treated as authentic. This limits models' ability to capture transferable evidence and generalize across generators and
datasets. To address these issues, we propose ReGFLoW, a Reconstruction-Guided Fake Localization framework under Weak supervision, which is the first weakly supervised approach for
diffusion-edited fake region localization. ReGFLoW requires only real/fake labels at the image level and uses
diffusion reconstruction errors as dense spatial guidance to inject them into both feature and score spaces. Furthermore, by artifact-centric multiple instance learning, ReGFLoW utilizes localized
diffusion evidence without relying on semantic-affinity or boundary-based pseudo-mask priors. Extensive experiments show competitive cross-generator localization, while ReGFLoW outperforms all evaluated fully supervised baselines when evaluation includes both partially edited and fully synthetic images and in cross-
dataset tests, without target-domain adaptation.
Read the full abstract ▾
All authors Junhee Lee, Donghyeon Jeon, Taeoh Kim, Beomyoung Kim, MyeongAh Cho
Computer Vision
S4VY: Segment Anything in Feed-Forward 4D Visual Geometry
Jingdong Zhang, Xin Li, Jan Kautz et al. (+2 more)
Accurate instance segmentation in dynamic scenes is important for downstream applications such as robotics and autonomous driving. Existing Segment Anything models operate primarily on 2D image or
video masks and preserve identity through sequential
memory, while promptable 4D instance segmentation built upon feed-forward visual geometry remains underexplored. We introduce S4VY, a Segment Anything model built on feed-forward 4D visual geometry. From a set of RGB observations, S4VY transforms shared visual-
geometric features into an exhaustive set of class-agnostic 4D instance masks through a space-time query decoder, with each persistent object query binding one entity across all observations. This representation supports
prompt-independent segmentation as well as point- and box- conditioned selection, without requiring a seed mask or temporal ordering. We further develop an agentic harness for natural-
language grounding in the large observation space of a 4D scene. Active tree search identifies relevant frames without scanning every fixed window; a dual-stream grounder combines fine-grained VLM visual priors with geometry-consistent instance features through complementary bounding-box prediction and object-query matching; and an independent critic selects the final 4D instance mask from their predictions. Extensive experiments demonstrate state-of-the-art 4D instance segmentation and strong
language-guided grounding performance under a unified evaluation spanning static and dynamic scenes.
Read the full abstract ▾
All authors Jingdong Zhang, Xin Li, Jan Kautz, Wenping Wang, Chris Choy
Computer Vision
S2T-Unet: A Structure-to-Style Framework for Inter-Modality MRI Translation
Yichao Liu
Inter-modality MRI translation aims to synthesize missing MRI modalities from available acquisitions, reducing the need for additional scanning while preserving clinically relevant anatomical information. However, existing image translation methods often learn intensity mappings without explicitly separating modality-invariant structural information from modality-specific appearance, which may lead to structural information loss or unrealistic image details. In this work, we propose S2T-Unet, a structure-to-style framework that explicitly models these two aspects. Specifically, vector quantization is introduced at the lower-level bottleneck to encode modality-invariant structural information using a learned discrete codebook. At higher levels, a modality transformation module uses decoder features to condition and transform encoder representations toward the target modality, thereby recovering modality-specific intensity and contrast information. Experiments on the IXI multi-contrast MRI
dataset across four translation tasks demonstrate that S2T-Unet is comparable or outperform with state-of-art method.
Read the full abstract ▾
Computer Vision
Socialality Anchors: Towards Group-bounded Trajectory Prediction
Ziqian Zou, Conghao Wong, Qinmu Peng et al. (+1 more)
Trajectory prediction is a key component for understanding human behavior patterns in dynamic scenes. Researchers have devoted substantial efforts to modeling social interactions, especially group-wise interactions, since group membership often reflects shared intention, coordinated motion, and stable mutual adaptation, thus providing a persistent and semantically meaningful social prior for forecasting. However, existing group modeling methods may rely on a fixed threshold and infer groups mainly from
agents' relative positions within the observation window, overlooking the fact that grouping rules should be
agent-specific, temporally coherent, and context-adaptive across diverse personalities, culturalities, and evolving interaction contexts. Inspired by human social perception that alternates between interpersonal distance in boundary-sensitive situations and relative speed consistency in dynamic interactions, we propose Socialality, a human-inspired trajectory prediction framework with interpretable Socialality anchors and an extended grouping window for stable, context-aware grouping inference. Concretely, Socialality introduces a duo-scalar-controlled grouping kernel Socialality that jointly leverages historical observations and short-term future trajectory previews to learn
agent-specific grouping rules, and employs a group-wise perception mechanism to model in-group and out-of-group interactions in an intuitive and explainable manner. Furthermore, we conduct extensive experiments on standard
benchmarks to demonstrate the performance gains of Socialality, and provide qualitative analyses and statistical studies of anchor distributions to verify the interpretability and stability of the proposed Socialality anchors.
Read the full abstract ▾
All authors Ziqian Zou, Conghao Wong, Qinmu Peng, Xinge You
Computer Vision
RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts
Hongbin Lin, Chaoda Zheng, Yiming Yang et al. (+11 more)
End-to-end autonomous driving policies are commonly trained via imitation learning on logged demonstrations without observing the consequences of their own actions, leading to causal confusion in closed-loop real-world deployment. To address this issue, reinforcement learning (RL) post-training offers a promising alternative by leveraging world models as interactive training environments to enable future scene generation for policy improvement. Nevertheless, existing approaches either rely on reconstruction-based simulators, offering limited counterfactual interaction, or adopt synthetic simulators to enable long-horizon closed-loop interaction at the cost of a substantial sim-to-real gap. Recently,
video world models have exhibited the ability to generate realistic multi-step future rollouts but may not faithfully reflect action conditions, resulting in action-
vision mismatch. In this paper, we introduce RoXDrive, a plug-and-play closed-loop RL framework that enables reliable policy optimization by identifying action-faithful world-model rollouts, consisting of two stages: 1) Model pre-training: In addition to imitation-based policy pre-training, we devise an Action-
Vision Faithfulness Evaluator for inverse dynamics estimation with our geometry-aware auxiliary trajectory supervision, enabling long-horizon assessment of whether visual dynamics faithfully reflect the conditioning ego actions. 2) Action-faithful RL post-training:
Agents iteratively interact with world models to form long-horizon scene rollouts, retaining only action-faithful ones for dense safety-aware scoring and scene-level closed-loop RL post-training. Extensive experiments on nuScenes and an in-house
dataset with over 130K training scenarios demonstrate consistent gains across planners, reducing safety violations by 27.6% with DiffusionDrive on nuScenes and 33.7% with Qwen3-VL on the internal data.
Read the full abstract ▾
All authors Hongbin Lin, Chaoda Zheng, Yiming Yang, Xiangyu Li, Shijia Chen, Jinhao Deng, Kangjie Chen, Dongbin Zhang, Jie Feng, Yu Zhang, Xianming Liu, Shuguang Cui, Boyang Wang, Zhen Li
Computer Vision · Cross-listed from cs.CL
Rethinking Multimodal Fake News Detection in the Generative AI Era
Wenbin Shen, Guoxuan Qin, Guangxu Yao et al. (+3 more)
Generative content is increasingly entering the production and dissemination of news, transforming fake news from manually fabricated or simply manipulated material into complex forms in which native and generated content jointly participate. Existing
multimodal fake news detection research primarily focuses on veracity assessment and rarely characterizes how generativity differences affect the reliability of evidence. In contrast, AIGC detection primarily determines whether content is generated or modified by generative models, but it does not by itself establish whether the underlying news event is true. To bridge the separation between these tasks in data and evaluation, we construct Weibo26, a
multimodal fake news detection
dataset for generative-content scenarios. On this basis, we propose the Generativity-Aware Hierarchical
Reasoning (GAHR) framework, which combines global judgment with local correction so that generativity information participates in news-veracity
reasoning. Experiments on multiple existing fake news detection
benchmarks and Weibo26 show that GAHR achieves competitive veracity-detection performance while effectively identifying generative content.
Read the full abstract ▾
All authors Wenbin Shen, Guoxuan Qin, Guangxu Yao, Baodong Wang, Yuanbo Rui, Zhichao Lian
Computer Vision
GlassFormer: Learning Real-time Glass Segmentation using Radar-Depth Fusion
Suhani Grover, Astik Srivastava, Viswas Dinesh et al. (+2 more)
Transparent surfaces are ubiquitous in built environments, yet they remain a persistent failure case for robotic perception. RGB cameras perceive the background behind glass rather than the surface itself, while depth sensors such as
LiDAR, time-of-flight, and RGB-D often return invalid or background measurements in transparent regions. As a result, systems that rely solely on optical sensing may misinterpret glass walls, doors, or mirrors as free space, compromising safe and reliable navigation. Existing glass segmentation approaches address this by learning visual cues such as reflections, boundaries, and semantic context from RGB images. While effective under favourable lighting and viewing conditions, these cues degrade in low-light environments, under glare, or when glass surfaces are featureless or partially occluded. In this work, we propose a
multimodal framework that fuses millimetre-wave radar with RGB-D sensing for real-time transparent surface segmentation. Radar reflects strongly off glass surfaces, providing a
geometric cue that remains reliable precisely where
vision and depth fail. We exploit this cross-modal inconsistency to generate a radar-guided spatial prior, which is integrated into a lightweight
transformer-based segmentation network, GlassFormer, via cross-modal
attention. We report results on a mixed-condition test split covering all scene types and a dedicated low-light split designed to stress
vision-only methods. GlassFormer achieves 0.88 mIoU on the mixed split, and 0.59 mIoU on the low light split, demonstrating substantial robustness gains over
vision-only baselines while maintaining real-time performance on resource-constrained platforms.
Read the full abstract ▾
All authors Suhani Grover, Astik Srivastava, Viswas Dinesh, Avinash Sharma, K. Madhava Krishna
Computer Vision
Does the VGGT Family Need All Its Layers?
Fengyi Zhang, Holger Caesar, Xiangyu Sun et al. (+3 more)
Which layers of a feed-forward geometry model are needed to preserve both camera poses and dense 3D structure? We study layer redundancy in VGGT, $\pi^3$, and VGGT-$\Omega$: 3,018 pruned configurations, scored on seven camera-pose and dense-geometry metrics across four indoor and outdoor
datasets. Four findings follow: (i) Removable layers cluster in two redundancy regions: a dominant early region and a narrower late one, while deletions spanning the intervening layers are consistently more disruptive. This recurring pattern holds across models,
datasets, and metrics, and contrasts with the middle-to-late redundancy commonly reported in the literature. (ii) Within these regions, we observe that the joint degradation from deleting two intervals is approximately the sum of their individual degradations, reducing the number of model evaluations for
pruning search from $O(L^4)$ to $O(L^2)$, where $L$ is the aggregator depth. (iii) We find that CKA provides a cheaper representation-based proxy for interval degradation, offering a practical trade-off between
pruning quality and calibration cost. (iv) Closed-form linear calibration recovers accuracy after
pruning without end-to-end retraining. A least-squares analysis shows that using a shared map for special and patch
tokens generally incurs excess reconstruction loss, motivating
token-aware recovery. Recovery maps fitted on just 100 calibration scenes generalize to held-out scenes and unseen
datasets. The resulting models reduce aggregator parameters by up to 44% while maintaining accuracy comparable to their intact counterparts. Code and experimental results will be available at our project page: https://xian-bei.github.io/vggt-family-layer-redundancy/
Read the full abstract ▾
All authors Fengyi Zhang, Holger Caesar, Xiangyu Sun, Zheng Zhang, Zi Huang, Yadan Luo
Computer Vision
On-Policy Visual Evidence Distillation
Shaohang Wei, Feifan Song, Guangyue Peng et al. (+9 more)
Visual
agents solve problems by interleaving
reasoning with image operations, and on-policy
distillation (OPD) provides guidance from a strong teacher on student-generated interaction trajectories. However, image operations change the evidence available for subsequent
reasoning, so local errors in evidence acquisition (Acquire), reading (Read), or answer grounding (Ground) can propagate through the trajectory and lead to incorrect answers. Existing
multimodal OPD methods primarily construct or contrast auxiliary views of the original image to strengthen supervision, without explicitly modeling the connections between student actions, resulting observations, and subsequent
reasoning. This limits their ability to provide corrections tailored to different failure stages. We introduce Reflection on Visual Evidence (ReVuE), an on-policy
distillation method for visual
agents. ReVuE compares multiple student-generated trajectories for the same query, summarizes the observed visual evidence, and diagnoses the first failure across the Acquire, Read, and Ground stages. The resulting reflections provide training-time context for the teacher. We group and reweight
token-level
distillation losses according to how strongly these reflections affect the teacher's predictions. This design translates trajectory-level evidence diagnosis into targeted
token-level supervision, guiding students to improve their visual evidence acquisition and
reasoning. Across 11
benchmarks spanning the Qwen2.5-VL and InternVL3.5 model families, ReVuE outperforms all evaluated OPD baselines in weighted-average scores for perception, mathematical
reasoning, and general tasks. ReVuE also reduces redundancy in
reasoning and tool calls while improving tool-call accuracy and task accuracy. Code is available at https://github.com/sylvain-wei/ReVuE
Read the full abstract ▾
All authors Shaohang Wei, Feifan Song, Guangyue Peng, Wenhao Yu, Wei Li, Wen Luo, Yang Xu, Yufan Shen, Luke Mao, Yang Du, Asher Qin, Houfeng Wang
Computer Vision
You Cannot Recover What Was Never Measured: Quantifying the Information Ceiling of Ultra-Low-Field MRI Super-Resolution
Prathamesh Pradeep Khole, Shreya Handa, Utkarsh Gupta et al. (+1 more)
Generative super-resolution models can turn portable 64 mT MRI into images that look like 3T scans, and the field evaluates them with PSNR, SSIM, and pixelwise uncertainty, most often on pairs built by synthetically degrading high-field images. Prior work acknowledges that these models hallucinate and that the problem is ill posed, but to our knowledge no study measures how much information about the individual subject the real low-field scan actually contains. We measure it. Using paired 64 mT and 3T scans of the same subjects from three public
datasets, and a measurement protocol validated on tests whose correct answer is known in advance, we find that, judged over the whole
brain, real 64 mT scans carry structure specific to the individual only down to approximately 3 to 4 mm half-pitch in plane, and coarser still through plane. Standard synthetic degradations preserve subject information roughly 1 mm beyond this ceiling, so models trained and benchmarked on synthetic pairs are evaluated on information that real scanners never record. We then test trained
diffusion models and a publicly released external model on real paired acquisitions; 24 trained runs of five architectures (GAN,
diffusion, and
transformer families) give the coverage of the audit. On every subject where faithfulness can be measured, fine output detail is no more correlated with the subject's own 3T scan than with a stranger's, while sample-variance uncertainty does not distinguish fabricated structure from reconstruction difficulty. Because PSNR and SSIM score resemblance to a reference rather than whether detail belongs to the subject, a
benchmark scored by them cannot tell recovery from fabrication. Code for the measurement protocol will be released so that recoverability claims can be tested for newer models.
Read the full abstract ▾
All authors Prathamesh Pradeep Khole, Shreya Handa, Utkarsh Gupta, Razvan Marinescu
Computer Vision
Motion Concept Unlearning in Video Diffusion Models
Ping Liu and Chi Zhang
Text-to-
video (T2V)
diffusion models can generate realistic depictions of actions such as kicking, stabbing, and shooting, raising safety concerns that motivate targeted concept erasure. Although concept erasure has been extensively studied for static concepts in text-to-image and T2V models, erasing motion concepts remains largely unexplored. We present a systematic study of motion concept erasure in
video Diffusion Transformers (DiTs). Through causal interventions, we show that text-conditioning
attention carries concept-specific motion information and supports selective intervention, whereas perturbing temporal positional encoding suppresses both target and non-target dynamics. We further find that directly adapting ESD, a representative weight-level image erasure method, to a
video DiT yields modest and uneven motion suppression: reducing its erasure training loss does not by itself remove the concept signal from the difference between the conditional and unconditional predictions, which classifier-free guidance (CFG) then scales at every denoising step. From these findings, we derive three requirements for motion concept erasure: concept specificity, spatial selectivity, and temporal naturalness. Each determines one component of MUTE (Motion concept Unlearning in Text-to-
video gEneration): at each denoising step, MUTE extracts a concept direction through
token neutralization, derives a spatial gate from the direction's intrinsic structure, and subtracts the resulting correction from the velocity output before CFG is applied. MUTE is training-free and requires no weight modification. Experiments on 20 motion concepts show that MUTE outperforms representative
prompt-level, weight-level, and inference-time baselines on Wan2.1-T2V, and the same formulation transfers to CogVideoX, supporting its applicability across distinct T2V
attention architectures.
Read the full abstract ▾
Computer Vision
Zhenghao Ni, Weimin Qiu and Meng Tang
Video generation models have demonstrated emerging zero-shot capabilities for visual
reasoning, perception, and other
vision tasks. However,
diffusion-based
video generation is inherently
stochastic, while many downstream
vision tasks are deterministic. Motivated by the effectiveness of self-consistency in chain-of-thought
reasoning for large
language models, we investigate whether self-consistency can similarly improve
diffusion-based
video reasoning. We first introduce a training-free test-time scaling method that samples multiple
video generations and aggregates their predictions through self-consistency. Specifically, we aggregate extracted paths, locations, or masks from multiple rollouts into a consensus prediction. To reduce the inference overhead of multi-rollout generation, we read out predictions early in the denoising trajectory, which preserves consensus quality while reducing denoising steps by more than half. We further propose Rejection Fine-Tuning (RFT) to distill consensus predictions into the
video generation model. The resulting model internalizes the benefit of multi-sample consensus and requires only a single generation at inference time, while substantially outperforming the original model. Experiments on three tasks, including maze solving, visual search, and referring segmentation, show that both our self-consistency inference and consensus
distillation dramatically improve
video-based perception and
reasoning, without requiring ground-truth
videos or task-specific verification. For visual search, self-consistency raises task accuracy from 48.4% for a single generation to 99.0%. The distilled model retains much of the consensus benefit with a single rollout. For 4-by-4 maze solving, consensus-based training improves the single-generation strict success rate from 72.0% to 84.0% with the same inference latency.
Read the full abstract ▾
Computer Vision
RED: Reconstruction Evolution Dynamics for Generalizable AI-Generated Image Detection
Wenpeng Mu, Junshan Jin, Tanfeng Sun et al. (+2 more)
The rapid
evolution of image generators calls for forensic cues that generalize beyond known generation mechanisms. Existing detectors often rely on static image representations or endpoint reconstruction discrepancies, leaving the
evolution of intermediate reconstruction stages underexplored. We observe that the relative
token predictability of real and generated images can reverse across reconstruction scales, suggesting that intermediate stages may expose forensic evidence overlooked by endpoint comparisons. Motivated by this observation, we propose RED (Reconstruction
Evolution Dynamics), a framework that captures transferable forensic cues from coarse-to-fine reconstruction
evolution. To our knowledge, RED is the first framework to use scale-wise
token predictability to guide forensic evidence aggregation across intermediate reconstruction states. It represents the reconstruction trajectory produced by a frozen multiscale VQ-VAE in the shared feature space of a frozen CLIP encoder. To connect the observed predictability variations with visual evidence, RED learns image-adaptive stage weights from scale-wise
token negative log-likelihoods provided by a frozen VAR model. A cross-stage evidence aggregation module then jointly models the original-image representation and the weighted reconstruction features, capturing complementary forensic cues through interactions along the reconstruction trajectory. Experiments on six diverse
benchmarks demonstrate that RED achieves the highest average accuracy of 92.5% and average precision of 97.5% among the evaluated methods. Further evaluations show strong robustness to common image degradations, supporting the value of reconstruction
evolution for generalizable AI-generated image detection. The code will be made publicly available upon acceptance of this paper.
Read the full abstract ▾
All authors Wenpeng Mu, Junshan Jin, Tanfeng Sun, Xinghao Jiang, Qiang Xu
Computer Vision
CurvSpec: Adaptive Multi-Curvature Learning for Partial Relevant Video Retrieval
Zhen Liu, Letian Li, Jinpeng Wang et al. (+4 more)
Partially Relevant
Video Retrieval (PRVR) seeks to retrieve untrim-med
videos containing a moment that matches a text query, without temporal annotations. The relevant moment may last only seconds within a
video spanning several minutes, creating an extremely low signal-to-noise ratio that makes PRVR more challenging than standard full-
video retrieval. This task presents two intertwined challenges: (1) signal dilution, where coarse global representations blur the brief relevant signal into the dominant irrelevant surroundings;(2)
curvature rigidity, where
embedding all
videos in the same fixed-geometry space distorts representations for
videos that range from flat atomic events to deep compositional hierarchies. Existing PRVR methods have improved moment selection and cross-modal matching, but they still typically encode all
videos in a single fixed-
curvature retrieval space, limiting their ability to model diverse
video structures. To address both challenges, we propose CurvSpec, a framework that learns content-adaptive
curvature for
video retrieval representations rather than imposing a fixed
geometric prior. CurvSpec processes features through parallel Euclidean and
hyperbolic attention layers, with independently learned
curvatures assigned to the
hyperbolic layers, and a content-aware fusion mechanism routes each input to its most suitable
geometric regime. To further suppress signal dilution, CurvSpec represents each
video with semantic centroids whose number is determined by the
video's content complexity, projects them onto the learned
manifold, and matches each query against its nearest centroid by geodesic distance. Experiments on ActivityNet Captions, TVR, and Charades-STA demonstrate state-of-the-art
retrieval performance.
Read the full abstract ▾
All authors Zhen Liu, Letian Li, Jinpeng Wang, Shuzhao Xie, Yuzhi Huang, Jingyan Jiang, Zhi Wang
Computer Vision
MeteoVerse: Unified Weather-Controllable Video World Model
Renlong Wu, Guanqiao Wang, Xuan Shang et al. (+5 more)
Video world models aim to predict future content from an observed scene while following prescribed camera motion. Real-world scene
evolution is determined not only by changes in viewpoint and object dynamics, but also by environmental conditions such as weather, which can substantially alter scene appearance and visibility. Modeling such realistic weather
evolution is challenging because the required weather modification depends jointly on the observed and desired weather states. Depending on their relation, the model may need to preserve, introduce, or remove a weather effect. Existing
video world models typically leave this weather transition implicit, forcing the generation backbone to infer weather
evolution together with scene dynamics and camera motion, which leads to imprecise weather control. To address this limitation, we propose MeteoVerse, a unified weather-controllable
video world model that generates future
videos from a single sunny or adverse-weather image, conditioned on a weather-free scene description, a target-weather instruction, and a camera trajectory. Rather than conditioning only on the desired weather, MeteoVerse explicitly estimates the observed and target weather states and represents the required weather transition. A transition-aware mixture of weather experts then translates this transition into category-specific residual weather features, unifying weather preservation, introduction, and removal while enabling fine-grained control over introduced weather intensity. We further construct the MeteoVerse
dataset with over 50K real-world weather
video clips, generated sunny counterparts, disentangled scene and weather descriptions, weather-intensity annotations, and camera trajectories. Extensive experiments demonstrate substantially improved weather controllability while retaining competitive scene consistency and camera-control performance.
Read the full abstract ▾
All authors Renlong Wu, Guanqiao Wang, Xuan Shang, Yin Hanming, Xiaoxiao Sheng, Tianyu Huang, Hui Li, Wangmeng Zuo
Computer Vision
EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding
Yuwei Miao, Xuesheng Zhang, Wenhao Zou et al. (+5 more)
Real-time
video understanding requires incrementally maintaining a
memory of streaming content, and optimizing this requires dense process signals. On-Policy Self-
Distillation (OPSD), which lets one model serve as both teacher and student with the teacher receiving additional privileged information such as the question and ground-truth (GT) answer, can supply such
token-level signals. However, applying it directly to streaming
video raises two problems. (1) The student cannot be optimized end-to-end, where
memory is written before the question arrives, yet the teacher scores it with the question-and-GT privilege, misaligning their preferences. (2) Effective-entity
memory collapses, where the question-and-GT privilege makes the teacher favor only question-relevant entities, and
token-mean averaging over a
memory renders its signal invariant to how many entities that
memory covers, both driving
memory against the streaming need for diversity. To address these issues, we propose Event-Grounded Self-
Distillation (EGSD), which characterizes streaming
memory as an incremental update over verifiable Events (key visual entities, actions, and details) and targets the two problems on this basis. For problem (1), we adapt the OPSD signal into a multiplicative weight combined with the outcome reward; for problem (2), we re-weight the teacher with Events as privileged information to counter its question-relevance bias, and add an entity-coverage reward to supply the coverage preference the
token-mean teacher lacks. Extensive experiments on mainstream online and offline
benchmarks show EGSD achieves strong performance, reaching 79.8% on StreamingBench and 73.4% on the OVO-Bench Real-Time track, while
memory analysis shows effective-entity recall rises 17.4% at only 6.8% more
memory length.
Read the full abstract ▾
All authors Yuwei Miao, Xuesheng Zhang, Wenhao Zou, Jixia Zhang, Jianwei Lv, Bo Yuan, Junfeng Wang, Shiao Xie
Computer Vision
Scene Retargeting: Learning Object Placement with Analogical Transfer
Minkwan Kim, Junho Kim, Seungmin Lee et al. (+2 more)
Interactive simulations of embodied AI or spatial computing applications build on realistic 3D scenes that support daily activities. However, sparse, irregular layout structures impose scene-specific physical constraints, making it hard to define a generalizable framework for generating similar functional context. We formalize Scene Retargeting as stably transferring the semantically coherent spatial organization across layouts, rather than relying on textual descriptions or pairwise relationships. Our cluster-wise transfer flexibly handles mismatched object instances and adapts to distinctive floor plans. We optimize to preserve the rich semantic context of individual clusters by respecting the spatial distribution of foundation features. We can then impose physical constraints to refine wall contacts, pairwise
alignment, or clear passageways and openings. Our framework outperforms state-of-the-art methods on layout generation on the 3D-FRONT
dataset, and demonstrates downstream applications including real-to-sim transfer, analogical trajectory transfer, and multi-reference composition.
Read the full abstract ▾
All authors Minkwan Kim, Junho Kim, Seungmin Lee, Changwoon Choi, Young Min Kim
Computer Vision
Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs
Yueran Ma and Ronghao Lin
Omni-modal large
language models (LLMs) are expected to answer a question using the modality it explicitly refers to. However, existing training paradigms rarely verify whether models actually follow this modality, because
multimodal inputs from the same sample often provide redundant evidence for the same answer. In this work, we uncover a pervasive cross-modal shortcut in omni-modal LLMs: when asked an audio-related question, models rely on the image as much as on the audio, and sometimes even more. To systematically diagnose this behavior, we introduce the Factorized Modality Diagnostic, which independently swaps audio and images between samples to isolate each modality's causal contribution. Across two model families in different settings, we find that this shortcut persists throughout supervised fine-tuning and reinforcement learning post-training, while judge-based RL may further amplify such reliance on irrelevant visual information. Based on this finding, we propose DMC-Repair, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question. This prevents models from exploiting the spurious correspondence between modalities within the same clip. Experiments demonstrate that DMC-Repair reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-question answering performance. The reduction in shortcut reliance generalizes across two model families and zero-shot to an unseen
dataset and an unseen
benchmark, and persists through subsequent post-training. Code is available at https://anonymous.4open.science/r/DMC-Repair.
Read the full abstract ▾
Computer Vision
Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning
Cheng Ye, Weidong Chen, Zhaobo Qi et al. (+2 more)
While
multimodal large
language models (MLLMs) have demonstrated exceptional capabilities in objective understanding tasks, their performance in affective
reasoning still falls significantly short of human standards. We attribute it to a central capability gap: MLLMs are difficult to reliably distinguish semantically proximal emotions based on fine-grained visual evidence, which could be decoupled as two limitations: 1) Insufficient Attribution. The global
reasoning paradigm of conventional MLLMs severely dilutes fine-grained emotion cues, where subtle emotional states are usually implicitly encoded, thereby generating emotional misjudgments in complex scenarios. 2) Insufficient Discrimination. Existing methods could only identify regions generally associated with emotions, which fails to distinguish discriminative regions between semantically similar emotions, leading to ambiguous emotion judgements. To overcome these limitations, we present a training-free inference-time optimization framework, named Decoding Affective Nuances (DAN). Specifically, we propose a Hierarchical Emotional
Reasoning Chain (HERC) that enhances the insufficient attribution by harmonizing fine-grained scene/object-level cues and performing a soft-gated
reasoning. Furthermore, to discriminate between semantically proximal emotions, we design a Contrastive Discriminative Visual
Pruning (CDVP), which isolates discriminative visual
tokens to reason the final emotion category by computing the absolute discrepancy between the
attention distributions of similar emotions. Performances on several
benchmarks demonstrate that DAN significantly improves discrimination for affective nuances without consuming additional training resources, especially achieving +10.47% improvements with Qwen3-VL-8B-Instruct on WebEmo25
dataset that contains 25 fine-grained emotion categories.
Read the full abstract ▾
All authors Cheng Ye, Weidong Chen, Zhaobo Qi, Beier Zhu, Zhendong Mao
Computer Vision · Cross-listed from cs.RO
DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients
Xiaotian Zhang, Yusheng Wang, Naoya Kagawa et al. (+4 more)
Accurate hand-eye calibration is crucial for precision manipulation. Traditional methods rely on markers, with their precision dependent on marker accuracy and observability. In contrast, markerless methods, such as learning-based approaches, use deep neural networks to directly extract keypoints or features from images, enabling the computation of hand-eye transformation with a single image and without the need for physical markers. Recently, differentiable rendering-based methods for hand-eye calibration have leveraged physical models to render binary masks and compare them with observations, enabling hand-eye calibration without fiducial markers in the calibration stage and providing interpretable optimization. While the state-of-the-art differentiable rendering methods achieve remarkable accuracy, the use of binary masks can result in the loss of internal profile details, reducing precision. Additionally, these methods can also suffer from unstable optimization and local minima. In this study, we propose a novel RGB-based differentiable rendering framework that provides richer
geometric and appearance cues by incorporating color and mask
geometric features, thereby improving calibration accuracy and optimization stability. Additionally, we propose a mask-guided image-to-image translation method to ensure explicit preservation of color and
geometric consistency throughout the translation. Our approach is validated through both simulation and real-world experiments, with results demonstrating strong accuracy and robustness and clear improvements over existing differentiable rendering methods. Our method achieves a grasping success rate of 88.9% and insertion success rate of 57.4% on the UR5e real-world experiment, outperforming the state-of-the-art differentiable rendering hand-eye calibration method EasyHeC by 46.3 and 48.1 percentage points, respectively.
Read the full abstract ▾
All authors Xiaotian Zhang, Yusheng Wang, Naoya Kagawa, Noritaka Takamura, Keiji Okuhara, Hiroyasu Baba, Jun Ota