Computer Vision
SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding
Yue Zhang, Yingzhao Jian, Yunqiu Xu et al. (+2 more)
Understanding 3D scenes is fundamental to
embodied intelligence, requiring joint
reasoning over heterogeneous information from multiple modalities, including visual and
geometric cues. However, the relevance of these modalities often varies across queries. Existing
Multimodal Large
Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted
reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTing (SMART) module that selects task-relevant modalities using semantic priors, text-modality
alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in
multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding
benchmarks, and attains competitive results on RGB-only
video understanding
benchmarks. In our diagnostic
benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: https://yuecheong.github.io/SmartMage/.
Read the full abstract ▾
All authors Yue Zhang, Yingzhao Jian, Yunqiu Xu, Xiaoxiao Sun, Hehe Fan
Computer Vision
Predicting Brain Morphometry with MT-GNN: Mesh Evolution in Continuous Time with Graph-Based Metric Tensor Embeddings
Hao Ding, Daniel Semchin, Paul M. Thompson et al. (+1 more)
Predicting how a subcortical structure's shape will evolve from a few prior scans could support prognosis and clinical-trial enrichment. Existing longitudinal mesh predictors either extrapolate shape trajectories via high-dimensional
embeddings or regress vertex deformations directly. We instead predict the surface's intrinsic geometry in continuous time: a single per-structure graph network predicts the future per-vertex first fundamental form (metric tensor) for an arbitrary causal multiple-visit history and an arbitrary prediction horizon, conditioned on a
Fourier encoding of the lead time. The predicted metric is decoded into a surface by a differentiable As-Rigid-As-Possible solver, and the model is trained end-to-end on the rigid-aligned vertex error. Training through the reconstruction keeps the decoded prediction a valid surface and consistently improves it. On 14 subcortical structures from the ADNI
dataset, the proposed mesh
evolution model (MT-GNN) predicts best among the evaluated methods at every horizon ($-2.29\%$ mean vertex error vs. the temporal mean, $p{=}6.1{\times}10^{-5}$, beating it on 14/14 structures), ahead of geodesic shape regression (DCM, $-0.19\%$) and a mesh
transformer (TransforMesh, $-0.45\%$; $p{=}1.2{\times}10^{-4}$), with the lead widening as the horizon grows.
Read the full abstract ▾
All authors Hao Ding, Daniel Semchin, Paul M. Thompson, Boris Gutman
Computer Vision
OPD-V: Visual On-Policy Self-Distillation with Modality Balance
Aniri, Jinhe Bi, Peng Liao et al. (+5 more)
On-Policy Self-
Distillation (OPSD) has become a standard post-training approach for improving visual
reasoning in
multimodal large
language models (MLLMs). Existing methods draw privileged information from diverse input sources to guide self-
distillation. Yet these designs overlook Modality Imbalance, a challenge inherent to MLLM
reasoning. When textual information dominates generation, the model cannot fully integrate its
multimodal input. Consequently, carefully designed privileged information remains underused, limiting the effectiveness of OPSD. To examine this limitation, we construct a Positive Teacher with the Zoom-In Image and a Negative Teacher with the Mask Image, which exhibit different degrees of Modality Imbalance. Changes in their
reasoning correctness and
token logits reveal that Modality Balance can itself serve as privileged information. Motivated by this finding, we introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher. Positive Modality-Balance Logits Margins define a Modality-Balance Trust Region that selects the on-policy
tokens used for self-
distillation. Experiments across 6
benchmarks, 4 MLLM backbones, and 5 post-training methods show that OPD-V consistently improves
reasoning performance while reducing training cost.
Read the full abstract ▾
All authors Aniri, Jinhe Bi, Peng Liao, Zengjie Jin, Volker Tresp, Fei Shen, Yunpu Ma, Tat-Seng Chua
Computer Vision
IRIS: A Visual Cortex-Inspired Framework for Analyzing Orientation Selectivity in Vision Transformers
Vaishnavi B Mohan, Vijayakrishna Naganoor, Yashas Annadani et al. (+1 more)
Vision transformers (ViTs) have become the de facto standard for image encoding across many perception tasks. Despite their empirical success, it remains mechanistically unclear how they encode low-level features, given their lack of inductive biases: ViTs process information globally rather than relying on local structure. Biological visual systems, in contrast, build low-level features, such as orientation selectivity in the primary visual cortex, by combining information from small, localized regions of the visual field. These features are general-purpose representations, shared and required across multiple specialized neural pathways, unlike higher-level, task-specific semantic features. This raises the question if such biologically-grounded features arise in ViTs. In this work, we systematically study how orientation selectivity emerges in ViTs by introducing a suite of neuroscience-inspired metrics: representational similarity score (RSS), orientation recruitment score (ORS), and orientation tuning bandwidth to quantify how orientation is encoded in representational geometry and as a function of model depth. Through extensive analysis, we find that: (1) the training paradigm is the strongest determinant of orientation selectivity, with models sharing an objective, peaking at comparable relative depths regardless of scale (2) many units are orientation-selective early in training, with early-to-middle layers recruiting more such units over time, while deeper layers lose selectivity and broaden their tuning toward semantic encoding and (3) our metrics offer a mechanistic heuristic for how many layers to unfreeze for best downstream generalization. Our framework presents a way to track biologically-grounded features during ViT training, probes how desired properties are encoded in
transformer representations, and builds a systematic understanding of how ViTs generalize across tasks.
Read the full abstract ▾
All authors Vaishnavi B Mohan, Vijayakrishna Naganoor, Yashas Annadani, Shashank Hegde
Computer Vision
Robust and Efficient Motion Reasoning for Privacy-Aware Classroom Incident Recognition
Paritosh Parmar, Landy Lan, Hong Yang et al. (+2 more)
Can computer
vision help make classrooms safer? In this pilot study, we investigate
privacy-aware and computationally efficient classroom incident recognition from CCTV-style observations. This setting remains underexplored, with limited
benchmarks and few methods designed for the
privacy, efficiency, and generalization demands of real-world deployment. We introduce a novel hybrid
benchmark combining generative CCTV-style
videos with real-world classroom pose data, and propose a lightweight, but robust motion-
reasoning framework motivated by the observation that many incidents differ more in motion direction, speed, acceleration, and intensity than in pose alone. To that end, our method first constructs hierarchical kinematic representations of human actions. Our method then distills hierarchical, multi-order kinematic
reasoning from a large teacher into a much smaller single-order student, enabling efficient per-person inference while preserving expressive motion understanding. Experiments show that our model outperforms substantially larger baselines at less than one-tenth of their computational cost, while also demonstrating stronger out-of-domain motion
reasoning and zero-shot synthetic-to-real generalization. We will publicly release the
benchmark, codebase, and supporting tools to facilitate further research in
privacy-aware classroom safety.
Read the full abstract ▾
All authors Paritosh Parmar, Landy Lan, Hong Yang, Chen Yi, Chiat Pin Tay
Computer Vision
HexMIL: Hierarchical Attention MIL for Ante-Hoc Explainable Detection of AI-Manipulated CT Volumes
Orazio Pontorno, Luca Guarnera, Zahid Akhtar et al. (+1 more)
The emergence of medical deepfakes, i.e., medical images manipulated by deep generative models, poses a significant threat to clinical workflows. However, existing detectors suffer from two critical limitations: poor generalization to unseen generative architectures for manipulation detection and lack of interpretability. In this context, we present HexMIL (Hierarchical EXplainable Multiple Instance Learning), a mask-free medical deepfake detector that simultaneously addresses both limitations using only binary volume-level supervision. HexMIL decomposes each CT volume into a two-level hierarchy of patches and slices, aggregated via independent Gated
Attention modules whose weights are directly combined into a full-resolution 3D
attention volume that localizes the manipulated sub-region without any pixel-level annotation. Unlike post-hoc methods such as Grad-CAM, HexMIL's
attention weights constitute the exact forward computation driving the classification decision, providing ante-hoc and structurally faithful spatial attribution. We evaluate HexMIL on M3DSynth and CT-GAN
datasets under a rigorous cross-generator generalization protocol, training on a single generative architecture and testing on unseen ones. HexMIL outperforms all baselines by $+9.1$ AUC and $+9.4$ F1 in out-of-domain classification, and achieves the best average IoU and Pointing Game score in localization. Project page: opontorno.github.io/hexmil.
Read the full abstract ▾
All authors Orazio Pontorno, Luca Guarnera, Zahid Akhtar, Sebastiano Battiato
Computer Vision
Lesion Detection in CT with Frozen Self-Distilled Features: SALT, a Spatially Adaptive Label-Guided Temperature
Mahmut S. Gokmen, Evan W. Damron, Mitchell A. Klusty et al. (+3 more)
Self-supervised pretraining objectives are spatially uniform: the teacher temperature and the per-patch loss weight are identical everywhere in the image, so a lesion a few patches wide contributes no more to the training signal than the surrounding parenchyma. Prior work biases the views toward annotated regions, which changes what the model sees but adds no pressure on the objective. We instead condition the targets of self-
distillation, a method we call SALT (Spatially Adaptive Label-guided Temperature). Weak, box-derived labels, available only during pretraining, define a compact region on the encoder's patch grid, inside which the teacher's softmax temperature is sharpened and the masked-patch loss is up-weighted. The objectives, the masking policy and the centering statistics are otherwise unchanged, and at every downstream use the encoder is a plain feature extractor with no labels and no conditioning. We evaluate by freezing the encoder and training only a lightweight multi-depth CenterNet-style head, detecting lesions in 3D on four CT cohorts, and we isolate the mechanism against a backbone identical in architecture, pretraining data, schedule and label-guided cropping but with no target conditioning. We report patch-level separability, 3D detection stratified by cohort and by lesion size, box quality, and a detector-free probe in which a single frozen patch
embedding re-identifies a lesion in a follow-up scan without registration, masks or fine-tuning. Because the conditioning is expressed through a spatial indicator rather than through label semantics, the formulation admits any weak spatial annotation; we instantiate and validate it for lesions.
Read the full abstract ▾
All authors Mahmut S. Gokmen, Evan W. Damron, Mitchell A. Klusty, Caroline N. Leach, Emily B. Collier, V. K. Cody Bumgardner
Computer Vision
Bag-of-Visual-Words for Spatial Mapping of Lung Adenocarcinoma Growth Patterns
Darya Ardan, Valentin Oreiller and Henning Müller
Spatial mapping of lung adenocarcinoma (LUAD) growth patterns across whole slide images (WSIs) requires resolving architectural context at the region level, yet existing methods operate at the individual tile level and produce generic morphological clusters rather than clinically defined pattern maps. We propose a weakly supervised Bag-of-Visual-Words (BoVW) pipeline that learns a visual vocabulary from frozen foundation model
embeddings extracted from a small set of annotated regions of interest (ROIs). Pattern prototypes are constructed as mean BoVW histograms of same-label ROIs and used for nearest-prototype classification of sliding-window regions under Jensen–Shannon divergence. The resulting predictions are projected onto the WSI tile grid to produce interpretable spatial pattern maps. We evaluate the method on 87 CPTAC-LUAD patients using three foundation model encoders and multiple vocabulary sizes on two clinically motivated tasks. For tumour/healthy classification, the best configuration achieves a balanced accuracy of $0.974$ with H-Optimus-1, approaching the $0.987$ obtained by a supervised SVM trained on mean-pooled WSI
embeddings. For binary histologic grade classification, the BoVW pipeline achieves higher balanced accuracy than the supervised baseline for all encoders, suggesting that ROI-level pattern decomposition preserves grade-relevant heterogeneity that is attenuated by global mean pooling.
Read the full abstract ▾
Computer Vision
HelloWorld: Enabling Socially Interactive Characters in Video World Models
Liangyang Ouyang, Ruicong Liu, Xuangeng Chu et al. (+2 more)
Despite the remarkable recent progress of
video world models, social interaction between users and the characters within these worlds remains unsupported. To fill this gap, we present HelloWorld, a
video world model that enables social interaction with in-world characters. With a single button press, users can
prompt the on-screen character to respond toward the camera, e.g., turning to the viewer, waving, nodding, or speaking a short greeting. To make these interactions natural, we propose a self-
distillation pipeline that finetunes the
video generation model on data synthesized by itself. Each synthesized clip contains both social interactions and camera motion, allowing the model to learn camera-pose conditioning without degrading interaction quality. At inference, we further introduce a training-free module that determines when the interaction occurs. Upon a button press, it modulates the cross-
attention masks of the DiT so that the interaction-related text
prompt attends only to the frames within the press window, temporally localizing the character's response. We further build HelloWorldBench, a 400-sample
benchmark with three social interaction metrics alongside three conventional metrics, for evaluation. Experiments demonstrate that HelloWorld surpasses a variety of baselines in interaction quality, while maintaining state-of-the-art picture aesthetics and camera-pose following. Project page: https://github.com/AlayaLab/HelloWorld
Read the full abstract ▾
All authors Liangyang Ouyang, Ruicong Liu, Xuangeng Chu, Kaipeng Zhang, Yoichi Sato
Computer Vision
VQ-VAD: Vector-quantized Motion Representation Learning for Human-centric Video Anomaly Detection
Narges Rashvand, Ghazal Alinezhad Noghre, Shanle Yao et al. (+2 more)
Video Anomaly Detection (VAD) is inherently challenging due to the scarcity of anomalies and the large visual variability in surveillance footage, including changes in lighting, viewpoint, and human appearance. To mitigate visual noise and address
privacy concerns, recent work has shifted to pose-based VAD, which focuses on motion dynamics rather than raw
video data. However, existing pose-based approaches model human behavior in continuous latent spaces, limiting their ability to learn compact motion patterns necessary for robust behavior analysis. We address this by proposing Vector-Quantized
Video Anomaly Detection (VQ-VAD), a novel human-centric anomaly detection framework that learns discrete motion representations. VQ-VAD adapts Vector-Quantized GAN (VQ-GAN), originally developed for image generation, to operate on keypoint sequences and construct a motion codebook of normal behavior. Trained exclusively on normal motion sequences, VQ-VAD detects anomalies by identifying high reconstruction errors when an observed motion sequence cannot be mapped to the learned codebook. We conduct extensive experiments across three complementary evaluation settings, including in-domain, cross-domain, and cross-
dataset generalization, on four anomaly detection
benchmarks. VQ-VAD achieves strong in-domain accuracy (81.83% on HR-SHT [15]), effective cross-domain transfer from CMU Panoptic [14] (76.69% on HR-SHT [15] without retraining), and competitive cross-
dataset robustness. The code base for this work is available at https://github.com/TeCSAR-UNCC/VQ-VAD.
Read the full abstract ▾
All authors Narges Rashvand, Ghazal Alinezhad Noghre, Shanle Yao, Gabriel Maldonado, Hamed Tabkhi
Computer Vision
Beyond Reprojection Error: Camera Calibration with 3D Targets
Dennis Ruppel, Hasan Kutlu, Kai A. Neumann et al. (+4 more)
In 3D reconstruction, camera calibration is an essential element for achieving high fidelity and accuracy of the reconstructed geometry. While existing approaches rely upon 2D planar calibration, this work proposes a framework tailored for 3D reconstruction that is based on predicting scene rays, which adds flexibility to the reconstruction pipeline and enables the use of recent advances in camera models. Novel metrics, reconstruction and intersection error, derived from predicted scene rays are employed in combination with a bootstrapping procedure that statistically evaluates different calibration objects and calibration pipelines for both intrinsic and extrinsic camera parameters. The results show that the generalized distortion model more faithfully captures physical camera effects and yields an improvement in calibration accuracy. Reprojection error is shown to be a potentially misleading indicator of 3D accuracy, and the proposed ray-based metrics provide a more holistic assessment. An icosahedron calibration target is designed to enrich calibration information for 3D reconstruction together with a ring-feature-based detector. The icosahedral target yields approximately 40% lower mean intersection and more stable calibration across bootstrap trials on synthetic data, while real-data performance demands very tight fabrication tolerances.
Read the full abstract ▾
All authors Dennis Ruppel, Hasan Kutlu, Kai A. Neumann, Martin Knuth, Pedro Santos, Andreas Weinmann, Arjan Kuijper
Computer Vision · Cross-listed from astro-ph.EP
MarsCast: Transfer Learning of AI Weather Foundation Models to Planetary Atmospheres
M.L. Carroll, J. Li, S.D. Guzewich et al. (+3 more)
We investigate the transferability of Earth weather foundation models to planetary atmospheres by adapting the GraphCast graph neural weather
forecasting model to Mars. While GraphCast achieves state-of-the-art performance for terrestrial
forecasting, its applicability to non-Earth environments remains unexplored. Using the Mars
Climate Database (MCD), which provides global atmospheric fields across vertical altitude levels (similar to Earth pressure levels), we evaluate zero-shot and fine-tuned GraphCast predictions of Martian temperature and wind fields. Zero-shot forecasts produce a surprisingly accurate depiction of current conditions but fail to reproduce diurnal variability and rapidly decay toward climatological mean states. To address this limitation, we fine-tune GraphCast using MCD variables and top-of-atmosphere
solar radiation forcing while holding humidity constant. Fine-tuning enables rapid learning of Martian thermal variability. Within as few as 10 training epochs, the model begins to capture the diurnal cycle and forecasts up to 10 days reproduce seasonal and vertical temperature structure. Prediction quality improves with training sample size and exhibits sensitivity to seasonal initialization. These results demonstrate that Earth-trained AI weather models can be adapted to simulate Martian atmospheric dynamics, providing a pathway toward rapid planetary weather prediction to support mission operations,
dust storm risk mitigation, and future human exploration.
Read the full abstract ▾
All authors M.L. Carroll, J. Li, S.D. Guzewich, G.Villanueva, J.A. Caraballo-Vega, M.J. Frost
Computer Vision
OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editing
Chenxuan Miao, Yutong Feng, Yi Lu et al. (+6 more)
Instruction-based
video editing (IVE) is an emerging field with broad applications, yet evaluating editing models remains challenging. Existing
benchmarks suffer from two major limitations: limited task coverage inherited from image editing, which overlooks
video-specific dimensions, and inadequate metrics that fail to measure instruction fidelity, allowing incorrect edits to receive high scores due to strong visual priors from the original
video. To address these issues, we introduce a comprehensive and structured
benchmark for IVE. Our
benchmark decomposes editing tasks into multiple
video-specific dimensions, including spatial, temporal, audio, and reference-based editing, extending beyond conventional frame-level evaluation. It also distinguishes explicit and implicit instructions and incorporates
reasoning-based scenarios to better reflect real-world requirements. Furthermore, we propose an evaluation framework that assesses editing quality from four complementary dimensions: accuracy, preservation, realism, and consistency, using both human judgments and state-of-the-art
vision-
language models. To emphasize instruction fidelity, we introduce an accuracy-aware penalty mechanism that conditions other scores on accuracy, preventing visually plausible but incorrect edits from receiving inflated evaluations. Extensive experiments on representative open-source and commercial models show that current IVE models remain far from satisfactory. OmniEdit-Bench provides a comprehensive and reliable testbed for evaluating instruction-based
video editing and offers insights into future research directions. The project page is https://omniedit-bench.github.io/.
Read the full abstract ▾
All authors Chenxuan Miao, Yutong Feng, Yi Lu, Yunfeng Yan, Donglian Qi, Shiwei Zhang, Yu Liu, Xi Chen, Hengshuang Zhao
Computer Vision
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Junlin Han, Shengbang Tong, David Fan et al. (+4 more)
Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified
multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of
multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world
datasets yield four key insights into the physics of
multimodal pretraining: (i) Knowledge Flow: We disentangle how
language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared
attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late
alignment or sequential training. This process uncovers a
vision laziness phenomenon, where delayed integration leads models to rely on
language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T
tokens. We hope this study provides a principled foundation for understanding and scaling
multimodal pretraining.
Read the full abstract ▾
All authors Junlin Han, Shengbang Tong, David Fan, Minghao Chen, Philip Torr, Filippos Kokkinos, Mike Lewis
Computer Vision
Promptable Animal Pose Tracking Across Species
Le Li, Daniela Ivanova and Nicolas Pugeault
Animal pose estimation and tracking is important for wildlife monitoring and conservation research, and with limited expert time for labelling automated approaches are imperative. While human pose estimation and tracking has seen rapid progress thanks to large annotated
datasets, animal pose remain challenging, due to large morphological and behavioural differences between species and limited annotated data. Existing approaches either optimise generic keypoint localisation from annotated
datasets (such as APTv2) with poor generalisation, or track custom keypoints using visual tracking, at the cost of performance. In this paper, we demonstrate that
vision foundation models trained on large
datasets can be used effectively to track animal pose with limited labelled data. We propose two models, one unsupervised and the other supervised, to track user-selected keypoints in
videos. The supervised approach delivers superior tracking accuracy by employing a keypoint
prompt encoder to explicitly inject structural priors from a reference frame into feature matching. In parallel, the unsupervised route provides strong cross-species robustness by leveraging diverse foundation-model features for training-free correspondence matching. Extensive evaluation on challenging animal
video benchmarks APTv2 and TigDog demonstrates that our framework achieves strong performance while maintaining an effective balance between accuracy and generalisation, offering a practical solution for real-world animal behaviour analysis and conservation applications.
Read the full abstract ▾
Computer Vision
ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing
Xu Guo, Zhengxuan Wei, Xinghui Li et al. (+11 more)
Recent
video models increasingly support generation, reference conditioning, and editing within a single model, yet typically expose them as separate operations over fixed inputs. Practical creation unfolds across multiple shots, requiring one model to generate from text, follow a reference, or edit source footage while maintaining shared history. We formalize this setting as interactive multi-shot
video creation (IMVC) and introduce ContextMaster, a unified model with a role-aware context representation for these operations. An interactive model must retain access to an expanding history without allowing the context read cost at each denoising step to grow. ContextMaster combines reusable clean context states with fixed budget sparse context routing and uses ConstraintSink to keep task constraints visible. To address the dual challenges of sparse context access and inference with few denoising steps, we propose a two-stage privileged context
distillation framework, which transfers full context behavior from a dense teacher through consistency
distillation and then refines deployment rollouts with distribution matching. Experiments on the three primitive tasks demonstrate improved task fulfillment and consistency across shots over specialized baselines. User studies further validate flexibly composed workflows, while the model reaches 16 FPS on a single GPU.
Read the full abstract ▾
All authors Xu Guo, Zhengxuan Wei, Xinghui Li, Hanzhuo Huang, Xinyu Liu, Xiangyang Luo, Min Wei, Yiran Zhu, Qiulin Wang, Yulong Xu, Xintao Wang, Pengfei Wan, Qi Fan, Xiangwang Hou
Computer Vision
Towards Valid B-Rep Generation: Training-Free Wireframe Anomaly Detection and Repair
Jingyu Wu, Youcheng Cai, Tengyu Luo et al. (+1 more)
Multi-stage boundary representation (B-Rep) generation leverages intermediate wireframes to synthesize CAD models. However,
geometric and topological risks in these wireframes – such as self-intersections, edge collapses, and disconnected vertices – can propagate to invalid final B-Reps. Mitigating such failures by retraining large generative models is computationally prohibitive. We propose Wireframe Detection and Repair (WDR), a training-free framework that intervenes at the intermediate wireframe stage to improve downstream B-Rep validity. WDR features a
Geometric-
Topology Anomaly Detector (GTAD) that combines parallel VLM-based coarse screening with
geometric and topological detectors to predict downstream invalidity risk and route generation to dedicated branches. An Energy-Guided
Geometric-
Topology Repair (EGGTR) module then performs detector-triggered guided regeneration through geometry and
topology branches. By scaling test-time computation via Energy-Guided Resampling and training-free guidance for
diffusion models, WDR can be integrated into autoregressive and
diffusion pipelines without retraining. Extensive experiments demonstrate consistent improvements in kernel-checked validity while largely retaining the measured diversity and distributional quality of synthesized CAD models. The code will be made publicly available upon acceptance.
Read the full abstract ▾
All authors Jingyu Wu, Youcheng Cai, Tengyu Luo, Ligang Liu
Computer Vision
UG-UMRE: Uncertainty-Guided Modality Augmentation and Distributional Calibration for Unified Multimodal Relation Extraction
Bo Kong, Liruiz Jia, Yi Liang et al. (+5 more)
Unified
Multimodal Relation Extraction (UMRE) aims to identify intra-modal and cross-modal relations between textual entities and visual objects. However, existing UMRE studies still encounter two critical issues: ignoring inherent aleatoric uncertainty causes noise propagation, and deep-seated heterogeneity between distinct modal distributions hinders
alignment. To address these issues, we propose the Uncertainty-Guided UMRE Network (UG-UMRE). Specifically, we design an Uncertainty-Driven Unimodal Augmentation (UDUA) module, which models features as Gaussian distributions based on the Variational Information Bottleneck. By incorporating an uncertainty-aware self-supervised contrastive learning mechanism, UDUA effectively filters out noise while maintaining semantic consistency. Furthermore, we introduce the Joint Aleatoric Uncertainty
Alignment (JAUA) module as a global semantic pre-calibration mechanism. JAUA leverages probabilistic distribution consistency to construct a shared latent space, eliminating the distributional gap by synchronizing cross-modal statistical properties, thereby laying a robust foundation for fine-grained interaction. Experiments on three
benchmark datasets (UMRE, MORE, and MNRE) demonstrate that UG-UMRE achieves state-of-the-art performance. Further analysis validates the pluggable and effective performance of the proposed UDUA and JAUA modules.
Read the full abstract ▾
All authors Bo Kong, Liruiz Jia, Yi Liang, Chao Liu, Dongfang Han, Tianwei Yan, Yuan Liu, Shengquan Liu
Computer Vision
Unleashing the Potential of Vision-Language Models for Generalizable AI-Generated Image Detection
Weihan Cai, Hao Tan, Zichang Tan et al. (+2 more)
Recent work has shown that a simple linear probe on frozen representations from modern
vision foundation models (VFMs) can achieve state-of-the-art AIGI detection performance, substantially outperforming specialized detectors in challenging in-the-wild scenarios. This finding has established DINOv3 as the dominant foundation-model baseline for subsequent improvements. However, we find that the
vision-
language model Perception Encoder (PE) holds greater potential for AIGI detection, because its
language-aligned representation preserves high-level provenance semantics. Specifically, PE exhibits stronger local provenance organization than DINOv3 in its frozen feature space. However, semantic-agnostic linear probing fails to exploit this structure, as PE-Linear still underperforms DINOv3-Linear by 4.1% on In-the-Wild. Based on this observation, we propose Semantic Prototype Calibration (SPC), which constructs category prototypes from forensic semantic information and calibrates them with supervised data. We apply SPC to PE and refer to the resulting detector as PE-SPC. Our analysis shows that this simple design achieves stronger generalization. Across cross-generator, post-processing, and in-the-wild
benchmarks, PE-SPC surpasses the previous DINOv3 baseline and achieves new state-of-the-art results.
Read the full abstract ▾
All authors Weihan Cai, Hao Tan, Zichang Tan, Jun Wan, Xinping Gao
Computer Vision
An active-learning framework for real-time depth perception from monocular vision streams
Xiaorong Zeng, Weiqiang Chen, Peng Shi et al. (+4 more)
Biological visual systems can perceive depth from
monocular vision flow, continuously integrating temporal visual cues while maintaining a balance between stability and plasticity in dynamic environments. In contrast, artificial perception models deployed on resource-constrained edge devices are typically trained in a static offline manner and remain frozen after deployment, often suffering severe performance degradation under domain shifts. While large-scale models may encode broad knowledge through massive parameter redundancy, lightweight networks face a static optimization dilemma: forcing compact models to learn universal
geometric representations is computationally inefficient and often leads to performance saturation. To resolve this issue, an Online Active Learning (OAL) mechanism is introduced to endow compact neural networks with the capability to adapt continuously during operation. A closed-loop Predict-Evaluate-Correct learning paradigm is established to actively select high-confidence, information-rich signals from streaming visual input. Crucially, Elastic Weight Consolidation (EWC) is employed not merely to prevent catastrophic forgetting, but to enforce Selective Plasticity, preserving parameters that encode globally relevant structural knowledge while allowing local
alignment to newly observed environments. Built upon a MobileNetV3-Small backbone, the proposed system achieves approximately a 75% reduction in computational cost while maintaining competitive depth estimation accuracy. Experimental results demonstrate that adaptability is not solely determined by model size, but rather by how effectively parameter plasticity is regulated in dynamic environments.
Read the full abstract ▾
All authors Xiaorong Zeng, Weiqiang Chen, Peng Shi, Liang Su, Zirui Wang, Xuewu Ji, Shuiwen Shen
Computer Vision
Enhancing Low Back Pain Assessment with Diffusion Models for Lumbar Spine MRI Segmentation
Maria Monzon, Thomas Iff, Ender Konukoglu et al. (+1 more)
This study introduces a
diffusion-based framework for robust and accurate semantic
segmentation of lumbar spine MRI scans from patients with low back pain (LBP), regardless of whether the scans are T1- or T2-weighted. We compared with advanced models for segmenting vertebrae, intervertebral discs (IVDs), and spinal canal using the SPIDER
dataset. The results showed that SpineSegDiff achieved a
segmentation performance comparable to that of the state-of-the-art non-
diffusion nnUnet, particularly in improving the identification of degenerated IVDs. In addition, the uncertainty maps generated by our model provide valuable insights for clinical review, enhancing the robustness and reliability of the
segmentation results. The potential of
diffusion models to enhance the diagnosis and management of LBP through more precise analysis of pathological spine MRI is underscored by our findings.
Read the full abstract ▾
All authors Maria Monzon, Thomas Iff, Ender Konukoglu, Catherine R. Jutzeler
Computer Vision
Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation
Zehua Chen, Junyou Wang, Yuxuan Jiang et al. (+5 more)
Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional
diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or
reasoning from large
multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during
diffusion training and sampling process, respectively. Extensive experiments on
benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.
Read the full abstract ▾
All authors Zehua Chen, Junyou Wang, Yuxuan Jiang, Zhenying Fang, Yusheng Dai, Jianfei Chen, Ziwei Liu, Jun Zhu
Computer Vision · Cross-listed from cs.AI
When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit
Ziang Wei, Minjun Yu, Zheyuan Lai et al. (+2 more)
Defensive driving scores are useful only when they preserve distinctions between policies that observe surrounding actors and those that do not. Re-simulation
benchmarks may use reference-conditioned forgiveness, under which an
agent receives credit when the logged human reference fails a compliance channel. When
agent and reference share an unstable rollout transformation, this rule can propagate shared reference failures into broad compliance credit. We audit this risk in NAVSIM v2.2 original scene single-stage scoring. Under the affected documented-stack condition on the audited numerical backend, the route-blind Ignore-All probe and a route-aware actor-blind probe outrank human replay and PDM-Closed over the complete 12,146-
token navtest split. A fresh installation following the public specification reproduces rollout divergence on a fixed 32-
token diagnostic set. A same-source dependency stack control and an exact-input diagnostic isolate dependency-sensitive numerical behavior in the shared velocity refit. On a 450-
token control pool, replacing only the solver eliminates rollout divergence and restores blind-last ordering while keeping forgiveness enabled. Thus, the numerical instability is the direct trigger. Reference-conditioned forgiveness propagates the resulting shared reference failures into compliance credit. We contribute an audit protocol requiring score basis and stack disclosure, blind probes, overwrite reporting, and rollout stability tests before using such scores for defensive driving claims.
Read the full abstract ▾
All authors Ziang Wei, Minjun Yu, Zheyuan Lai, Mingjie Pang, Wei Li
Computer Vision
STEP-OPD: Rethinking Output Targets and Internal Dynamics in On-Policy Distillation for Diffusion Models
Qingyan Wei, Guangzhao Li, Xiaobing Tu et al. (+5 more)
On-policy
distillation (OPD) has become an effective approach for consolidating multiple task-specialized image generation models into a single student. However, existing OPD methods optimize the student mainly to match the teacher's output velocity, making the teacher the upper limit of the optimization objective. While output-level supervision alone leaves the student's blockwise representation
evolution underconstrained, which weakens the transfer of capabilities that must be progressively developed across layers. We propose STEP-OPD, an on-policy
distillation framework for image generation that extends the student's learning target beyond the teacher and introduces explicit constraints on its internal representation
evolution. Instead of treating the teacher as the final target, we use the velocity difference between each task-specific teacher and the shared base model as a direction for further learning and add a scaled version of this difference to the teacher velocity. In addition, we align the direction and magnitude of representation changes between the student and teacher, enabling the student to learn how representations are progressively transformed across network blocks. Experiments on compositional
alignment, text rendering, and human preference show that our method consistently improves Standard OPD methods. In particular, it increases the GenEval score of DiffusionOPD from 0.927 to 0.961, while also improving OCR and all preference-based metrics. The resulting unified student surpasses the corresponding single-task teachers across all three capability groups, showing that output extrapolation enables beyond-teacher learning. And representation change
alignment provides complementary guidance for the student's internal transformations.
Read the full abstract ▾
All authors Qingyan Wei, Guangzhao Li, Xiaobing Tu, Yinggui Wang, Xiantao Zhang, Jinkui Ren, Xiaohong Liu, Linfeng Zhang
Computer Vision
Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations
Ali Khoramfar, Mohammad Javad Dousti, Alireza Mohamadian et al. (+1 more)
Standard accuracy metrics for VLMs often mask significant reliability failures in sensitive domains. In this work, we utilize a histopathology-validated
brain MRI
dataset to systematically assess the diagnostic robustness of four VLM families under evidence-preserving perturbations. By reordering anatomical slices and swapping target label positions, we evaluate whether models maintain consistent predictions when clinical evidence remains invariant. Our results reveal significant vulnerabilities in presentation-order stability, with models exhibiting prediction flips in up to 48.9% of cases under simple sequence reversals. We further identify a textual selection bias, where label reordering triggers inconsistent diagnoses in up to 67.8% of cases despite identical visual inputs. Negative-control tests further reveal diagnostic overcommitment: models generate categorical diagnoses in up to 76.1% of cases after expert-annotated lesion slices are removed. These results demonstrate that high accuracy can overestimate clinical reliability, masking sensitivity to sequential presentation and textual framing that is not captured by aggregate accuracy. Our findings highlight the necessity of stability-based metrics for the deployment of VLMs in safety-critical clinical applications. Our evaluation data and code will be made public upon acceptance.
Read the full abstract ▾
All authors Ali Khoramfar, Mohammad Javad Dousti, Alireza Mohamadian, Heshaam Faili
Computer Vision · Cross-listed from cs.LG
Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision
Grzegorz Gruszczynski, Pawel Olszowiec, Michal Byra et al. (+2 more)
Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized. Single-block recurrent ViTs (bViT) remove this growth by repeatedly applying one shared block. Rather than proposing a new architecture, we fix a bViT and provide a controlled empirical characterization of three training and inference regimes under a common CIFAR-100 protocol, asking: (i)~when does recurrence beat independently parameterized depth—at matched FLOPs or at matched parameter
memory? (ii)~when a residual recurrent block is trained through an ODE solver, does solver order act as numerical refinement or as an architectural bias? and (iii)~what does robustness beyond the training horizon cost in nominal accuracy? We find that standard ViTs remain preferable when FLOPs are the primary constraint, whereas recurrent ViTs offer a better accuracy–parameter trade-off under
memory constraints. Consistent with the standard view of residual networks as Euler discretizations of ODEs, the continuous-time analogue of a residual recurrent block is the state-subtracted vector field $\dot{z}=F_\theta(z)-z$; although known in principle, this distinction is easy to violate when the block is wrapped as a black-box vector field, and we qualify the cost at few accuracy points. Because the vector field is learned jointly with the solver, higher-order solvers act as a solver-induced architectural bias rather than a numerical-accuracy improvement, and their gains are not uniform. Finally, stage-wise deep supervision traces an accuracy–robustness frontier: it does not improve nominal accuracy, but degrades gracefully far beyond the training horizon, where naive recurrence collapses to near-random performance.
Read the full abstract ▾
All authors Grzegorz Gruszczynski, Pawel Olszowiec, Michal Byra, Grzegorz Stefanski, Alberto Presta
Computer Vision
Persistent Object Narratives for Token-Efficient Video Language Models
Junzhe Chen, Siyuan Meng and Xiaojie Guo
Video large
language models (
Video-LLMs) have made strong progress in open-ended
video understanding. However, their visual interfaces remain
token-intensive and provide limited explicit structure for linking recurring object evidence across time. We introduce SlotNarrative, a slot-based interface that organizes a
video into persistent object narratives represented by compact object-state
tokens. Rather than compressing frame-wise features before establishing temporal correspondence, SlotNarrative first groups visual features into object-like slots and then associates recurring observations with clip-level object entries through a lightweight, parameter-free
memory that integrates multiple complementary matching cues. Each retained entry is serialized into two
token types: an identity
token that summarizes persistent object appearance and a set of state
tokens that encode segment-level appearance, geometry, visibility, and trajectory information. This design yields an interface of only 144 allocated visual-
token positions for a frozen
Video-LLM, independent of the number of sampled frames. Across multiple
datasets, SlotNarrative achieves a favorable trade-off between accuracy and visual-
token count compared with prior compact
Video-LLM interfaces. Experimental results establish persistent object narratives as a compact, structured, and temporally organized visual interface for
Video-LLMs. Our code will be made publicly available.
Read the full abstract ▾
Computer Vision
Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchen
Chengming Feng, Hesam Araghi, Liming Zheng et al. (+4 more)
Event cameras, also known as neuromorphic cameras, have gained significant
attention in recent years due to their high temporal resolution, high dynamic range, and low power consumption. While many studies and
datasets in neuromorphic
vision have focused on automotive and drone applications, human-centric daily-life scenarios remain largely underrepresented, despite their importance for developing and benchmarking event-based perception systems. Moreover, the few existing event-based human activity
datasets are typically recorded with scripted human actions, limiting their ability to capture natural human behaviors. In this paper, we introduce EventKitchen, a large-scale stereo event camera
benchmark dataset of human cooking activities in the kitchen. EventKitchen is egocentrically collected from 10 participants in 13 diverse kitchens, where the participants wear a helmet with multiple sensors and naturally perform cooking activities, without any scripted actions. EventKitchen comprises 5.5 hours of stereo event recordings with synchronized RGB, depth, and IMU data. We provide human annotations for 10,762 action segments and 13,482 bounding boxes. We train baseline models on EventKitchen to perform multiple event-based tasks, including action recognition, object detection, and stereo depth estimation. By capturing natural, real-world human activities, EventKitchen establishes a challenging
benchmark for neuromorphic
vision beyond autonomous driving.
Read the full abstract ▾
All authors Chengming Feng, Hesam Araghi, Liming Zheng, Julien Dupeyroux, Xucong Zhang, Jan van Gemert, Nergis Tömen
Computer Vision
Towards a satellite image manipulation and deepfake localization benchmark dataset
Jacob Arndt, Debvrat Varshney, Philipe Dias et al. (+1 more)
Verifying the authenticity of satellite imagery has become increasingly critical given advances in generative artificial intelligence. Highly realistic synthetic imagery produced for malicious purposes (deepfakes) can have major consequences in the remote sensing domain, where this data is a fundamental source of information for science applications, planning, logistics, and monitoring. The remote sensing community lacks high-quality, fine-grained manipulation
datasets suitable for training and evaluating detection and image forensics
algorithms. Existing
datasets are lacking and those that do exist either provide no ground truth masks for evaluating manipulation localization, or consist of entire images generated by GANs or
diffusion models, which are inadequate for measuring localization performance. To address this gap, we describe a preliminary
dataset construction process and prototype
benchmark dataset for satellite image manipulation detection and localization. The
dataset contains 60 images total, with 30 images carefully manipulated using three manipulation types including copy-paste splicing and
diffusion model inpainting, and 30 authentic images. Each image is accompanied by a ground-truth mask and acquisition metadata, enabling both pixel-level localization metrics, image metadata studies, and analyses of how manipulation detection performance relates to image collection parameters. We describe the
dataset construction process and present this initial release to support further research in image forensics and geospatial deepfake detection. The prototype
dataset can be downloaded at https://huggingface.co/
datasets/geodf/fmow-fake-small.
Read the full abstract ▾
All authors Jacob Arndt, Debvrat Varshney, Philipe Dias, Nivedita Nukavarapu
Computer Vision
RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection
Zian Wang, Hangchuan Liang, Yuehua Chen et al. (+4 more)
RGB-infrared (RGB-IR) object detection benefits from complementary visible and thermal cues, but effective fusion remains challenging under illumination changes, weather variation, and cluttered scenes. Existing RGB-IR fusion methods often trade expressive patch-level interaction for lighter but more constrained adaptation mechanisms. We empirically observe that pretrained register
tokens contain both modality-shared and modality-specific information on paired RGB-IR inputs, suggesting that they can serve as a compact substrate for cross-modal communication. Building on this observation, we propose RegisterBridgeMM, a register-mediated fusion framework organized as a three-stage register lifecycle. Aggregate preserves per-modality register summarization inherited from pretraining; Bridge performs bidirectional register-to-patch reading with consensus-residual regulation; and Project translates the resulting register summary into spatially adaptive calibration of patch features. This register pathway avoids dense patch-to-patch cross-modal interaction while preserving the pretrained patch representation. With both backbone streams frozen, RegisterBridgeMM achieves the highest mAP50-95 among the evaluated methods on all four
benchmarks: LLVIP, M3FD, DroneVehicle, and FLIR-Aligned.
Read the full abstract ▾
All authors Zian Wang, Hangchuan Liang, Yuehua Chen, Changchun Li, Chaoyi Guo, Mingzhe Liu, Fangming Gu