Computer Vision
HPC: Hierarchical Point-based Latent Representation for Streaming Dynamic Gaussian Splatting Compression
Yangzhi Ma, Bojun Liu, Wenting Liao et al. (+3 more)
While dynamic Gaussian Splatting has driven significant advances in free-viewpoint
video, maintaining its rendering quality with a small
memory footprint for efficient streaming transmission still presents an ongoing challenge. Existing streaming dynamic Gaussian Splatting compression methods typically leverage a latent representation to drive the neural network for predicting Gaussian residuals between frames. Their core latent representations can be categorized into structured grid-based and unstructured point-based paradigms. However, the former incurs significant parameter redundancy by inevitably modeling unoccupied space, while the latter suffers from limited compactness as it fails to exploit local correlations. To relieve these limitations, we propose HPC, a novel streaming dynamic Gaussian Splatting compression framework. It employs a hierarchical point-based latent representation that operates on a per-Gaussian basis to avoid parameter redundancy in unoccupied space. Guided by a tailored aggregation scheme, these latent points achieve high compactness with low spatial redundancy. To improve compression efficiency, we further undertake the first investigation to compress neural networks for streaming dynamic Gaussian Splatting through mining and exploiting the inter-frame correlation of parameters. Combined with latent compression, this forms a fully end-to-end compression framework. Comprehensive experimental evaluations demonstrate that HPC substantially outperforms state-of-the-art methods. It achieves a storage reduction of 67% against its baseline while maintaining high reconstruction fidelity.
Read the full abstract ▾
All authors Yangzhi Ma, Bojun Liu, Wenting Liao, Dong Liu, Zhu Li, Li Li
Computer Vision
Improving Neuropathological Reconstruction Fidelity via AI Slice Imputation
Marina Crespo Aguirre, Jonathan Williams-Ramirez, Dina Zemlyanker et al. (+13 more)
Neuropathological analyses benefit from spatially precise volumetric reconstructions that enhance anatomical delineation and improve morphometric accuracy. Our prior work has shown the feasibility of reconstructing 3D
brain volumes from 2D dissection photographs. However these outputs sometimes exhibit coarse, overly smooth reconstructions of structures, especially under high anisotropy (i.e., reconstructions from thick slabs). Here, we introduce a computationally efficient super-resolution step that imputes slices to generate anatomically consistent isotropic volumes from anisotropic 3D reconstructions of dissection photographs. By training on domain-randomized synthetic data, we ensure that our method generalizes across dissection protocols and remains robust to large slab thicknesses. The imputed volumes yield improved automated
segmentations, achieving higher Dice scores, particularly in cortical and white matter regions. Validation on surface reconstruction and atlas registration tasks demonstrates more accurate cortical surfaces and MRI registration. By enhancing the resolution and anatomical fidelity of photograph-based reconstructions, our approach strengthens the bridge between neuropathology and neuroimaging. Our method is publicly available at https://surfer.nmr.mgh.harvard.edu/fswiki/mri_3d_photo_recon
Read the full abstract ▾
All authors Marina Crespo Aguirre, Jonathan Williams-Ramirez, Dina Zemlyanker, Xiaoling Hu, Lucas J. Deden-Binder, Rogeny Herisse, Mark Montine, Theresa R. Connors, Christopher Mount, Christine L. MacDonald, C. Dirk Keene, Caitlin S. Latimer, Derek H. Oakley, Bradley T. Hyman, Ana Lawry Aguila, Juan Eugenio Iglesias
Computer Vision
Schrödinger-Inspired Time-Evolution for 4D Deformation Forecasting
Ahsan Raza Siyal, Markus Haltmeier, Ruth Steiger et al. (+2 more)
Spatiotemporal
forecasting of complex three-dimensional phenomena (4D: 3D + time) is fundamental to applications in medical imaging, fluid and material dynamics, and geophysics. In contrast to unconstrained neural
forecasting models, we propose a Schrödinger-inspired, physics-guided neural architecture that embeds an explicit time-
evolution operator within a deep convolutional framework for 4D prediction. From observed volumetric sequences, the model learns voxelwise amplitude, phase, and potential fields that define a complex-valued wavefunction $\psi = A e^{i\phi}$, which is evolved forward in time using a differentiable, unrolled Schrödinger time stepper. This physics-guided formulation yields several key advantages: (i) temporal stability arising from the structured
evolution operator, which mitigates drift and error accumulation in long-horizon
forecasting; (ii) an interpretable latent representation, where phase encodes transport dynamics, amplitude captures structural intensity, and the learned potential governs spatiotemporal interactions; and (iii) natural compatibility with deformation-based synthesis, which is critical for preserving anatomical fidelity in medical imaging applications. By integrating physical priors directly into the learning process, the proposed approach combines the expressivity of deep networks with the robustness and interpretability of physics-based modeling. We demonstrate accurate and stable prediction of future 4D states, including volumetric intensities and deformation fields, on synthetic
benchmarks that emulate realistic shape deformations and topological changes. To our knowledge, this is the first end-to-end 4D neural
forecasting framework to incorporate a Schrödinger-type
evolution operator, offering a principled pathway toward interpretable, stable, and anatomically consistent spatiotemporal prediction.
Read the full abstract ▾
All authors Ahsan Raza Siyal, Markus Haltmeier, Ruth Steiger, Elke Ruth Gizewski, Astrid Ellen Grams
Computer Vision
Lukas Kuhn, Giuseppe Serra and Florian Buettner
Vision-
language models have transformed
multimodal representation learning, yet dominant contrastive approaches like CLIP require large batch sizes, careful negative sampling, and extensive hyperparameter tuning. We introduce NOVA, a NOn-contrastive
Vision-
language Alignment framework based on joint
embedding prediction with distributional regularization. NOVA aligns visual representations to a frozen, domain-specific text encoder by predicting text
embeddings from augmented image views, while enforcing an isotropic Gaussian structure via Sketched Isotropic Gaussian Regularization (SIGReg). This eliminates the need for negative sampling, momentum encoders, or stop-gradients, reducing the training objective to a single hyperparameter. We evaluate NOVA on zeroshot chest X-ray classification using ClinicalBERT as the text encoder and
Vision Transformers trained from scratch on MIMIC-CXR. On zero-shot classification across three
benchmark datasets, NOVA outperforms multiple standard baselines while exhibiting substantially more consistent training runs. Our results demonstrate that non-contrastive
vision-
language pretraining offers a simpler, more stable, and more effective alternative to contrastive methods.
Read the full abstract ▾
Computer Vision
A Hybrid Mamba-SAM Architecture for Efficient 3D Medical Image Segmentation
Mohammadreza Gholipour Shahraki, Mehdi Rezaeian and Mohammad Ghasemzadeh
Accurate
segmentation of 3D medical images such as MRI and CT is essential for clinical diagnosis and treatment planning. Foundation models like the Segment Anything Model (SAM) provide powerful general-purpose representations but struggle in medical imaging due to domain shift, their inherently 2D design, and the high computational cost of fine-tuning. To address these challenges, we propose Mamba-SAM, a novel and efficient hybrid architecture that combines a frozen SAM encoder with the linear-time efficiency and long-range modeling capabilities of Mamba-based State Space Models (SSMs). We investigate two parameter-efficient adaptation strategies. The first is a dual-branch architecture that explicitly fuses general features from a frozen SAM encoder with domain-specific representations learned by a trainable VMamba encoder using cross-
attention. The second is an adapter-based approach that injects lightweight, 3D-aware Tri-Plane Mamba (TPMamba) modules into the frozen SAM ViT encoder to implicitly model volumetric context. Within this framework, we introduce Multi-Frequency Gated Convolution (MFGC), which enhances feature representation by jointly analyzing spatial and frequency-domain information via 3D discrete cosine transforms and adaptive gating. Extensive experiments on the ACDC cardiac MRI
dataset demonstrate the effectiveness of the proposed methods. The dual-branch Mamba-SAM-Base model achieves a mean Dice score of 0.906, comparable to UNet++ (0.907), while outperforming all baselines on Myocardium (0.910) and Left Ventricle (0.971)
segmentation. The adapter-based TP MFGC variant offers superior inference speed (4.77 FPS) with strong accuracy (0.880 Dice). These results show that hybridizing foundation models with efficient SSM-based architectures provides a practical and effective solution for 3D medical image
segmentation.
Read the full abstract ▾
Computer Vision
Diff-PC: Identity-preserving and 3D-aware Controllable Diffusion for Zero-shot Portrait Customization
Yifang Xu, Benxiang Zhai, Chenyu Zhang et al. (+3 more)
Portrait customization (PC) has recently garnered significant
attention due to its potential applications. However, existing PC methods lack precise identity (ID) preservation and face control. To address these tissues, we propose Diff-PC, a
diffusion-based framework for zero-shot PC, which generates realistic portraits with high ID fidelity, specified facial attributes, and diverse backgrounds. Specifically, our approach employs the 3D face predictor to reconstruct the 3D-aware facial priors encompassing the reference ID, target expressions, and poses. To capture fine-grained face details, we design ID-Encoder that fuses local and global facial features. Subsequently, we devise ID-Ctrl using the 3D face to guide the
alignment of ID features. We further introduce ID-Injector to enhance ID fidelity and facial controllability. Finally, training on our collected ID-centric
dataset improves face similarity and text-to-image (T2I)
alignment. Extensive experiments demonstrate that Diff-PC surpasses state-of-the-art methods in ID preservation, facial control, and T2I consistency. Furthermore, our method is compatible with multi-style foundation models.
Read the full abstract ▾
All authors Yifang Xu, Benxiang Zhai, Chenyu Zhang, Ming Li, Yang Li, Sidan Du
Computer Vision
VIZOR: Viewpoint-Invariant Zero-Shot Scene Graph Generation for 3D Scene Reasoning
Vivek Madhavaram, Vartika Sengar, Arkadipta De et al. (+1 more)
Scene understanding and
reasoning has been a fundamental problem in 3D computer
vision, requiring models to identify objects, their properties, and spatial or comparative relationships among the objects. Existing approaches enable this by creating scene graphs using multiple inputs such as 2D images, depth maps, object labels, and annotated relationships from specific reference view. However, these methods often struggle with generalization and produce inaccurate spatial relationships like "left/right", which become inconsistent across different viewpoints. To address these limitations, we propose Viewpoint-Invariant Zero-shot scene graph generation for 3D scene
Reasoning (VIZOR). VIZOR is a training-free, end-to-end framework that constructs dense, viewpoint-invariant 3D scene graphs directly from raw 3D scenes. The generated scene graph is unambiguous, as spatial relationships are defined relative to each object's front-facing direction, making them consistent regardless of the reference view. Furthermore, it infers open-vocabulary relationships that describe spatial and proximity relationships among scene objects without requiring annotated training data. We conduct extensive quantitative and qualitative evaluations to assess the effectiveness of VIZOR in scene graph generation and downstream tasks, such as query-based object grounding. VIZOR outperforms state-of-the-art methods, showing clear improvements in scene graph generation and achieving 22% and 4.81% gains in zero-shot grounding accuracy on the Replica and Nr3D
datasets, respectively.
Read the full abstract ▾
All authors Vivek Madhavaram, Vartika Sengar, Arkadipta De, Charu Sharma
Computer Vision
S$^3$POT: Contrast-Driven Face Occlusion Segmentation via Self-Supervised Prompt Learning
Lingsong Wang, Mancheng Meng, Ziyan Wu et al. (+3 more)
Existing face parsing methods usually misclassify occlusions as facial components. This is because occlusion is a high-level concept, it does not refer to a concrete category of object. Thus, constructing a real-world face
dataset covering all categories of occlusion object is almost impossible and accurate mask annotation is labor-intensive. To deal with the problems, we present S$^3$POT, a contrast-driven framework synergizing face generation with self-supervised spatial prompting, to achieve occlusion
segmentation. The framework is inspired by the insights: 1) Modern face generators' ability to realistically reconstruct occluded regions, creating an image that preserve facial geometry while eliminating occlusion, and 2) Foundation
segmentation models' (e.g., SAM) capacity to extract precise mask when provided with appropriate
prompts. In particular, S$^3$POT consists of three modules: Reference Generation (RF), Feature enhancement (FE), and
Prompt Selection (PS). First, a reference image is produced by RF using structural guidance from parsed mask. Second, FE performs contrast of
tokens between raw and reference images to obtain an initial
prompt, then modifies image features with the
prompt by cross-
attention. Third, based on the enhanced features, PS constructs a set of positive and negative
prompts and screens them with a self-
attention network for a mask decoder. The network is learned under the guidance of three novel and complementary objective functions without occlusion ground truth mask involved. Extensive experiments on a dedicatedly collected
dataset demonstrate S$^3$POT's superior performance and the effectiveness of each module.
Read the full abstract ▾
All authors Lingsong Wang, Mancheng Meng, Ziyan Wu, Terrence Chen, Fan Yang, Dinggang Shen
Computer Vision
FaceSnap: Enhanced ID-fidelity Network for Tuning-free Portrait Customization
Benxiang Zhai, Yifang Xu, Guofeng Zhang et al. (+2 more)
Benefiting from the significant advancements in text-to-image
diffusion models, research in personalized image generation, particularly customized portrait generation, has also made great strides recently. However, existing methods either require time-consuming fine-tuning and lack generalizability or fail to achieve high fidelity in facial details. To address these issues, we propose FaceSnap, a novel method based on Stable
Diffusion (SD) that requires only a single reference image and produces extremely consistent results in a single inference stage. This method is plug-and-play and can be easily extended to different SD models. Specifically, we design a new Facial Attribute Mixer that can extract comprehensive fused information from both low-level specific features and high-level abstract features, providing better guidance for image generation. We also introduce a Landmark Predictor that maintains reference identity across landmarks with different poses, providing diverse yet detailed spatial control conditions for image generation. Then we use an ID-preserving module to inject these into the UNet. Experimental results demonstrate that our approach performs remarkably in personalized and customized portrait generation, surpassing other state-of-the-art methods in this domain.
Read the full abstract ▾
All authors Benxiang Zhai, Yifang Xu, Guofeng Zhang, Yang Li, Sidan Du
Computer Vision
Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steering
Guangtao Lyu, Xinyi Cheng, Qi Liu et al. (+5 more)
LVLMs achieve remarkable
multimodal understanding and generation but remain susceptible to
hallucinations. Existing mitigation methods predominantly focus on output-level adjustments, leaving the internal mechanisms that give rise to these
hallucinations largely unexplored. To gain a deeper understanding, we adopt a representation-level perspective by introducing sparse autoencoders (SAEs) to decompose dense visual
embeddings into sparse, interpretable
neurons. Through
neuron-level analysis, we identify distinct
neuron types, including always-on
neurons and image-specific
neurons. Our findings reveal that
hallucinations often result from disruptions or spurious activations of image-specific
neurons, while always-on
neurons remain largely stable. Moreover, selectively enhancing or suppressing image-specific
neurons enables controllable intervention in LVLM outputs, improving visual grounding and reducing
hallucinations. Building on these insights, we propose Contrastive
Neuron Steering (CNS), which identifies image-specific
neurons via contrastive analysis between clean and noisy inputs. CNS selectively amplifies informative
neurons while suppressing perturbation-induced activations, producing more robust and semantically grounded visual representations. This not only enhances visual understanding but also effectively mitigates
hallucinations. By operating at the prefilling stage, CNS is fully compatible with existing decoding-stage methods. Extensive experiments on both
hallucination-focused and general
multimodal benchmarks demonstrate that CNS consistently reduces
hallucinations while preserving overall
multimodal understanding.
Read the full abstract ▾
All authors Guangtao Lyu, Xinyi Cheng, Qi Liu, Chenghao Xu, Jiexi Yan, Muli Yang, Fen Fang, Cheng Deng
Computer Vision
Tune-Your-Style: Intensity-tunable 3D Style Transfer with Gaussian Splatting
Yian Zhao, Rushi Ye, Ruochong Zheng et al. (+6 more)
3D style transfer refers to the artistic stylization of 3D assets based on reference style images. Recently, 3DGS-based stylization methods have drawn considerable
attention, primarily due to their markedly enhanced training and rendering speeds. However, a vital challenge for 3D style transfer is to strike a balance between the content and the patterns and colors of the style. Although the existing methods strive to achieve relatively balanced outcomes, the fixed-output paradigm struggles to adapt to the diverse content-style balance requirements from different users. In this work, we introduce a creative intensity-tunable 3D style transfer paradigm, dubbed Tune-Your-Style, which allows users to flexibly adjust the style intensity injected into the scene to match their desired content-style balance, thus enhancing the customizability of 3D style transfer. To achieve this goal, we first introduce Gaussian
neurons to explicitly model the style intensity and parameterize a learnable style tuner to achieve intensity-tunable style injection. To facilitate the learning of tunable stylization, we further propose the tunable stylization guidance, which obtains multi-view consistent stylized views from
diffusion models through cross-view style
alignment, and then employs a two-stage optimization strategy to provide stable and efficient guidance by modulating the balance between full-style guidance from the stylized views and zero-style guidance from the initial rendering. Extensive experiments demonstrate that our method not only delivers visually appealing results, but also exhibits flexible customizability for 3D style transfer. Project page is available at https://zhao-yian.github.io/TuneStyle.
Read the full abstract ▾
All authors Yian Zhao, Rushi Ye, Ruochong Zheng, Zesen Cheng, Chaoran Feng, Jiashu Yang, Pengchong Qiao, Chang Liu, Jie Chen
Computer Vision
Pix2Fact: When Vision Is Not Enough – Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes
Yifan Jiang, Cong Zhang, Bofei Zhang et al. (+4 more)
Despite progress on general tasks,
vision-
language models (VLMs) still struggle with challenges that demand both fine-grained visual grounding and external knowledge, a synergy overlooked by existing
benchmarks that evaluate these abilities in isolation. To fill this void, we introduce Pix2Fact, a visual question-answering
benchmark designed to assess expert-level visual perception and knowledge search. Pix2Fact comprises 1,000 high-resolution (4K+) images spanning eight scenarios. Its questions and answers are meticulously crafted by PhD-holding annotators from top global universities across diverse disciplines. Each question requires detailed visual grounding and the integration of external knowledge. Evaluating ten state-of-the-art VLMs, including proprietary models such as Gemini-3.1-Pro and GPT-5.4, we find that Pix2Fact poses a formidable challenge: the most advanced model (Gemini-3.1-Pro) achieves only 51.7% average accuracy, even with access to visual ground truth and search tools. Our analysis attributes this low accuracy to three factors, frequent visual grounding errors even with visual ground truth, shallow search harnessing, and VLM's inability to retrieve long-tail, unstructured local information. This striking gap exposes the limitations of current models in assisting humans with real-world scenarios that demand overwhelming visual comprehension. We believe Pix2Fact will serve as a critical
benchmark to drive the next generation of
language-
vision agents that seamlessly integrate fine-grained perception with robust knowledge search.
Read the full abstract ▾
All authors Yifan Jiang, Cong Zhang, Bofei Zhang, Qiaofeng Zheng, Yifan Yang, Bingzhang Wang, Yew-Soon Ong
Computer Vision
MAUGen: A Unified Diffusion Approach for Multi-Identity Facial Expression and AU Label Generation
Xiangdong Li, Ye Lou, Ao Gao et al. (+2 more)
The lack of large-scale, demographically diverse face images with precise Action Unit (AU) occurrence and intensity annotations has long been recognized as a fundamental bottleneck in developing generalizable AU recognition systems. In this paper, we propose MAUGen, a
diffusion-based multi-modal framework that jointly generates a large collection of photorealistic facial expressions and anatomically consistent AU labels, including both occurrence and intensity, conditioned on a single descriptive text
prompt. Our MAUGen involves two key modules: (1) a Multi-modal Representation Learning (MRL) module that captures the relationships among the paired textual description, facial identity, expression image, and AU activations within a unified latent space; and (2) a
Diffusion-based Image label Generator (DIG) that decodes the joint representation into aligned facial image-label pairs across diverse identities. Under this framework, we introduce Multi-Identity Facial Action (MIFA), a large-scale
multimodal synthetic
dataset featuring comprehensive AU annotations and identity variations. Extensive experiments demonstrate that MAUGen outperforms existing methods in synthesizing photorealistic, demographically diverse facial images along with semantically aligned AU labels.
Read the full abstract ▾
All authors Xiangdong Li, Ye Lou, Ao Gao, Wei Zhang, Siyang Song
Computer Vision
Bridging Degradation Discrimination and Generation for Universal Image Restoration
JiaKui Hu, Zhengjian Yao, Lujia Jin et al. (+1 more)
Universal image restoration is a critical task in low-level
vision, requiring the model to remove various degradations from low-quality images to produce clean images with rich detail. The challenges lie in sampling the distribution of high-quality images and adjusting the outputs on the basis of the degradation. This paper presents a novel approach, Bridging Degradation discrimination and Generation (BDG), which aims to address these challenges concurrently. First, we propose the Multi-Angle and multi-Scale Gray Level Co-occurrence
Matrix (MAS-GLCM) and demonstrate its effectiveness in performing fine-grained discrimination of degradation types and levels. Subsequently, we divide the
diffusion training process into three distinct stages: generation, bridging, and restoration. The objective is to preserve the
diffusion model's capability of restoring rich textures while simultaneously integrating the discriminative information from the MAS-GLCM into the restoration process. This enhances its proficiency in addressing multi-task and multi-degraded scenarios. Without changing the architecture, BDG achieves significant performance gains in all-in-one restoration and real-world super-resolution tasks, primarily evidenced by substantial improvements in fidelity without compromising perceptual quality. The code and pretrained models are provided in https://github.com/MILab-PKU/BDG.
Read the full abstract ▾
All authors JiaKui Hu, Zhengjian Yao, Lujia Jin, Yanye Lu
Computer Vision · Cross-listed from cs.LG
When Classes Evolve: A Benchmark and Framework for Stage-Aware Class-Incremental Learning
Zheng Zhang, Tao Hu, Xueheng Li et al. (+4 more)
Class-Incremental Learning (CIL) aims to sequentially learn new classes while mitigating catastrophic forgetting of previously learned knowledge. Conventional CIL approaches implicitly assume that classes are morphologically static, focusing primarily on preserving previously learned representations as new classes are introduced. In practice, however, instances of the same semantic class may undergo substantial morphological
evolution, such as a larva turning into a butterfly. Consequently, a model must both discriminate between classes and adapt to evolving appearances within a single class. To systematically address this challenge, we formalize Stage-Aware CIL (Stage-CIL), a paradigm in which each class is learned progressively through distinct morphological stages. We further introduce Stage-Bench, a 10-domain, two-stage
benchmark and protocol for evaluating both inter-class forgetting and stage-level degradation within classes. Finally, we propose STAGE, an
evolution-aware reference baseline that disentangles semantic identity from
evolution dynamics through a fixed-size
memory pool, enabling stage-aware prediction of later morphological forms from earlier representations. Extensive experiments show that conventional CIL reductions and existing continual-learning baselines remain insufficient under Stage-CIL, while STAGE consistently outperforms strong competitors, demonstrating the promise of explicit
evolution-aware modeling for this new setting.
Read the full abstract ▾
All authors Zheng Zhang, Tao Hu, Xueheng Li, Yang Wang, Rui Li, Jie Zhang, Chengjun Xie
Computer Vision
GLAD: Generative Language-Assisted Visual Tracking for Low-Semantic Templates
Xingyu Luo, Yidong Cai, Jie Liu et al. (+3 more)
Vision-
language tracking has gained increasing
attention in many scenarios. This task simultaneously deals with visual and linguistic information to localize objects in
videos. Despite its growing utility, the development of
vision-
language tracking methods remains in its early stage. Current
vision-
language trackers usually employ
Transformer architectures for interactive integration of template, search, and text features. However, persistent challenges about low-semantic images including prevalent image blurriness, low resolution and so on, may compromise model performance through degraded cross-modal understanding. To solve this problem,
language assistance is usually used to deal with the obstacles posed by low-semantic images. However, due to the existing gap between current textual and visual features, direct concatenation and fusion of these features may have limited effectiveness. To address these challenges, we introduce a pioneering Generative
Language-AssisteD tracking model, GLAD, which utilizes
diffusion models for the generative multi-modal fusion of text description and template image to bolster compatibility between
language and image and enhance template image semantic information. Our approach demonstrates notable improvements over the existing fusion paradigms. Blurry and semantically ambiguous template images can be restored to improve multi-modal features in the generative fusion paradigm. Experiments show that our method establishes a new state-of-the-art on multiple
benchmarks and achieves an impressive inference speed. The code and models will be released at: https://github.com/Confetti-lxy/GLAD
Read the full abstract ▾
All authors Xingyu Luo, Yidong Cai, Jie Liu, Jie Tang, Gangshan Wu, Limin Wang
Computer Vision
Wenbin Xing, Quanxing Zha, Lizheng Zu et al. (+3 more)
Current research on
video hallucination mitigation primarily focuses on isolated error types, leaving compositional
hallucinations, arising from incorrect
reasoning over multiple interacting spatial and temporal factors largely underexplored. We introduce OmniVCHall, a
benchmark designed to systematically evaluate both isolated and compositional
hallucinations in
video multimodal large
language models (VLLMs). OmniVCHall spans diverse
video domains, introduces a novel camera-based
hallucination type, and defines a fine-grained taxonomy, together with
adversarial answer options (e.g., "All are correct" and "None of the above") to prevent shortcut
reasoning. The evaluations of 39 representative VLLMs reveal that even advanced models (e.g., Qwen3-VL and GPT-5) exhibit substantial performance degradation. We propose TriCD, a contrastive decoding framework with a triple-pathway calibration mechanism. An adaptive perturbation controller dynamically selects distracting operations to construct negative
video variants, while a saliency-guided enhancement module adaptively reinforces grounded
token-wise visual evidences. These components are optimized via reinforcement learning to encourage precise decision-making under compositional
hallucination settings. Experimental results show that TriCD consistently improves performance across two representative backbones, achieving an average accuracy improvement of over 10%. The data and code can be find at https://github.com/BMRETURN/OmniVCHall.
Read the full abstract ▾
All authors Wenbin Xing, Quanxing Zha, Lizheng Zu, Mengran Li, Ming Li, Junchi Yan
Computer Vision · Cross-listed from cs.RO
APEX: A Decoupled Memory-based Explorer for Asynchronous Aerial Object Goal Navigation
Daoxuan Zhang, Ping Chen, Xiaobo Xia et al. (+4 more)
Aerial Object Goal
Navigation, a challenging frontier in
Embodied AI, requires an Unmanned Aerial Vehicle (
UAV)
agent to autonomously explore, reason, and identify a specific target using only visual perception and
language description. However, existing methods struggle with the memorization of complex spatial representations in aerial environments, reliable and interpretable action decision-making, and inefficient exploration and information gathering. To address these challenges, we introduce APEX (Aerial Parallel Explorer), a novel hierarchical
agent designed for efficient exploration and target acquisition in complex aerial settings. APEX is built upon a modular, three-part architecture: 1) Dynamic Spatio-Semantic Mapping
Memory, which leverages the zero-shot capability of a
Vision-
Language Model (VLM) to dynamically construct high-resolution 3D Attraction, Exploration, and Obstacle maps, serving as an interpretable
memory mechanism. 2) Action Decision Module, trained with reinforcement learning, which translates this rich spatial understanding into a fine-grained and robust control policy. 3) Target Grounding Module, which employs an open-vocabulary detector to achieve definitive and generalizable target identification. All these components are integrated into a hierarchical, asynchronous, and parallel framework, effectively bypassing the VLM's inference latency and boosting the
agent's proactivity in exploration. Extensive experiments show that APEX outperforms the previous state of the art by +4.2% SR and +2.8% SPL on challenging
UAV-ON
benchmarks, demonstrating its superior efficiency and the effectiveness of its hierarchical asynchronous design. Our source code is provided in \href{https://github.com/4amGodvzx/apex}{GitHub}
Read the full abstract ▾
All authors Daoxuan Zhang, Ping Chen, Xiaobo Xia, Xiu Su, Ruichen Zhen, Jianqiang Xiao, Shuo Yang
Computer Vision
NPNet: A Non-Parametric Network with Adaptive Gaussian-Fourier Positional Encoding for 3D Classification and Segmentation
Mohammad Saeid, Amir Salarpour, Pedram MohajerAnsari et al. (+1 more)
We present NPNet, a fully non-parametric approach for 3D point-cloud classification and part
segmentation. NPNet contains no learned weights; instead, it builds point features using deterministic operators such as farthest point sampling, k-nearest neighbors, and pooling. Our key idea is an adaptive Gaussian-
Fourier positional encoding whose bandwidth and Gaussian-cosine mixing are chosen from the input geometry, helping the method remain stable across different scales and sampling densities. For
segmentation, we additionally incorporate fixed-frequency
Fourier features to provide global context alongside the adaptive encoding. Across ModelNet40/ModelNet-R, ScanObjectNN, and ShapeNetPart, NPNet achieves strong performance among non-parametric baselines, and it is particularly effective in few-shot settings on ModelNet40. NPNet also offers favorable
memory use and inference time compared to prior non-parametric methods
Read the full abstract ▾
All authors Mohammad Saeid, Amir Salarpour, Pedram MohajerAnsari, Mert D. Pesé
Computer Vision
SADER: Structure-Aware Diffusion Framework with DEterministic Resampling for Multi-Temporal Remote Sensing Cloud Removal
Yifan Zhang, Qian Chen, Yi Liu et al. (+2 more)
Cloud contamination severely degrades the usability of remote sensing imagery and poses a fundamental challenge for downstream Earth observation tasks. Recently,
diffusion-based models have emerged as a dominant paradigm for remote sensing cloud removal due to their strong generative capability and stable optimization. However, existing
diffusion-based approaches often suffer from limited sampling efficiency and insufficient exploitation of structural and temporal priors in multi-temporal remote sensing scenarios. In this work, we propose SADER, a structure-aware
diffusion framework for multi-temporal remote sensing cloud removal. SADER first develops a scalable Multi-Temporal Conditional
Diffusion Network (MTCDN) to fully capture multi-temporal and
multimodal correlations via temporal fusion and hybrid
attention. Then, a cloud-aware
attention loss is introduced to emphasize cloud-dominated regions by accounting for cloud thickness and brightness discrepancies. In addition, a deterministic resampling strategy is designed for continuous
diffusion models to iteratively refine samples under fixed sampling steps by replacing outliers through guided correction. Extensive experiments on multiple multi-temporal
datasets demonstrate that SADER consistently outperforms state-of-the-art cloud removal methods across all evaluation metrics. The code of SADER is publicly available at https://github.com/zyfzs0/SADER.
Read the full abstract ▾
All authors Yifan Zhang, Qian Chen, Yi Liu, Wengen Li, Jihong Guan
Computer Vision
Enhancing Open-Vocabulary Object Detection through Multi-Level Fine-Grained Visual-Language Alignment
Tianyi Zhang, Antoine Simoulin, Kai Li et al. (+5 more)
Traditional object detection systems are typically constrained to predefined categories, limiting their applicability in dynamic environments. In contrast, open-vocabulary object detection (OVD) enables the identification of objects from novel classes not present in the training set. Recent advances in visual-
language modeling have led to significant progress of OVD. However, prior works face challenges in either adapting the single-scale image backbone from CLIP to the detection framework or ensuring robust visual-
language alignment. We propose Visual-
Language Detection (VLDet), a novel framework that revamps feature pyramid for fine-grained visual-
language alignment, leading to improved OVD performance. With the VL-PUB module, VLDet effectively exploits the visual-
language knowledge from CLIP and adapts the backbone for object detection through feature pyramid. In addition, we introduce the SigRPN block, which incorporates a sigmoid-based anchor-text contrastive
alignment loss to improve detection of novel categories. Through extensive experiments, our approach achieves 58.7 AP for novel classes on COCO2017 and 24.8 AP on LVIS, surpassing all state-of-the-art methods and achieving significant improvements of 27.6% and 6.9%, respectively. Furthermore, VLDet also demonstrates superior zero-shot performance on closed-set object detection.
Read the full abstract ▾
All authors Tianyi Zhang, Antoine Simoulin, Kai Li, Sana Lakdawala, Shiqing Yu, Arpit Mittal, Hongyu Fu, Yu Lin
Computer Vision
SAGE: Accelerating Vision-Language Models via Entropy-Guided Adaptive Speculative Decoding
Yujia Tong, Tian Zhang, Yunyang Wan et al. (+3 more)
Speculative decoding has emerged as a promising approach to accelerate inference in
vision-
language models (VLMs) by enabling parallel verification of multiple draft
tokens. However, existing methods rely on static tree structures that remain fixed throughout the decoding process, failing to adapt to the varying prediction difficulty across generation steps. This leads to suboptimal acceptance lengths and limited speedup. In this paper, we propose SAGE, a novel framework that dynamically adjusts the speculation tree structure based on real-time prediction uncertainty. Our key insight is that output
entropy serves as a natural confidence indicator with strong temporal correlation across decoding steps. SAGE constructs deeper-narrower trees for high-confidence predictions to maximize speculation depth, and shallower-wider trees for uncertain predictions to diversify exploration. SAGE improves acceptance lengths and achieves faster acceleration compared to static tree baselines. Experiments on multiple
benchmarks demonstrate the effectiveness of SAGE: without any loss in output quality, it delivers up to $3.36\times$ decoding speedup for LLaVA-OneVision-72B and $3.18\times$ for Qwen2.5-VL-72B.
Read the full abstract ▾
All authors Yujia Tong, Tian Zhang, Yunyang Wan, Kaiwei Lin, Jingling Yuan, Chuang Hu
Computer Vision
MRAD: Zero-Shot Anomaly Detection with Memory-Driven Retrieval
Chaoran Xu, Chengkan Lv, Qiyu Chen et al. (+2 more)
Zero-shot anomaly detection (ZSAD) often leverages pretrained
vision or
vision-
language models, but many existing methods use
prompt learning or complex modeling to fit the data distribution, resulting in high training or inference cost and limited cross-domain stability. To address these limitations, we propose
Memory-
Retrieval Anomaly Detection method (MRAD), a unified framework that replaces parametric fitting with a direct
memory retrieval. The train-free base model, MRAD-TF, freezes the CLIP image encoder and constructs a two-level
memory bank (image-level and pixel-level) from auxiliary data, where feature-label pairs are explicitly stored as keys and values. During inference, anomaly scores are obtained directly by similarity
retrieval over the
memory bank. Based on the MRAD-TF, we further propose two lightweight variants as enhancements: (i) MRAD-FT fine-tunes the
retrieval metric with two linear layers to enhance the discriminability between normal and anomaly; (ii) MRAD-CLIP injects the normal and anomalous region priors from the MRAD-FT as dynamic biases into CLIP's learnable text
prompts, strengthening generalization to unseen categories. Across 16 industrial and medical
datasets, the MRAD framework consistently demonstrates superior performance in anomaly classification and
segmentation, under both train-free and training-based settings. Our work shows that fully leveraging the empirical distribution of raw data, rather than relying only on model fitting, can achieve stronger anomaly detection performance. The code will be publicly released at https://github.com/CROVO1026/MRAD.
Read the full abstract ▾
All authors Chaoran Xu, Chengkan Lv, Qiyu Chen, Feng Zhang, Zhengtao Zhang
Computer Vision
SPARK: Stochastic Propagation via Affinity-guided Random walK for training-free unsupervised segmentation
Kunal Mahatha, Jose Dolz and Christian Desrosiers
We argue that existing training-free
segmentation methods rely on an implicit and limiting assumption, that
segmentation is a spectral graph partitioning problem over
diffusion-derived affinities. Such approaches, based on global graph partitioning and eigenvector-based formulations of affinity
matrices, suffer from several fundamental drawbacks, they require pre-selecting the number of clusters, induce boundary oversmoothing due to spectral relaxation, and remain highly sensitive to noisy or multi-modal affinity distributions. Moreover, many prior works neglect the importance of local neighborhood structure, which plays a crucial role in stabilizing affinity propagation and preserving fine-grained contours. To address these limitations, we reformulate training-free
segmentation as a
stochastic flow
equilibrium problem over
diffusion-induced affinity graphs, where
segmentation emerges from a
stochastic propagation process that integrates global
diffusion attention with local neighborhoods extracted from stable
diffusion, yielding a sparse yet expressive affinity structure. Building on this formulation, we introduce a Markov propagation scheme that performs random-walk-based label
diffusion with an adaptive
pruning strategy that suppresses unreliable transitions while reinforcing confident affinity paths. Experiments across seven widely used semantic
segmentation benchmarks demonstrate that our method achieves state-of-the-art zero-shot performance, producing sharper boundaries, more coherent regions, and significantly more stable masks compared to prior spectral-clustering-based approaches.
Read the full abstract ▾
Computer Vision
DuoGen: Towards General Purpose Interleaved Multimodal Generation
Min Shi, Xiaohui Zeng, Jiannan Huang et al. (+13 more)
Interleaved
multimodal generation enables capabilities beyond unimodal generation models, such as step-by-step instructional guides, visual planning, and generating visual drafts for
reasoning. However, the quality of existing interleaved generation models under general instructions remains limited by insufficient training data and base model capacity. We present DuoGen, a general-purpose interleaved generation framework that systematically addresses data curation, architecture design, and evaluation. On the data side, we build a large-scale, high-quality instruction-tuning
dataset by combining
multimodal conversations rewritten from curated raw websites, and diverse synthetic examples covering everyday scenarios. Architecturally, DuoGen leverages the strong visual understanding of a pretrained
multimodal LLM and the visual generation capabilities of a
diffusion transformer (DiT) pretrained on
video generation, avoiding costly unimodal pretraining and enabling flexible base model selection. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences. Across public and newly proposed
benchmarks, DuoGen outperforms prior open-source models in text quality, image fidelity, and image-context
alignment, and also achieves state-of-the-art performance on text-to-image and image editing among unified generation models. Data and code will be released at https://research.nvidia.com/labs/dir/duogen/.
Read the full abstract ▾
All authors Min Shi, Xiaohui Zeng, Jiannan Huang, Yin Cui, Francesco Ferroni, Jialuo Li, Shubham Pachori, Zhaoshuo Li, Yogesh Balaji, Haoxiang Wang, Tsung-Yi Lin, Xiao Fu, Yue Zhao, Chieh-Yun Chen, Ming-Yu Liu, Humphrey Shi
Computer Vision
Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models
Jingrui Zhang, Feng Liang, Yong Zhang et al. (+3 more)
With the remarkable success of large
language models (LLMs) in natural
language understanding and generation,
multimodal large
language models (MLLMs) have rapidly advanced in their ability to process data across multiple modalities. While most existing efforts focus on scaling up
language models or constructing higher-quality training data, limited
attention has been paid to effectively integrating cross-modal knowledge into the
language space. In
vision-
language models, for instance, aligning modalities using only high-level visual features often discards the rich semantic information present in mid- and low-level features, limiting the model's ability of cross-modality understanding. To address this issue, we propose SparseCut, a general cross-modal fusion architecture for MLLMs, introducing sparse shortcut connections between the cross-modal encoder and the LLM. These shortcut connections enable the efficient and hierarchical integration of visual features at multiple levels, facilitating richer semantic fusion without increasing computational overhead. We further introduce an efficient multi-grained feature fusion module, which performs the fusion of visual features before routing them through the shortcuts. This preserves the original
language context and does not increase the overall input length, thereby avoiding an increase in computational complexity for the LLM. Experiments demonstrate that SparseCut significantly enhances the performance of MLLMs across various
multimodal benchmarks with generality and scalability for different base LLMs.
Read the full abstract ▾
All authors Jingrui Zhang, Feng Liang, Yong Zhang, Wei Wang, Runhao Zeng, Xiping Hu
Computer Vision
RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding
Jiahe Wu, Bing Cao, Qilong Wang et al. (+3 more)
Multimodal Large
Language Models (MLLM) are primarily pre-trained on the RGB modality, thereby limiting their performance on other modalities, such as infrared, depth, and event data, which are crucial for complex scenarios. To address this, we propose RGBX-R1, a framework to enhance MLLM's perception and
reasoning capacities across various X visual modalities. Specifically, we employ an Understand-Associate-Validate (
UAV) prompting strategy to construct the Visual Modality Chain-of-Thought (VM-CoT), which aims to expand the MLLMs' RGB understanding capability into X modalities. To progressively enhance
reasoning capabilities, we introduce a two-stage training paradigm: Cold-Start Supervised Fine-Tuning (CS-SFT) and Spatio-Temporal Reinforcement Fine-Tuning (ST-RFT). CS-SFT supervises the
reasoning process with the guidance of VM-CoT, equipping the MLLM with fundamental modality cognition. Building upon GRPO, ST-RFT employs a Modality-understanding Spatio-Temporal (MuST) reward to reinforce modality
reasoning. Notably, we construct the first RGBX-Grounding
benchmark, and extensive experiments verify our superiority in
multimodal understanding and spatial perception, outperforming baselines by 22.71% on three RGBX grounding tasks.
Read the full abstract ▾
All authors Jiahe Wu, Bing Cao, Qilong Wang, Qinghua Hu, Dongdong Li, Pengfei Zhu
Computer Vision
HSSDCT: Factorized Spatial-Spectral Correlation for Hyperspectral Image Fusion
Chia-Ming Lee, Yu-Hao Ho, Yu-Fan Lin et al. (+3 more)
Hyperspectral image (HSI) fusion aims to reconstruct a high-resolution HSI (HR-HSI) by combining the rich spectral information of a low-resolution HSI (LR-HSI) with the fine spatial details of a high-resolution multispectral image (HR-MSI). Although recent deep learning methods have achieved notable progress, they still suffer from limited receptive fields, redundant spectral bands, and the quadratic complexity of self-
attention, which restrict both efficiency and robustness. To overcome these challenges, we propose the Hierarchical Spatial-Spectral Dense Correlation Network (HSSDCT). The framework introduces two key modules: (i) a Hierarchical Dense-Residue
Transformer Block (HDRTB) that progressively enlarges windows and employs dense-residue connections for multi-scale feature aggregation, and (ii) a Spatial-Spectral Correlation Layer (SSCL) that explicitly factorizes spatial and spectral dependencies, reducing self-
attention to linear complexity while mitigating spectral redundancy. Extensive experiments on
benchmark datasets demonstrate that HSSDCT delivers superior reconstruction quality with significantly lower computational costs, achieving new state-of-the-art performance in HSI fusion. Our code is available at https://github.com/jemmyleee/HSSDCT.
Read the full abstract ▾
All authors Chia-Ming Lee, Yu-Hao Ho, Yu-Fan Lin, Jen-Wei Lee, Li-Wei Kang, Chih-Chung Hsu
Computer Vision
Refining Strokes by Learning Offset Attributes between Strokes for Flexible Sketch Edit at Stroke-Level
Sicong Zang, Tao Sun and Cairong Yan
Sketch edit at stroke-level aims to transplant source strokes onto a target sketch via stroke expansion or replacement, while preserving semantic consistency and visual fidelity with the target sketch. Recent studies addressed it by relocating source strokes at appropriate canvas positions. However, as source strokes could exhibit significant variations in both size and orientation, we may fail to produce plausible sketch editing results by merely repositioning them without further adjustments. For example, anchoring an oversized source stroke onto the target without proper scaling would fail to produce a semantically coherent outcome. In this paper, we propose SketchMod to refine the source stroke through transformation so as to align it with the target sketch's patterns, further realize flexible sketch edit at stroke-level. As the source stroke refinement is governed by the patterns of the target sketch, we learn three key offset attributes (scale, orientation and position) from the source stroke to another, and align it with the target by: 1) resizing to match spatial proportions by scale, 2) rotating to align with local geometry by orientation, and 3) displacing to meet with semantic layout by position. Besides, a stroke's profiles can be precisely controlled during sketch edit via the exposed captured stroke attributes. Experimental results indicate that SketchMod achieves precise and flexible performances on stroke-level sketch edit.
Read the full abstract ▾
Computer Vision
GTATrack: Winner Solution to SoccerTrack 2025 with Deep-EIoU and Global Tracklet Association
Rong-Lin Jian, Ming-Chi Luo, Chen-Wei Huang et al. (+3 more)
Multi-object tracking (MOT) in sports is highly challenging due to irregular player motion, uniform appearances, and frequent occlusions. These difficulties are further exacerbated by the
geometric distortion and extreme scale variation introduced by static fisheye cameras. In this work, we present GTATrack, a hierarchical tracking framework that win first place in the SoccerTrack Challenge 2025. GTATrack integrates two core components: Deep Expansion IoU (Deep-EIoU) for motion-agnostic online association and Global Tracklet Association (GTA) for trajectory-level refinement. This two-stage design enables both robust short-term matching and long-term identity consistency. Additionally, a pseudo-labeling strategy is used to boost detector recall on small and distorted targets. The synergy between local association and global
reasoning effectively addresses identity switches, occlusions, and tracking fragmentation. Our method achieved a winning HOTA score of 0.60 and significantly reduced false positives to 982, demonstrating state-of-the-art accuracy in fisheye-based soccer tracking. Our code is available at https://github.com/ron941/GTATrack-STC2025.
Read the full abstract ▾
All authors Rong-Lin Jian, Ming-Chi Luo, Chen-Wei Huang, Chia-Ming Lee, Yu-Fan Lin, Chih-Chung Hsu