Computer Vision · Cross-listed from cs.MM
A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision-Language Models
Iosif Tsangko, Andreas Triantafyllopoulos, George Margetis et al. (+2 more)
Blind and low-
vision (BLV) audiences remain underserved by visual art descriptions, particularly across
languages and in museum settings where
privacy and intellectual-property constraints may favour small on-premise
vision-
language models (VLMs). This pilot study investigates curator-guided
multilingual art description with Qwen2.5-VL-3B-Instruct for German, Romanian, and Serbian. We construct a parallel BLV-oriented caption corpus from artwork images and metadata, and compare
language-specific LoRA adapters with a single
multilingual adapter under a fixed backbone and training budget. Evaluation combines automatic lexical and
embedding-based metrics with an LLM-as-Judge protocol calibrated against a small Romanian BLV pilot study. Under our pilot setup,
language-specific adapters show more stable controllability and visually grounded description quality for Romanian and Serbian, while
multilingual adaptation remains competitive in German. We frame these findings as deployment-oriented evidence for small on-premise VLMs, and highlight the need for larger BLV user studies and broader
language coverage before drawing general conclusions about
multilingual accessibility.
Read the full abstract ▾
All authors Iosif Tsangko, Andreas Triantafyllopoulos, George Margetis, Ioana Crihana, Björn W. Schuller
Computer Vision
Tao Zou, Yichen He, Tian Qiu et al. (+2 more)
Long-term
memory is essential for
multimodal agents to build coherent experience, accumulate world knowledge, and achieve continual learning. However, constructing effective
memory goes beyond
memory module design and basic requirements such as accuracy and fidelity; the key challenge lies in determining what to memorize.
Multimodal agents, such as
embodied agents, continuously perceive, reason, and act in real or virtual environments, receiving an unbounded stream of
multimodal observations. From this combinatorial explosion of information, an
agent must selectively retain content that is relevant to its role in the environment and valuable for future tasks. To bridge this gap, we frame
memory generation as a learnable memorization policy and introduce TaskMem (Task-focused Memorization Policy Learning), a reinforcement-learning-based framework that enables the policy to dynamically adjust its focus to the demands of real tasks encountered in the environment. TaskMem adopts a two-phase training paradigm: Phase One learns how to memorize by optimizing
memory quality under fundamental fidelity requirements; Phase Two occurs after deployment, where the
agent learns what to memorize by tuning an adapter on its base MLLM, using recent environment tasks to define a reward model that guides the memorization policy toward task-relevant content. To evaluate our approach, we reformulate VideoMME, EgoLife, and EgoTempo into streaming
benchmarks that simulate a realistic setting in which an
agent processes streaming observations and handles tasks arriving online. To isolate
memory assessment, the questions must be answered using only the
agent's
memory, without access to raw
video. Built on Qwen3-VL-30B-A3B, TaskMem improves VQA accuracy by 6.3%, 7.0%, and 5.3% on these
benchmarks, respectively.
Read the full abstract ▾
All authors Tao Zou, Yichen He, Tian Qiu, Yuan Lin, Hang Li
Computer Vision
Towards Effective Long-Video Event Prediction via Multi-Level Event Semantics Mining
Bo Peng, YuanJie Lyu, PengGang Qin et al. (+1 more)
Accurately predicting future events is fundamental to content understanding and decision-making across various domains. While prior research has primarily focused on text or short-
video scenarios, long-
video event prediction, characterized by vast
multimodal context and more complex narratives, remains underexplored. Meanwhile, although recent Long-
Video Language Models (LVLMs), built on Large
Language Models (LLMs) and
Vision-
Language Models (VLMs), have shown promise in long-
video question answering and summarization, they struggle to generalize to event prediction, as they can neither precisely extract event-related details nor perform fine-grained analysis of event development. To address this gap, we propose VISTA, a multi-level event semantics mining framework for long-
video event prediction. Initially, VISTA applies a character-centric visual
prompt to precisely extract event-related visual details, enhancing detail-level semantics; subsequently, it employs a knowledge-enhanced iterative
retrieval strategy, guiding the LLM to progressively construct logically coherent event chains, thereby improving event-level narratives; ultimately, VISTA adopts a human-like propose-then-retrieve strategy to generate diverse future-oriented proposals and integrate multi-level clues, producing robust and accurate predictions. Extensive experiments on real-world
datasets validate the effectiveness of VISTA for long-
video event prediction.
Read the full abstract ▾
All authors Bo Peng, YuanJie Lyu, PengGang Qin, Tong Xu
Computer Vision
HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learning
Md Aminur Hossain, Ayush V. Patel, Sanjay K. Singh et al. (+1 more)
We introduce HQ-JEPA, a hybrid
quantum-classical joint-
embedding predictive architecture for cross-modal remote sensing representation learning. The proposed framework extends JEPA-style masked latent prediction to paired Sentinel-1 and Sentinel-2 imagery by predicting masked target representations from visible context regions while aligning heterogeneous modality features in a shared
embedding space. To improve representation quality, HQ-JEPA combines four complementary objectives: latent
token prediction, cross-modal
token alignment, SIGReg-based Gaussian regularization in the fused latent space, and a differentiable SWAP-test-based Fidelity
Quantum Similarity (FQS) loss. Unlike pixel reconstruction methods, HQ-JEPA learns semantic representations directly in latent space and uses
quantum state-overlap-based similarity as an additional regularization signal. We evaluate the pretrained encoder on GeoBench classification and
segmentation tasks under linear probing and fine-tuning settings. Results show that HQ-JEPA achieves competitive and often superior performance over strong self-supervised and remote sensing foundation-model baselines, demonstrating the benefit of integrating predictive self-supervision, cross-modal
geometric regularization, and
quantum fidelity-based representation learning for remote sensing applications.
Read the full abstract ▾
All authors Md Aminur Hossain, Ayush V. Patel, Sanjay K. Singh, Biplab Banerjee
Computer Vision
Gael Glorian, Ioannis Lamprou, Zhen Zhang et al. (+2 more)
Dense self-
attention is the compute and quality bottleneck of long-
video diffusion inference: cost grows quadratically with the sequence length, and beyond the training horizon the model converges to near-static output, that is, "frozen" repetitive
video. State of the art approaches are either too costly, e.g., they require retraining, or fail to satisfy both performance and quality objectives in a scalable manner. To this end, we introduce Long
Video Sparse
Attention (LVSA), a training-free model-agnostic block-sparse
attention for
video diffusion transformers that combines a structured window pattern with rotating global anchors, thus removing the fixed-grid bias which causes long-range temporal artifacts. LVSA, combined with a FlashInfer kernel, reduces compute up to 3.17x on Wan 2.1 1.3B at a 6x horizon, 2.98x on Wan 2.1 14B at a 6x horizon, and 3.33x on HunyuanVideo 1.5 at a 1.5x horizon, compared to dense
attention. Beyond reducing compute, LVSA enables HunyuanVideo 1.5 generation at a 2x horizon, which is otherwise out-of-
memory on a single GPU. Moreover, LVSA provides speedups up to 2.41x compared to RIFLEx and 3.27x compared to UltraViCo on Wan 2.1 1.3B. To demonstrate applicability across diverse platforms, we apply LVSA on NPUs and achieve speedups up to 2.71x on Wan 2.2 A14B and 3.24x on Wan 2.1 1.3B compared to dense
attention. To evaluate quality in a fair way, we introduce VQeval, a tool properly scoring loopy
video failures, which instead are rewarded in state of the art evaluators like VBench-Long. LVSA is quality-neutral for generation at training horizon length and quality-positive at extended lengths.
Read the full abstract ▾
All authors Gael Glorian, Ioannis Lamprou, Zhen Zhang, Yujie Yuan, Hongsheng Liu
Computer Vision
Rethinking Efficient Crack Segmentation with Task-Aligned Structural-Directional Modeling
Shipeng Liu, Liang Zhao, Dengfeng Chen et al. (+1 more)
Recent crack
segmentation methods often follow generic semantic
segmentation designs, using stronger backbones, hybrid CNN-
Transformer-Mamba encoders, and auxiliary enhancement branches. Although effective, this raises whether stronger generic feature mixing is the most suitable direction for crack
segmentation. We instead formulate crack
segmentation as sparse structural recovery. Cracks have limited category-level semantics but strong morphological regularities, being thin, sparse, anisotropic, locally fragmented, and easily confused with textures or shadows. Thus, the key bottleneck lies in preserving weak structural evidence, recovering directional continuity, and suppressing background coupling. We propose RIFT, a compact family of morphology-aligned crack
segmentation models. Rather than compressing a complex generic architecture, RIFT is simple by design, preserving local evidence, aggregating cooperative directional continuity, and restoring crack structures through lightweight multi-scale fusion. Experiments on four public
benchmarks show that RIFT achieves the best or tied-best results across the 16 main metrics against reproduced representative baselines. RIFT-B gives the strongest overall accuracy, while RIFT-T provides the best deployment efficiency with only 0.47M parameters and high inference speed.
Topology-aware evaluation, ablations, transfer experiments, and visualizations further verify that task-aligned simplicity can match or surpass complex hybrid architectures when its inductive bias fits crack morphology. Code: https://github.com/xauat-liushipeng/RIFT
Read the full abstract ▾
All authors Shipeng Liu, Liang Zhao, Dengfeng Chen, Weihua Zhang
Computer Vision
Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?
Jingtao He, Hongliang Lu, Xiaoyun Qiu et al. (+2 more)
Vision-
Language-Action (VLA) models have demonstrated promising capability in autonomous driving, highlighting the potential of unified
multimodal architectures for jointly modeling perception and planning. However, how current VLA-based driving behavior is grounded in visual information remains poorly understood. Existing evaluation protocols mainly focus on aggregate performance metrics, lacking structured and practical diagnostics to quantify visual-behavior dependency. In this work, we introduce a structured multi-level visual perturbation framework to analyze visual-behavior dependency in VLA-based driving models systematically. The framework organizes controlled visual perturbations along three complementary dimensions: channellevel degradation, information-level disruption, and structurelevel modification. We apply it to VLA-based driving systems and evaluate behavioral responses under both open-loop trajectory prediction and interactive closed-loop safety evaluation. Experimental results reveal evaluation-dependent dependency patterns and uneven visual grounding across abstraction levels. These findings call for more structured analyses and principled design of VLA driving models to better understand how visual information shapes behavior and develop safer, more robust systems.
Read the full abstract ▾
All authors Jingtao He, Hongliang Lu, Xiaoyun Qiu, Yixuan Wang, Xinhu Zheng
Computer Vision
GGT-100K: Generative Ground Truth for Generalizable Real-World Image Restoration
Xiangtao Kong, Jixin Zhao, Lingchen Sun et al. (+2 more)
Real-world image restoration (IR) is bottlenecked by the scarcity of high-quality paired training data. Synthetic
datasets are abundant but often fail to model real-world degradations, while real-world paired
datasets are expensive and difficult to capture. As a result, IR models trained on these
datasets show limited generalization in real-world scenarios. In this work, we propose Generative Ground Truth (GGT) by using generative
multimodal foundation models (MFMs) to produce high-quality (HQ) targets from real-world low-quality (LQ) images. We first conduct a systematic evaluation of nine state-of-the-art MFMs, including Nano-Banana-2 and GPT-Image-2, on images of various scenes and degradation types. The results demonstrate that Nano-Banana-2 with VLM-based adaptive prompting shows the highest capability to synthesize perceptually realistic and content-faithful HQ targets, which can serve as the GGT for the LQ input. We then employ Nano-Banana-2 to build a GGT synthesis pipeline, which involves multi-stage quality control to ensure data reliability, and construct GGT-100K, an LQ-HQ paired
dataset comprising 103,707 training pairs and covering diverse scenes and complex real-world degradations. A test set of 500 image pairs is also established. Extensive experiments show that GGT-100K consistently improves the real-world generalization of a wide range of IR models, with particularly strong benefits for finetuning generative models for IR tasks. Our results suggest that MFMs can serve as practical tools for restoration-oriented data generation, and GGT-100K is a useful resource to expand the generalization boundaries of real-world IR models.
Read the full abstract ▾
All authors Xiangtao Kong, Jixin Zhao, Lingchen Sun, Rongyuan Wu, Lei Zhang
Computer Vision
SlotMemory: Object-Centric KV Memory for Streaming Long-Video Generation
Weijia Dou, Hui Li, Jiahao Cui et al. (+3 more)
Streaming
video generation models typically rely on temporal-centric
memory, organizing historical context as latent queues, compressed context
tokens, or unclustered
token sets. This often causes identity drift and semantic inconsistency when entities exit the frame or during interactive
prompt transitions. To address these limitations, we propose SlotMemory, an object-centric Key-Value (KV)
memory mechanism for streaming
video diffusion. Our approach shifts
memory abstraction from “when” an event occurred to “what” is represented by decomposing the
transformer's KV
manifold into discrete, reusable semantic slots. Because the slots serve only as routing addresses and the stored content remains the original
transformer KV
tokens, the mechanism adds entity-level persistence and
prompt-aware
retrieval across long horizons. Evaluated on 60-second interactive narratives using the Wan2.1-T2V-1.3B backbone, SlotMemory achieves the highest Dynamic Degree among evaluated streaming methods and the second-highest aggregate Quality Score, preserving
prompt adherence rather than trading it away for consistency. At the 30-second horizon, SlotMemory leads in Total, Quality, and Dynamic Degree, improving Dynamic Degree by 31.2% over MemFlow, the strongest
memory-based baseline. Our results support treating structured semantic representation, rather than raw temporal capacity, as a critical primitive for persistent long-form
video synthesis.
Read the full abstract ▾
All authors Weijia Dou, Hui Li, Jiahao Cui, Lei Zhou, Jingdong Wang, Siyu Zhu
Computer Vision
PEEK: Picking Essential frames via Efficient Knowledge distillation
Killian Steunou, Anas Filali Razzouki, Khalil Guetari et al. (+2 more)
Video-
language models can process only a limited number of frames, making frame selection a key bottleneck for efficient
video captioning. Most captioning pipelines still rely on uniform sampling, which is computationally cheap but agnostic to visual content. Adaptive frame sampling selects the most informative frames from a
video, but existing methods remain computationally expensive. We introduce PEEK, an efficient dynamic frame sampling method that distills caption-conditioned frame relevance rankings from a stronger teacher model into a lightweight temporal model that operates only on visual content. On ActivityNet Captions and MSR-VTT, PEEK outperforms state-of-the-art methods across all evaluated downstream
vision language models, obtaining the best CIDEr for most frame budgets, especially when only one or two frames are selected, winning 14 out of 16 configurations on ActivityNet Captions. Zero-shot on MSR-VTT, it transfers best at low frame budgets, while results at four and eight frames are more mixed as temporal coverage and visual diversity become increasingly competitive. Compared with recent adaptive baselines, PEEK is both more accurate in the low-budget regime and more efficient: it adds only $5.2\%$ to the captioning time, compared with $65.4\%$ for CSTA and $211.9\%$ for MaxInfo. We release our code and pre-trained checkpoint at https://github.com/momentslab/peek.
Read the full abstract ▾
All authors Killian Steunou, Anas Filali Razzouki, Khalil Guetari, Mounîm A. El-Yacoubi, Yannis Tevissen
Computer Vision
Iterative Framework For Data Augmentation Of Segmented Fingerprints
João Leonardo H. D. Agnol, Wesley Augusto de Bona, Erick Oliveira Rodrigues et al. (+4 more)
Infant biometrics presents unique challenges due to the physiological differences between infants and adults, compounded by the scarcity of available data for research that limits the development of robust matching systems. This paper proposes a novel data augmentation method that uses iterative techniques to generate diverse variants of segmented fingerprints by inducing errors in a convolutional neural network trained to extract fingerprint ridges and valleys. Experiments on real infant fingerprints demonstrate the method's effectiveness in expanding fingerprint variability, with augmentations exhibiting significant fluctuations in minutiae counts while still retaining visual similarity to the originals. The study also highlights the method's customizable nature for applying varying levels of changes to fingerprint
segmentations. Future research includes training
segmentation and matching neural networks using
datasets augmented by the proposed framework.
Read the full abstract ▾
All authors João Leonardo H. D. Agnol, Wesley Augusto de Bona, Erick Oliveira Rodrigues, Luiz Fernando Puttow Southier, Jefferson Oliva, Marcelo Filipak, Dalcimar Casanova
Computer Vision · Cross-listed from cs.LG
Parallel Tempering Initial Sampling in Inference-Time Reward Alignment
Myeongjun Oh, Gwangho Kim and Sungyoon Lee
Inference-time reward
alignment steers pretrained
diffusion and flow-based generative models to satisfy user-specified rewards without retraining. Recently, Sequential Monte Carlo (SMC) has emerged as a powerful framework for this task by iteratively filtering and propagating multiple particles. However, we show that standard SMC-based methods often suffer from poor performance because they initialize particles from a standard prior, whereas high-reward regions in complex reward landscapes are extremely rare. Further, we show that even recent reward-aware initial sampling approaches remain vulnerable to getting trapped in local modes, as complex reward landscapes are often multi-modal. To overcome these limitations, we propose PATHS (PArallel Tempering for High-complexity reward Sampling), a novel initialization method that couples multiple sampling chains through parallel tempering. PATHS maintains a ladder of reward-tempered chains and periodically performs Metropolis swaps, enabling efficient exploration across flattened reward landscapes, thereby mitigating the mode-trapping issues. Our analysis reveals that this mechanism substantially enhances the finite-budget exploration of rare, high-reward regions that are typically challenging to sample. Experiments on layout-to-image and quantity-aware generation show that PATHS achieves consistent gains in
alignment quality, particularly on complex
prompts.
Read the full abstract ▾
Computer Vision
Benchmarking Single-Step Inpainting Methods for Multi-Object 3D Gaussian Splatting Scenes
Finn Dröge, Cecilia Curreli, Abhishek Saroha et al. (+1 more)
The tasks of object removal and inpainting 3D Gaussian Splatting (3DGS) scenes face challenges such as 3D consistency across camera views. In comparing 2D inpainters and their suitability for the 3D domain, we find that reconstruction-based inpainters outperform generative
diffusion models in 3D consistency. Integrating these 2D inpainters into different single-step methods for creating and finetuning 3DGS scenes, our results indicate that initializing the scene from scratch produces higher quality results than finetuning the existing scene. Using a state-of-the-art generative 2D inpainter, we create a straightforward baseline to underline the importance of object removal before inpainting in the 3D setting. Since 360{\deg}
datasets rarely include real-world ground truths, and challenging occlusion scenarios are equally sparse, we introduce a novel multi-object scene with recorded ground truth data and many views with object occlusions.
Read the full abstract ▾
All authors Finn Dröge, Cecilia Curreli, Abhishek Saroha, Daniel Cremers
Computer Vision
Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation
Tom Maye-Lasserre, Yitong Li, Bailiang Jian et al. (+3 more)
Modern 3D medical
vision-
language models (VLMs) can generate fluent radiology-style text while exhibit critically low pathology detection and output diversity, collapsing to generic templates that under-report rare yet critical findings. We identify this failure mode as Template Collapse. This failure stems from the unique constraints of 3D medical imaging, e.g., limited data, severe label imbalance, and weak signals from volumetric encoders. Under these constraints, text-generation objectives encourage shortcut learning and fluent but weakly grounded reports. We systematically diagnose the Template Collapse through clinical fidelity, output diversity, normal-template bias, and rare-finding survival. To mitigate it, we propose CLarGen, a decoupled framework that separates what to say (clinical detection) from how to say it (
language synthesis). CLarGen uses (i) a Latent Query
Transformer for multi-label pathology detection, (ii) pathology-guided
retrieval for clinically matched exemplars, and (iii) a medical
language model to synthesize the final report from detected findings and retrieved context. Across state-of-the-art 3D CT report generation baselines, CLarGen mitigates Template Collapse and substantially improves clinical accuracy (macro-F1 0.487 vs. 0.189; CRG 0.472 vs. 0.368) while maintaining fluent reporting. Our results suggest that explicit, measurable clinical grounding is essential for template-collapse-resistant 3D CT report generation. Code is available at https://github.com/ai-med/CLarGen.
Read the full abstract ▾
All authors Tom Maye-Lasserre, Yitong Li, Bailiang Jian, Morteza Ghahremani, Benedikt Wiestler, Christian Wachinger
Computer Vision
Can BEV Perception Gracefully Degrade under Sensor Failures?
Haifa Zhang, Yijing Wang and Zhiqiang Zuo
Despite the remarkable success of multi-modal bird's-eye view (BEV) perception in autonomous driving, standard fusion models can be highly brittle to sensor corruptions, causing catastrophic performance degradation. This vulnerability arises when corrupted features contaminate the shared BEV representation and the detector fails to preserve a usable visual fallback. To this end, we present Grace-BEV, a lightweight framework for robust fusion in Lift-Splat-Shoot (LSS)-based detectors. Operating in the aligned BEV space, Grace-BEV complements a
LiDAR-guided image-BEV expert with a
LiDAR-independent visual expert. A TrustGate Router learns a sample-level mixture of their outputs, while the FailSafe Fusion Block recalibrates the routed image representation together with raw
LiDAR features. Furthermore, we devise a Three-Phase Training strategy with Modality Dropout to establish the visual fallback before learning routing and fusion, followed by joint fine-tuning. Extensive experiments on nuScenes-R and nuScenes-C across three LSS-based detectors show that Grace-BEV maintains robust performance across diverse corruption settings. Under complete
LiDAR loss, where the corresponding baseline models exhibit a collapse to 0.0 mAP, Grace-BEV restores performance to as high as 34.7 mAP. It also improves clean accuracy by up to 1.4 mAP points with 1.2M additional parameters per detector. Component and training ablations support the complementary roles of visual initialization, expert routing, and adaptive fusion.
Read the full abstract ▾
Computer Vision
BiSegMamba: Efficient Bidirectional Tri-Oriented Mamba for 3D Medical Image Segmentation
Bakht Zada, Chao Tong, Qile Su et al. (+1 more)
Accurate 3D medical image
segmentation requires both fine spatial detail and long-range volumetric context. Although Mamba provides efficient long-sequence modeling, volumetric formulations remain sensitive to spatial serialization order, repeated multi-orientation processing, and fixed aggregation of orientation-specific representations. We propose BiSegMamba, an efficient bidirectional tri-oriented Mamba framework that follows a compact-to-detail and local-to-global processing strategy. A progressive compacting stem shifts semantic
reasoning to compact representations while retaining shallow high-resolution features for reconstruction, and a multi-scale spatial mixer captures local anatomical patterns. For global context modeling, volumetric features are serialized along three anatomical orientations and their reversed directions, the resulting six sequences are processed jointly by a shared Mamba operator, and reconciled using learned channel-wise forward-backward weighting. The restored orientation-specific features are subsequently integrated through input-dependent directional fusion. Experiments on an in-house carotid
dataset and ACDC, BraTS2023, and AMOS-CT achieve Dice scores of 96.3%, 91.63%, 92.57%, and 89.03%, respectively. Compared with SegMamba-V2 under standardized $128^3$ profiling, BiSegMamba reduces the parameter count by 65.9% (138.77M$\rightarrow$47.38M) and inference latency by 53.6% (129.08$\rightarrow$59.86 ms), while increasing throughput by 115.7% (7.75$\rightarrow$16.71 samples/s). These results demonstrate a favorable accuracy–efficiency trade-off among volumetric Mamba-based
segmentation models.
Read the full abstract ▾
All authors Bakht Zada, Chao Tong, Qile Su, Shuai Zhang
Computer Vision
Omni-Supervised Motion Editing: Balancing Change and Invariance through Positive-Negative Learning
Zhenwu Shi, Jingyu Gong, Peiwei Wang et al. (+7 more)
Text-based human motion editing aims to modify existing motion sequences according to natural
language instructions while maintaining the consistency of the original motion. Existing
diffusion-based approaches often rely on heuristic similarity cues or coarse global conditioning, leading to motion distortion and suboptimal semantic
alignment. The key challenge lies in balancing change (i.e. precisely editing target regions) and invariance (i.e. preserving unedited parts). To handle such challenge, we propose an Omni-Supervised Positive-Negative Learning framework, named OmniME. Our method integrates three complementary components: (1) retrospective feature supervision that enforces coarse-to-fine consistency across
transformer layers,(2) motion preservation mechanism that focuses on subtle variations according to the source-target similarity, and (3) triplet-based semantic
alignment that strengthens text-motion correspondence. Together, these components form a unified supervision paradigm that balances change and invariance. Extensive experiments on the MotionFix and STANCE Adjustment
datasets demonstrate that OmniME achieves state-of-the-art performance in editing
alignment, validating the effectiveness of our unified learning framework. Our source codes and models have been released at: https://github.com/rocket-ycyer/OmniME.git
Read the full abstract ▾
All authors Zhenwu Shi, Jingyu Gong, Peiwei Wang, Xingzan Wang, Tianwen Qian, Wenxi Li, Yuan Fang, Jiao Xie, Lizhuang Ma, Shaohui Lin
Computer Vision
Variational Adapter for Cross-modal Similarity Representation
WenZhang Wei, Zhipeng Gui, Dehua Peng et al. (+2 more)
The core of
vision-
language models lies in measuring cross-modal similarity within a unified representation space. However, most image-text matching or multi-class image classification
datasets lack fine-grained cross-modal matching annotations, forcing the continuous similarity space into binary classification boundaries. This compression induces false negative samples and significantly impairs the generalization performance of cross-modal tasks. While prior research has attempted to mitigate this by modeling intra-modal ambiguity, it often overlooks inherent annotation flaws, leading to suboptimal uncertainty allocation. To address these challenges, we propose a Variational Adapter for Cross-modal Similarity Representation (VACSR). This approach reformulates image-text matching with fine-grained semantic scarcity as a variational inference problem. It constructs a latent space for cross-modal similarity and uses regularization techniques to mitigate overfitting to binary annotations. Experiments on image-text
retrieval, domain generalization, and base-to-novel generalization demonstrate the proposed method's effectiveness and robust generalization ability.
Read the full abstract ▾
All authors WenZhang Wei, Zhipeng Gui, Dehua Peng, Tiandi Ye, Huayi Wu
Computer Vision
PRISM: Progressive Reasoning through Iterative Slot Memory for Vision
Ziyu Wang, Shuangpeng Han and Mengmi Zhang
Modern
vision models process images in a single feed-forward pass, which limits their ability to recover missing evidence or refine uncertain representations under incomplete observations. Inspired by the iterative nature of human perception, we introduce PRISM (Progressive
Reasoning through Iterative Slot
Memory), a pyramid
vision architecture that reasons over images through iterative refinement. At a high level, PRISM groups visual features into object-centric representations, retrieves relevant patterns from a learned
memory, and iteratively refines the representation to resolve ambiguity and recover missing information. This organize-recall-refine process operates recurrently across multiple scales, enabling progressive improvement of visual representations. Across standard
vision tasks, including image classification, object detection, and semantic
segmentation, PRISM achieves competitive performance while demonstrating improved robustness under incomplete observations such as occlusion. These results suggest that iterative
reasoning with structured representations and
memory is a promising direction for building more resilient and adaptive
vision models. Source code and models will be released.
Read the full abstract ▾
Computer Vision
IAF-Net: Illumination-Adaptive Fusion for Low-Light Urban Road Segmentation
Bingtao Wang, Daojie Peng, Fulong Ma et al. (+2 more)
Semantic road
segmentation is important for autonomous driving, but existing methods suffer severe performance degradation under low-light conditions. Many existing multi-modal fusion methods do not explicitly adapt to illumination-dependent changes in modality reliability, which can propagate degraded RGB features into the fused representation at night. We propose IAF-Net (Illumination-Adaptive Fusion Network), an end-to-end framework with illumination-adaptive fusion for robust road
segmentation across different lighting conditions. It dynamically adjusts fusion weights of RGB and
geometric features via the core Illumination-Adaptive Fusion (IAF) module, and enhances low-light feature selection with a brightness-modulated
attention decoder. We also construct two dedicated
datasets: nuScenes Nighttime Road
Segmentation (nuScenes-NRS) and CARLA Multi-Weather Road
Segmentation (CARLA-MWRS). Experiments on nuScenes-NRS show state-of-the-art overall performance among the compared methods, while CARLA-MWRS further validates robustness across adverse weather conditions. Ablation studies on a 40% training subset further highlight the importance of the IAF module, which provides the largest individual gain of 0.70% in MaxF.
Read the full abstract ▾
All authors Bingtao Wang, Daojie Peng, Fulong Ma, Jun Ma, Liang Zhang
Computer Vision
MultiAct: Text-to-Motion Generation from Composite Text via Tailored Attention Guidance
Nathan Sala, Ofir Abramovich, Ariel Shamir et al. (+3 more)
Text-to-motion generation has progressed rapidly in recent years, offering an expressive interface for animation and human-computer interaction. However, current models remain brittle when handling
prompts that describe multiple actions occurring at the same time. Rather than realizing all components of a composite description, models frequently prioritize a single dominant action and neglect the rest, leading to incomplete or ambiguous motion. We present MultiAct, an unpaired, inference-time framework for compositional text-to-motion synthesis that operates directly on pretrained motion generators without retraining or architectural modification. Our method counteracts semantic collapse by adaptively amplifying cross-
attention scores associated with underrepresented
prompt components. We note that effective modulation depends on
prompt-specific choices, such as which
tokens and layers to target, and introduce a lightweight auxiliary decision scheme that determines the most effective
attention-strengthening parametrization. Extensive quantitative and qualitative evaluations demonstrate that MultiAct consistently outperforms existing baselines on composite
prompts, achieving improved semantic coverage while preserving motion realism. Project page: https://natsala13.github.io/multiact.github.io.
Read the full abstract ▾
All authors Nathan Sala, Ofir Abramovich, Ariel Shamir, Daniel Cohen-Or, Andreas Aristidou, Sigal Raab
Computer Vision · Cross-listed from cs.IR
Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search
Gyu-Hwung Cho, Youngjune Lee, Kiyoon Jeong et al. (+5 more)
As large-scale visual-document corpora such as arXiv papers and enterprise PDFs continue to grow, visual-document
retrieval has gained increasing
attention; yet it still lacks a deployable system that lexically indexes visual documents to serve queries without neural encoding at scale. Existing methods either achieve strong
retrieval quality with VLM-based dense or multi-vector models but require neural query encoding at serving time, or avoid query encoding with OCR- or caption-based BM25 at the cost of time-consuming text extraction or generation. To fill this missing serving regime, we present V-SPLADE, an inference-free sparse retriever for visual-document
retrieval. However, such inference-free
multimodal learned sparse
retrieval systems remain underexplored and have not yet shown dense-level effectiveness under high sparsity. We attribute this limitation to a lexical grounding problem: visual sparse representations often fail to capture the lexical content embedded in document images. To address this problem, we introduce caption-gated
token supervision, a training-only signal that uses VLM-generated captions as lexical cues to activate
retrieval-relevant vocabulary dimensions. With this supervision, V-SPLADE improves average NDCG@5 across six visual-document
retrieval benchmarks by +13.8pp over the same-scale dense baseline and by up to +6.3pp over OCR- or caption-based BM25 baselines. On an 18.7M-document corpus, it more than doubles R@5 over the same-scale dense baseline and further improves competing retrievers through score fusion by up to +2.4pp R@5. Code will be released soon at https://github.com/naver/v-splade.
Read the full abstract ▾
All authors Gyu-Hwung Cho, Youngjune Lee, Kiyoon Jeong, Siyoung Lee, Sanggyu Han, Hervé Dejean, Stéphane Clinchant, Seung-won Hwang
Computer Vision
DiTTo: Scalable Order-aware All-in-One Image Restoration Agent
Seungho Choi and Jihyong Oh
Real-world images rarely suffer from a single degradation, and the order in which degradations are removed substantially affects the final restoration quality, motivating
agent-based image restoration (IR), where a
vision-
language model schedules a pool of pre-built restoration-experts. However, existing training-based
agents require $\mathcal{O}((N^{\mathbf{D}})^{2})$ restoration-expert calls per image to construct the Optimal Restoration-action Trajectory
Dataset (ORTD), where $N^{\mathbf{D}}$ denotes the number of degradation types in the universe $\mathbf{D}$, and couple
agent training to a fixed restoration-expert pool, preventing extension to newly introduced restoration-experts without full retraining. To overcome these efficiency and extensibility bottlenecks, we propose DiTTo, a novel order-aware image restoration
agent framework consisting of the DiTTo Simulator and the DiTTo
Agent. The DiTTo Simulator combines $\cup$S-IR for single-step restoration-action simulation and AiO-IQA for per-action quality prediction, reducing ORTD construction to $\mathcal{O}(N^{\mathbf{D}})$ simulator calls per image; the DiTTo
Agent is trained by SFT on the simulator-generated ORTD, followed by Order-aware Restoration
Alignment (ORA) that aligns degradation identification, restoration-action-ordering, and output format along independent axes. This enables plug-and-play scalable extensibility: adding a new restoration-expert requires updating only the lightweight ORA stage. On the MiO-100 evaluation set with up to five concurrent degradations, our DiTTo
Agent achieves state-of-the-art multi-degradation restoration quality among previous
agent-based IR methods.
Read the full abstract ▾
Computer Vision
Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR
Ruina Hu, Chen Wang, Lai Wei et al. (+5 more)
Reinforcement learning with verifiable rewards (RLVR) improves
vision-
language models (VLMs) by optimizing outcome rewards derived from final answers. However, such outcome-only rewards do not tell the model which image regions justify an answer. For questions that require visual grounding, these rewards cannot distinguish responses supported by relevant visual evidence from those produced by
language-prior shortcuts or lucky guesses. We introduce EASE (Evidence-Anchored Spatial
Attention), which augments
multimodal RLVR with visual-evidence process supervision. EASE converts annotated evidence regions into a smoothed visual-
token target and uses it to guide response-to-image
attention during RL training, but only on high-reward trajectories. The annotations are used solely as privileged training labels, while inference requires only the original image and question. Across Qwen2.5-VL-7B, Qwen3-VL-4B, and Qwen3-VL-8B, EASE raises average scores over DAPO by 2.5 to 3.1 points on perception,
hallucination, visual math, and
multimodal reasoning benchmarks. Diagnostics and ablations show that EASE better aligns visual
attention with annotated evidence regions.
Read the full abstract ▾
All authors Ruina Hu, Chen Wang, Lai Wei, Jionghao Bai, Bin Yu, Weiran Huang, Kai Wang, Yue Wang
Computer Vision
What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination Robustness
Yusheng He, Jizhe Zhou, Xia Du et al. (+3 more)
Hallucination remains one of the key challenges undermining the reliability of Large
Vision-
Language Models (LVLMs). But what makes an LVLM hallucinate less? Many existing efforts focus on improving internal components of the model. We argue that
hallucination fundamentally stems from how the model architecture is designed. To investigate this, we factor the architecture design into three dimensions: Linguistic Foundation (LF), Visual Representation (VR), and Semantic
Alignment (SA), and categorize
hallucinations into Co-occurrence, Similarity, and previously overlooked Uncertainty types. Building on this formulation, we propose CoSimUE, a
benchmark that creates fine-grained
hallucination scenarios through controlled textual perturbations and random perturbations, enabling mapping between design choices and
hallucination behaviors. Experiments across 7 design aspects show that: 1) the widely emphasized scaling of model parameters has only limited impact on reducing all three types of
hallucinations; 2) larger and better-trained
language foundations can reduce co-occurrence
hallucinations; 3) stronger visual encoders and higher resolutions mitigate similarity errors; 4) effective
alignment strategies alleviate uncertainty
hallucinations. 5) Furthermore, cross-dimensional analysis reveals that jointly enhancing visual fidelity and
alignment quality yields the most comprehensive improvements. This study provides the first systematic exploration linking architecture-level design to
hallucination robustness, offering practical guidance for developing reliable and efficient LVLMs.
Read the full abstract ▾
All authors Yusheng He, Jizhe Zhou, Xia Du, Zheng Lin, Jun Luo, Jiancheng Lv
Computer Vision
MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
Luyuan Zhang, Siyuan Li, Zedong Wang et al. (+7 more)
Most visual tokenizers for image generation are bifurcated into two families with complementary limitations: continuous VAEs offer high-fidelity reconstruction but suffer from dense, entangled latents that are poorly suited for semantic control, whereas discrete VQ-based models enable autoregressive generation yet struggle with gradient sparsity, unstable training, and codebook collapse. In this work, we introduce MergeTok, a unified tokenizer that jointly optimizes continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture, leveraging
token merging techniques as a semantic bridge. By clustering similar
tokens during encoding, MergeTok establishes a structural prior that provides dual supervision signals: (i) it imposes merged-
token semantic
alignment in the VAE branch, regularizing its latent space toward disentangled, semantic-aware representations; (ii) it derives group-wise constraints, promoting intra-group diversity and inter-group exclusivity that stabilize VQ training. MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched
token budgets, while producing semantically-organized
token representations compatible with both autoregressive and
diffusion generators. This shows that a single architecture can endow visual tokenizers with robust semantic organization and generator-friendly discreteness.
Read the full abstract ▾
All authors Luyuan Zhang, Siyuan Li, Zedong Wang, Qingsong Xie, Cheng Tan, Anna Wang, Yanhao Zhang, Chen Chen, Haonan Lu, Haoqian Wang
Computer Vision
SteerFace: Debiasing Synthetic Face Generation via Adaptive Residue Perturbation
Yuxi Mi, Qiuyang Yuan, Jianqing Xu et al. (+5 more)
The shortage of legally compliant data for face recognition training has sparked growing interest in using synthetic data as an alternative. While recent
diffusion-based methods enable the generation of photorealistic face images with strong identity adherence and data diversity, their downstream recognition performance still exhibits a significant synthetic-real gap. This paper identifies visual tendency as a previously underexplored limitation, whereby synthetic data exhibit an unrealistic prevalence of visual attributes and thus deviate from the real-data distribution. Visual tendency can be attributed to the generator's conditioning on identity
embeddings, through which co-occurring residual visual cues are unintentionally absorbed into learned identity semantics. To discourage the generator from exploiting such visual cues, this paper proposes SteerFace, a simple and efficient training framework that perturbs identity
embeddings by steering them toward random orthogonal directions on the
embedding hypersphere. The perturbation serves as an identity-preserving regularizer that penalizes the generator's reliance on non-identity components, as supported by theoretical analysis. This paper further introduces an adaptive strategy that learns perturbation strengths with both sample-wise preference and favorable overall statistics. Extensive experiments show that SteerFace effectively mitigates visual tendency, outperforms prior methods in downstream face recognition, and generalizes well across different training
datasets and generation pipelines.
Read the full abstract ▾
All authors Yuxi Mi, Qiuyang Yuan, Jianqing Xu, Yichun Zhou, Xuan Zhao, Jun Wang, Rizen Guo, Shuigeng Zhou
Computer Vision
Foundation VAEs for 3D CT Reconstruction, Augmentation, and Generation
Qi Chen, Shuhan Ding, Yu Gu et al. (+5 more)
Variational autoencoders (VAEs) compress high resolution CT volumes into compact latents while preserving clinically relevant structure. However, training CT-specific VAEs from scratch or heavily fine-tuning them incurs substantial computational and engineering cost, and often degrades under heterogeneous scanners, protocols, and diseases. This paper makes a progressive stride toward training-free medical VAEs by leveraging a critical observation: a single Foundation VAE, pretrained at scale on natural images and
videos, can serve as a unified interface for CT Reconstruction, Augmentation, and Generation. With both encoder and decoder frozen, the Foundation VAE reconstructs CT volumes with preserved anatomy while suppressing acquisition noise; training
segmentation models on these reconstructions improves surface accuracy by 3.9% NSD on average for pancreatic tumor and lung tumor. Within the same Foundation VAE latent space, a conditional latent
diffusion model achieves 3.9% lower average FVD with 36.2% higher CT CLIP score, and improves multi-disease generation faithfulness across 18 types by 2.76% AUC. These results demonstrate Foundation VAEs as a practical interface for scalable CT representation reuse and faithful CT generation. Our code and demo are available at https://github.com/qic999/Foundation-VAE.
Read the full abstract ▾
All authors Qi Chen, Shuhan Ding, Yu Gu, Nan Liu, Jiang Bian, Alan Yuille, Zongwei Zhou, Jingjing Fu
Computer Vision
GUI-C$^2$: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
Junlong Li, Chao Hao, Lap-Pui Chau et al. (+1 more)
Existing agentic reinforcement learning methods for GUI grounding have limitations at two levels. At the data level, current approaches typically treat all training samples equally, although their training value to the baseline model varies with difficulty. Overlooking this can greatly reduce training efficiency or even cause collapse. At the strategy level, existing frameworks struggle to balance the trade-off between cropping larger regions for sufficient context and smaller ones for reduced redundancy, a tension inherent to tool-augmented grounding
agents. In addition, overly complex decision-making is difficult for small-parameter models and significantly increases inference time. To address these issues, at the data level, we propose GUI-D, a data mining and difficulty scoring pipeline that identifies the training-worthy samples by proper testing and assigns difficulty scores to guide subsequent training weights. At the strategy level, we propose GUI-C$^2$, which employs an area-gated coarse-to-fine refinement mechanism that progressively narrows the visual field via model-internal uncertainty signals, adaptively reserving context for large targets while amplifying precision for small ones, reinforced by improvement-aware stage rewards that ensure each refinement genuinely advances grounding. Meanwhile, we simplify the decision-making process to greatly reduce additional inference time. Finally, extensive experiments show that our method achieves state-of-the-art performance. The code and data will be publicly available.
Read the full abstract ▾
All authors Junlong Li, Chao Hao, Lap-Pui Chau, Yi Wang
Computer Vision
DSD-GS: Dynamic-Static Decomposition of Gaussian Splatting for Efficient and High-Fidelity Dynamic Scene Reconstruction
Youngtae Han, Sung-hwan Han and Youngmin Yi
Dynamic scene reconstruction and novel view synthesis are fundamental to next-generation visual intelligence applications such as virtual reality, robotics, and digital twins. However, high-fidelity reconstruction of complex, time-varying scenes from arbitrary viewpoints remains a significant challenge. Existing dynamic 3DGS methods suffer from computational inefficiency, since they model all Gaussians as dynamic components. While recent decomposition-based approaches address this issue, they still struggle with degraded reconstruction quality and prolonged training time. To mitigate these limitations, we propose a novel dynamic reconstruction framework built upon an efficient static-dynamic decomposition strategy using a Feed-Forward Gaussian Splatting encoder and an optical flow model. By eliminating redundant computations on static regions, our method achieves state-of-the-art performance, outperforming existing baselines across rendering quality, training and rendering speed, and storage efficiency. Notably, on the Neural 3D
dataset, our framework requires only 10 minutes for training and achieves a rendering speed of over 700 FPS on a single NVIDIA RTX 5090 GPU at resolution of 1352x1014. Furthermore, our decomposition strategy eliminates the need for COLMAP preprocessing and enables deterministic initialization, thereby enhancing both efficiency and reproducibility.
Read the full abstract ▾