Computer Vision
Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation
Quoc-Huy Trinh, Mustapha Abdullahi, Bo Zhao et al. (+1 more)
Recent advances in
multimodal large
language models (MLLMs) have enabled impressive progress in
vision-
language understanding, yet their high computational cost limits deployment in resource-constrained scenarios such as personal assistants, document understanding, and smart cameras. Most existing methods rely on
Transformer-based cross-
attention, whose quadratic complexity hinders efficiency. Moreover, small
vision-
language models often struggle to precisely capture fine-grained, task-relevant visual regions, leading to degraded performance on fine-grained
reasoning tasks that limit their effectiveness in the real world. To address these issues, we introduce Firebolt-VL, an efficient
vision-
language model that replaces the
Transformer-based decoder with a
Liquid Foundation Model (LFM) decoder. To further enhance visual grounding, we propose a
Token-Grid Correlation Module, which computes lightweight correlations between text
tokens and image patches and modulates via the state-space model with FiLM conditioning. This enables the model to selectively emphasize visual regions relevant to the textual
prompt while maintaining linear-time inference. Experimental results across multiple
benchmarks demonstrate that Firebolt-VL achieves accurate, fine-grained understanding with significantly improved efficiency. Our model and code are available at: https://fireboltvl.github.io
Read the full abstract ▾
All authors Quoc-Huy Trinh, Mustapha Abdullahi, Bo Zhao, Debesh Jha
Computer Vision
PR-IQA: Partial-Reference Image Quality Assessment for Diffusion-Based Novel View Synthesis
Inseong Choi, Siwoo Lee, Seung-Hun Nam et al. (+1 more)
Diffusion models are promising for sparse-view novel view synthesis (NVS), as they can generate pseudo-ground-truth views to aid 3D reconstruction pipelines like 3D Gaussian Splatting (3DGS). However, these synthesized images often contain photometric and
geometric inconsistencies, and their direct use for supervision can impair reconstruction. To address this, we propose Partial-Reference Image Quality Assessment (PR-IQA), a framework that evaluates
diffusion-generated views using reference images from different poses, eliminating the need for ground truth. PR-IQA first computes a geometrically consistent partial quality map in overlapping regions. It then performs quality completion to inpaint this partial map into a dense, full-image map. This completion is achieved via a cross-
attention mechanism that incorporates reference-view context, ensuring cross-view consistency and enabling thorough quality assessment. When integrated into a
diffusion-augmented 3DGS pipeline, PR-IQA restricts supervision to high-confidence regions identified by its quality maps. Experiments demonstrate that PR-IQA outperforms existing IQA methods, achieving full-reference-level accuracy without ground-truth supervision. Thus, our quality-aware 3DGS approach more effectively filters inconsistencies, producing superior 3D reconstructions and NVS results. The project page is available at https://kakaomacao.github.io/pr-iqa-project-page/.
Read the full abstract ▾
All authors Inseong Choi, Siwoo Lee, Seung-Hun Nam, Soohwan Song
Computer Vision
Erasure or Erosion? Evaluating Compositional Degradation in Unlearned Text-To-Image Diffusion Models
Arian Komaei Koma, Seyed Amir Kasaei, Ali Aghayari et al. (+2 more)
Post-hoc unlearning has emerged as a practical mechanism for removing undesirable concepts from large text-to-image
diffusion models. However, prior work primarily evaluates unlearning through erasure success; its impact on broader generative capabilities remains poorly understood. In this work, we conduct a systematic empirical study of concept unlearning through the lens of compositional text-to-image generation. Focusing on nudity removal in Stable
Diffusion 1.4, we evaluate a diverse set of state-of-the-art unlearning methods using T2I-CompBench++ and GenEval, alongside established unlearning
benchmarks. Our results reveal a consistent trade-off between unlearning effectiveness and compositional integrity: methods that achieve strong erasure frequently incur substantial degradation in attribute binding, spatial
reasoning, and counting. Conversely, approaches that preserve compositional structure often fail to provide robust erasure. These findings highlight limitations of current evaluation practices and underscore the need for unlearning objectives that explicitly account for semantic preservation beyond targeted suppression.
Read the full abstract ▾
All authors Arian Komaei Koma, Seyed Amir Kasaei, Ali Aghayari, AmirMahdi Sadeghzadeh, Mohammad Hossein Rohban
Computer Vision
TAPE: A two-stage parameter-efficient adaptation framework for foundation models in OCT-OCTA analysis
Xiaofei Su, Zengshuo Wang, Minghe Sun et al. (+2 more)
Automated analysis of optical coherence
tomography (OCT) and OCT angiography (OCTA) images is critical for robust ophthalmic diagnosis. Existing mainstream methods trained from scratch rely heavily on massive data and model scale, thereby hindering their practical deployment in resource-constrained clinical settings. Although transfer learning based on foundation models (FMs) is promising, it still faces significant challenges: domain shift and task misalignment. To address these, we propose TAPE: A Two-stage Adaptation Framework via Parameter-Efficient Fine-tuning, which strategically decouples adaptation into domain
alignment and task fitting for downstream
segmentation. The domain adaptation stage notably applies parameter-efficient fine-tuning (PEFT) in the context of masked image modeling for medical image domain adaptation, a novel approach to the best of our knowledge. Applying TAPE to retinal layer
segmentation on both universal (masked auto-encoder, MAE) and specialized (RETFound) FMs, it demonstrates superior parameter efficiency and achieves state-of-the-art generalization performance across diverse pathologies.
Read the full abstract ▾
All authors Xiaofei Su, Zengshuo Wang, Minghe Sun, Xin Zhao, Mingzhu Sun
Computer Vision · Cross-listed from cs.RO
Abdelmoamen Nasser, Yousef Baba'a, Murad Mebrahtu et al. (+3 more)
Traditional approaches to off-road autonomy rely on separate models for terrain classification, height estimation, and quantifying slip or slope conditions. Utilizing several models requires training each component separately, having task specific
datasets, and fine-tuning. In this work, we present a zero-shot approach leveraging SAM2 for environment
segmentation and a
vision-
language model (VLM) to reason about drivable areas. Our approach involves passing to the VLM both the original image and the segmented image annotated with numeric labels for each mask. The VLM is then prompted to identify which regions, represented by these numeric labels, are drivable. Combined with planning and control modules, this unified framework eliminates the need for explicit terrain-specific models and relies instead on the inherent
reasoning capabilities of the VLM. Our approach surpasses state-of-the-art trainable models on high resolution
segmentation datasets and enables full stack
navigation in our Isaac Sim offroad environment.
Read the full abstract ▾
All authors Abdelmoamen Nasser, Yousef Baba'a, Murad Mebrahtu, Nadya Abdel Madjid, Jorge Dias, Majid Khonji
Computer Vision
Temporal Inversion for Learning Interval Change in Chest X-Rays
Hanbin Ko, Kyungmin Jeon, Doowoong Choi et al. (+1 more)
Recent advances in
vision–
language pretraining have enabled strong medical foundation models, yet most analyze radiographs in isolation, overlooking the key clinical task of comparing prior and current images to assess interval change. For chest radiographs (CXRs), capturing interval change is essential, as radiologists must evaluate not only the static appearance of findings but also how they evolve over time. We introduce TILA (Temporal Inversion-aware Learning and
Alignment), a simple yet effective framework that uses temporal inversion, reversing image pairs, as a supervisory signal to enhance the sensitivity of existing temporal
vision-
language models to directional change. TILA integrates inversion-aware objectives across pretraining, fine-tuning, and inference, complementing conventional appearance modeling with explicit learning of temporal order. We also propose a unified evaluation protocol to assess order sensitivity and consistency under temporal inversion, and introduce MS-CXR-Tretrieval, a
retrieval evaluation set constructed through a general protocol that can be applied to any temporal CXR
dataset. Experiments on public
datasets and real-world hospital cohorts demonstrate that TILA consistently improves progression classification and temporal
embedding alignment when applied to multiple existing architectures.
Read the full abstract ▾
All authors Hanbin Ko, Kyungmin Jeon, Doowoong Choi, Chang Min Park
Computer Vision
Geometry-Aware Epipolar Correspondence Graphs for Monocular Visual Odometry
Prateeth Rao and Sachit Rao
We demonstrate Graph Representation Learning to address challenges that arise in visual odometry. In this context, the change in pose of a camera is typically estimated using matched correspondences in frames that satisfy the epipolar constraint; finding this unique set under different situations is not a simple task. Employing an existing detector-free model to find matching correspondences in a pair of images, a correspondence graph is constructed whose
topology is guided by the epipolar constraint. Starting with a graph where every node is connected to $k$ closest neighbours (by distance), this graph is pruned based on an initial estimate of the Essential
Matrix; this feature is in stark contrast with traditional graph formulations where graphs from local spatial neighborhoods defined by heuristics are constructed. The graph learning module is then trained to generate an aggregated representative vector that summarises the
geometric context of the graph. This vector forms an input to a Multi-layer Perceptron (MLP), whose outputs are standard rotation and translation elements that define the change in pose. To investigate the effect of graph learning on this representation, four variations of Graph Neural Networks (GNNs): GCN, GAT, GIN, and Graph-based Cross
Attention are evaluated under a unified learning framework. The application to
monocular visual odometry is validated using outdoor
benchmark datasets such as KiTTi, ETH3D, King's College, and Tartan Air. Experimental results demonstrate consistent improvements in trajectory and relative pose estimation across diverse metrics when compared with classical consensus-based
estimators and learning-guided correspondence methods.
Read the full abstract ▾
Computer Vision
StableTTA: Improving Vision Model Performance by Training-free Test-Time Adaptation Methods
Zheng Li, Jerry Cheng and Huanying Helen Gu
Ensemble methods improve predictive performance but often incur high
memory and computational costs. We identify an aggregation instability induced by nonlinear projection and voting operations. To address both efficiency challenges and this inconsistency, we propose StableTTA, a training-free test-time adaptation method with two variants. StableTTA-I targets coherent-batch inference settings, where temporally or semantically adjacent observations are likely to belong to the same class. Examples include burst photography,
video streams, robotics perception, and industrial inspection. Under coherent-batch inference, StableTTA-I substantially improves prediction consistency and accuracy through variance-aware logit aggregation. StableTTA-II establishes feature-level cropping, enabling efficient logit aggregation with a single forward pass on a single model backbone. Experiments on ImageNet-1K across 71 models demonstrate that StableTTA-I consistently improves prediction accuracy under coherent-batch inference, while StableTTA-II provides lightweight and architecture-agnostic accuracy improvements with minimal computational overhead. These results suggest that inference-time semantic coherence and aggregation stability provide useful perspectives for improving practical test-time adaptation systems.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
G-EDF-Loc: 3D Continuous Gaussian Distance Field for Robust Gradient-Based 6DoF Localization
José E. Maese, Lucía Coto-Elena, Luis Merino et al. (+1 more)
This paper presents a robust 6-DoF localization framework based on a direct, CPU-based scan-to-map registration pipeline. The system leverages G-EDF, a novel continuous and
memory-efficient 3D distance field representation. The approach models the Euclidean Distance Field (EDF) using a Block-Sparse Gaussian Mixture Model with adaptive spatial partitioning, ensuring $C^1$ continuity across block transitions and mitigating boundary artifacts. By leveraging the analytical gradients of this continuous map, which maintain Eikonal consistency, the proposed method achieves high-fidelity spatial reconstruction and real-time localization. Experimental results on large-scale
datasets demonstrate that G-EDF-Loc performs competitively against state-of-the-art methods, exhibiting exceptional resilience even under severe odometry degradation or in the complete absence of IMU priors.
Read the full abstract ▾
All authors José E. Maese, Lucía Coto-Elena, Luis Merino, Fernando Caballero
Computer Vision · Cross-listed from cs.LG
Reproducibility study on how to find Spurious Correlations, Shortcut Learning, Clever Hans or Group-Distributional non-robustness and how to fix them
Ole Delzer and Sidney Bender
Deep Neural Networks (DNNs) are increasingly utilized in high-stakes domains like medical diagnostics and autonomous driving where model reliability is critical. However, the research landscape for ensuring this reliability is terminologically fractured across communities that pursue the same goal of ensuring models rely on causally relevant features rather than confounding signals. While frameworks such as distributionally robust optimization (DRO), invariant risk minimization (IRM), shortcut learning, simplicity bias, and the Clever Hans effect all address model failure due to spurious correlations, researchers typically only reference work within their own domains. This reproducibility study unifies these perspectives through a comparative analysis of correction methods under challenging constraints like limited data availability and severe subgroup imbalance. We evaluate recently proposed correction methods based on explainable artificial intelligence (XAI) techniques alongside popular non-XAI baselines using both synthetic and real-world
datasets. Findings show that XAI-based methods generally outperform non-XAI approaches, with Counterfactual Knowledge
Distillation (CFKD) proving most consistently effective at improving generalization. Our experiments also reveal that the practical application of many methods is hindered by a dependency on group labels, as manual annotation is often infeasible and automated tools like Spectral Relevance Analysis (SpRAy) struggle with complex features and severe imbalance. Furthermore, the scarcity of minority group samples in validation sets renders model selection and hyperparameter tuning unreliable, posing a significant obstacle to the deployment of robust and trustworthy models in safety-critical areas.
Read the full abstract ▾
Computer Vision
MPTF-Net: Multi-view Pyramid Transformer Fusion Network for LiDAR-based Place Recognition
Shuyuan Li, Zihang Wang, Xieyuanli Chen et al. (+5 more)
LiDAR-based place recognition (LPR) is essential for global localization and loop-closure detection in large-scale SLAM systems. Existing methods typically construct global descriptors from Range Images or BEV representations for matching. BEV is widely adopted due to its explicit 2D spatial layout encoding and efficient
retrieval. However, conventional BEV representations rely on simple statistical aggregation, which fails to capture fine-grained
geometric structures, leading to performance degradation in complex or repetitive environments. To address this, we propose MPTF-Net, a novel multi-view multi-scale pyramid
Transformer fusion network. Our core contribution is a multi-channel NDT-based BEV encoding that explicitly models local
geometric complexity and intensity distributions via Normal Distribution Transform, providing a noise-resilient structural prior. To effectively integrate these features, we develop a customized pyramid
Transformer module that captures cross-view interactive correlations between Range Image Views (RIV) and NDT-BEV at multiple spatial scales. Extensive experiments on the nuScenes, KITTI and NCLT
datasets demonstrate that MPTF-Net achieves state-of-the-art performance, specifically attaining a Recall@1 of 96.31% on the nuScenes Boston split while maintaining an inference latency of only 10.02 ms, making it highly suitable for real-time autonomous unmanned systems.
Read the full abstract ▾
All authors Shuyuan Li, Zihang Wang, Xieyuanli Chen, Wenkai Zhu, Xiaoteng Fang, Peizhou Ni, Junhao Yang, Dong Kong
Computer Vision
MedROI: Codec-Agnostic Region of Interest-Centric Compression for Medical Images
Jiwon Kim and Ikbeom Jang
Medical imaging archives are growing rapidly in both size and resolution, making efficient compression increasingly important for storage and data transfer. Most existing codecs compress full images/volumes(including non-diagnostic background) or apply differential ROI coding that still preserves background bits. We propose MedROI, a codec-agnostic, plug-and-play ROI-centric framework that discards background voxels prior to compression. MedROI extracts a tight tissue bounding box via lightweight intensity-based thresholding and stores a fixed 54byte meta data record to enable spatial restoration during decompression. The cropped ROI is then compressed using any existing 2D or 3D codec without architectural modifications or retraining. We evaluate MedROI on 200 T1-weighted
brain MRI volumes from ADNI using 6 codec configurations spanning conventional codecs (JPEG2000 2D/3D, HEIF) and neural compressors (LIC_TCM, TCM+AuxT, BCM-Net, SirenMRI). MedROI yields statistically significant improvements in compression ratio and encoding/decoding time for most configurations (two-sided t-test with multiple-comparison correction), while maintaining comparable reconstruction quality when measured within the ROI; HEIF is the primary exception in compression-ratio gains. For example, on JPEG20002D (lv3), MedROI improves CR from 20.35 to 27.37 while reducing average compression time from 1.701s to 1.380s. Code is available at https://github.com/labhai/MedROI.
Read the full abstract ▾
Computer Vision
Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward
Shizhan Gong, Minda Hu, Qiyuan Zhang et al. (+2 more)
Vision-
language models (VLMs) have achieved remarkable success across diverse tasks. However, concerns about their trustworthiness persist, particularly regarding tendencies to lean more on textual cues than visual evidence and the risk of producing ungrounded or fabricated responses. To address these issues, we propose Saliency-R1, a framework for improving the interpretability and faithfulness of VLMs
reasoning. Specifically, we introduce a novel saliency map technique that efficiently highlights critical image regions contributing to generated
tokens without additional computational overhead. This can further be extended to trace how visual information flows through the
reasoning process to the final answers, revealing the
alignment between the thinking process and the visual context. We use the overlap between the saliency maps and human-annotated bounding boxes as the reward function, and apply Group Relative Policy Optimization (GRPO) to align the salient parts and critical regions, encouraging models to focus on relevant areas when conduct
reasoning. Experiments show Saliency-R1 improves
reasoning faithfulness, interpretability, and overall task performance.
Read the full abstract ▾
All authors Shizhan Gong, Minda Hu, Qiyuan Zhang, Chen Ma, Qi Dou
Computer Vision
The Indra Representation Hypothesis for Multimodal Alignment
Jianglin Lu, Hailing Wang, Kuo Yang et al. (+3 more)
Recent studies have uncovered an interesting phenomenon: unimodal foundation models tend to learn convergent representations, regardless of differences in architecture, training objectives, or data modalities. However, these representations are essentially internal abstractions of samples that characterize samples independently, leading to limited expressiveness. In this paper, we propose The Indra Representation Hypothesis, inspired by the philosophical metaphor of Indra's Net. We argue that representations from unimodal foundation models are converging to implicitly reflect a shared relational structure underlying reality, akin to the relational ontology of Indra's Net. We formalize this hypothesis using the V-enriched Yoneda
embedding from category theory, defining the Indra representation as a relational profile of each sample with respect to others. This formulation is shown to be unique, complete, and structure-preserving under a given cost function. We instantiate the Indra representation using angular distance and evaluate it in cross-model and cross-modal scenarios involving
vision,
language, and audio. Extensive experiments demonstrate that Indra representations consistently enhance robustness and
alignment across architectures and modalities, providing a theoretically grounded and practical framework for training-free
alignment of unimodal foundation models. Our code is available at https://github.com/Jianglin954/Indra.
Read the full abstract ▾
All authors Jianglin Lu, Hailing Wang, Kuo Yang, Yitian Zhang, Simon Jenni, Yun Fu
Computer Vision
A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models
Tianmeng Fang, Yong Wang, Zetai Kong et al. (+4 more)
Multimodal large
language models have become an important infrastructure for unified processing of visual and linguistic tasks. However, such models are highly susceptible to
backdoor implantation during supervised fine-tuning and will steadily output the attacker's predefined harmful responses once a specific trigger pattern is activated. The core challenge of
backdoor defense lies in suppressing attack success under low poisoning ratios while preserving the model's normal generation ability. These two objectives are inherently conflicting. Strong suppression often degrades benign performance, whereas weak regularization fails to mitigate
backdoor behaviors. To this end, we propose a unified defense framework based on patch augmentation and cross-view regularity, which simultaneously constrains the model's anomalous behaviors in response to triggered patterns from both the feature representation and output distribution levels. Specifically, patch-level data augmentation is combined with cross-view output difference regularization to exploit the fact that
backdoor responses are abnormally invariant to non-semantic perturbations and to proactively pull apart the output distributions of the original and perturbed views, thereby significantly suppressing the success rate of
backdoor triggering. At the same time, we avoid over-suppression of the model during defense by imposing output
entropy constraints, ensuring the quality of normal command generation. Experimental results across three models, two tasks, and six attacks show that our proposed defense method effectively reduces the attack success rate while maintaining a high level of normal text generation capability. Our work enables the secure, controlled deployment of large-scale
multimodal models in realistic low-frequency poisoning and covert triggering scenarios.
Read the full abstract ▾
All authors Tianmeng Fang, Yong Wang, Zetai Kong, Zengzhen Su, Jun Wang, Chengjin Yu, Wei Wang
Computer Vision
Training-Free Image Editing with Visual Context Integration and Concept Alignment
Rui Song, Guo-Hua Wang, Qing-Guo Chen et al. (+6 more)
In image editing, it is essential to incorporate a context image to convey the user's precise requirements, such as subject appearance or image style. Existing training-based visual context-aware editing methods incur data collection effort and training cost. On the other hand, the training-free alternatives are typically established on
diffusion inversion, which struggles with consistency and flexibility. In this work, we propose VicoEdit, a training-free and inversion-free method to inject the visual context into the pretrained text-prompted editing model. More specifically, VicoEdit directly transforms the source image into the target one based on the visual context, thereby eliminating the need for inversion that can lead to deviated trajectories. Moreover, we design a posterior sampling approach guided by concept
alignment to enhance the editing consistency. Empirical results demonstrate that our training-free method achieves even better editing performance than the state-of-the-art training-based models.
Read the full abstract ▾
All authors Rui Song, Guo-Hua Wang, Qing-Guo Chen, Weihua Luo, Tongda Xu, Zhening Liu, Yan Wang, Zehong Lin, Jun Zhang
Computer Vision · Cross-listed from eess.IV
TM-BSN: Triangular-Masked Blind-Spot Network for Real-World Self-Supervised Image Denoising
Junyoung Park, Youngjin Oh and Nam Ik Cho
Blind-spot networks (BSNs) enable self-supervised image denoising by preventing access to the target pixel, allowing clean signal estimation without ground-truth supervision. However, this approach assumes pixel-wise noise independence, which is violated in real-world sRGB images due to spatially correlated noise from the camera's image signal processing (ISP) pipeline. While several methods employ downsampling to decorrelate noise, they alter noise statistics and limit the network's ability to utilize full contextual information. In this paper, we propose the Triangular-Masked Blind-Spot Network (TM-BSN), a novel blind-spot architecture that accurately models the spatial correlation of real sRGB noise. This correlation originates from demosaicing, where each pixel is reconstructed from neighboring samples with spatially decaying weights, resulting in a diamond-shaped pattern. To align the receptive field with this geometry, we introduce a triangular-masked convolution that restricts the kernel to its upper-triangular region, creating a diamond-shaped blind spot at the original resolution. This design excludes correlated pixels while fully leveraging uncorrelated context, eliminating the need for downsampling or post-processing. Furthermore, we use knowledge
distillation to transfer complementary knowledge from multiple blind-spot predictions into a lightweight U-Net, improving both accuracy and efficiency. Extensive experiments on real-world
benchmarks demonstrate that our method achieves state-of-the-art performance, significantly outperforming existing self-supervised approaches. Our code is available at https://github.com/parkjun210/TM-BSN.
Read the full abstract ▾
Computer Vision
MVis-Fold: A Three-Dimensional Microvascular Structure Inference Model for Super-Resolution Ultrasound
Jincao Yao, Ke Zhang, Yahan Zhou et al. (+8 more)
Super-resolution ultrasound (SRUS) technology has overcome the resolution limitations of conventional ultrasound, enabling micrometer-scale imaging of microvasculature. However, due to the nature of imaging principles, three-dimensional reconstruction of microvasculature from SRUS remains an open challenge. We developed microvascular visualization fold (MVis-Fold), an innovative three-dimensional microvascular reconstruction model that integrates a cross-scale network architecture. This model can perform high-fidelity inference and reconstruction of three-dimensional microvascular networks from two-dimensional SRUS images. It precisely calculates key parameters in three-dimensional space that traditional two-dimensional SRUS cannot readily obtain. We validated the model's accuracy and reliability in three-dimensional microvascular reconstruction of solid tumors. This study establishes a foundation for three-dimensional quantitative analysis of microvasculature. It provides new tools and methods for diagnosis and monitoring of various diseases.
Read the full abstract ▾
All authors Jincao Yao, Ke Zhang, Yahan Zhou, Jiafei Shen, Jie Liu, Mudassar Ali, Bojian Feng, Jiye Chen, Jinlong Fan, Ping Liang, Dong Xu
Computer Vision
Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
Jia Chengyu, AprilPyone MaungMaung, Huy H. Nguyen et al. (+2 more)
Recent advances in
vision-
language models (VLMs) trained on web-scale image-text pairs have enabled impressive zero-shot transfer across a diverse range of visual tasks. However, comprehensive and independent evaluation beyond standard
benchmarks is essential to understand their robustness, limitations, and real-world applicability. This paper presents a systematic evaluation framework for VLMs under natural
adversarial scenarios for diverse downstream tasks, which has been overlooked in previous evaluation works. We evaluate a wide range of VLMs (CLIP, robust CLIP, BLIP2, and SigLIP2) on curated
adversarial datasets (typographic attacks, ImageNet-A, and natural
language-induced
adversarial examples). We measure the natural
adversarial performance of selected VLMs for zero-shot image classification, semantic
segmentation, and visual question answering. Our analysis reveals that robust CLIP models can amplify natural
adversarial vulnerabilities, and CLIP models significantly reduce performance for natural
language-induced
adversarial examples. Additionally, we provide interpretable analyses to identify failure modes. We hope our findings inspire future research in robust and fair
multimodal pattern recognition.
Read the full abstract ▾
All authors Jia Chengyu, AprilPyone MaungMaung, Huy H. Nguyen, Jinyin Chen, Isao Echizen
Computer Vision
Group-DINOmics: Incorporating People Dynamics into DINO for Self-supervised Group Activity Feature Learning
Ryuki Tezuka, Chihiro Nakatani and Norimichi Ukita
This paper proposes Group Activity Feature (GAF) learning without group activity annotations. Unlike prior work, which uses low-level static local features to learn GAFs, we propose leveraging dynamics-aware and group-aware pretext tasks, along with local and global features provided by DINO, for group-dynamics-aware GAF learning. To adapt DINO and GAF learning to local dynamics and global group features, our pretext tasks use person flow estimation and group-relevant object location estimation, respectively. Person flow estimation is used to represent the local motion of each person, which is an important cue for understanding group activities. In contrast, group-relevant object location estimation encourages GAFs to learn scene context (e.g., spatial relations of people and objects) as global features. Comprehensive experiments on public
datasets demonstrate the state-of-the-art performance of our method in group activity
retrieval and recognition. Our ablation studies verify the effectiveness of each component in our method. Code: https://github.com/tezuka0001/Group-DINOmics.
Read the full abstract ▾
Computer Vision
Beyond Few-Step Inference: Accelerating Video Diffusion Transformer Model Serving with Inter-Request Caching Reuse
Hao Liu, Ye Huang, Chenghuan Huang et al. (+5 more)
Video Diffusion Transformer (DiT) models are a dominant approach for high-quality
video generation but suffer from high inference cost due to iterative denoising. Existing caching approaches primarily exploit similarity within the
diffusion process of a single request to skip redundant denoising steps. In this paper, we introduce Chorus, a caching approach that leverages similarity across requests to accelerate
video diffusion model serving. Chorus achieves up to 45% speedup on industrial 4-step distilled models, where prior intra-request caching approaches are ineffective. Particularly, Chorus employs a three-stage caching strategy along the denoising process. Stage 1 performs full reuse of latent features from similar requests. Stage 2 exploits inter-request caching in specific latent regions during intermediate denoising steps. This stage is combined with
Token-Guided
Attention Amplification to improve semantic
alignment between the generated
video and the conditional
prompts, thereby extending the applicability of full reuse to later denoising steps.
Read the full abstract ▾
All authors Hao Liu, Ye Huang, Chenghuan Huang, Zhenyi Zheng, Jiangsu Du, Ziyang Ma, Jing Lyu, Yutong Lu
Computer Vision
Parameter-Efficient Semantic Augmentation for Enhancing Open-Vocabulary Object Detection
Weihao Cao, Runqi Wang, Xiaoyue Duan et al. (+3 more)
Open-vocabulary object detection (OVOD) enables models to detect any object category, including unseen ones. Benefiting from large-scale pre-training, existing OVOD methods achieve strong detection performance on general scenarios (e.g., OV-COCO) but suffer severe performance drops when transferred to downstream tasks with substantial domain shifts. This degradation stems from the scarcity and weak semantics of category labels in domain-specific task, as well as the inability of existing models to capture auxiliary semantics beyond coarse-grained category label. To address these issues, we propose HSA-DINO, a parameter-efficient semantic augmentation framework for enhancing open-vocabulary object detection. Specifically, we propose a multi-scale
prompt bank that leverages image feature pyramids to capture hierarchical semantics and select domain-specific local semantic
prompts, progressively enriching textual representations from coarse to fine-grained levels. Furthermore, we introduce a semantic-aware router that dynamically selects the appropriate semantic augmentation strategy during inference, thereby preventing parameter updates from degrading the generalization ability of the pre-trained OVOD model. We evaluate HSA-DINO on OV-COCO, several vertical domain
datasets, and modified
benchmark settings. The results show that HSA-DINO performs favorably against previous state-of-the-art methods, achieving a superior trade-off between domain adaptability and open-vocabulary generalization.
Read the full abstract ▾
All authors Weihao Cao, Runqi Wang, Xiaoyue Duan, Jinchao Zhang, Ang Yang, Liping Jing
Computer Vision · Cross-listed from cs.LG
Estimating Central, Peripheral, and Temporal Visual Contributions to Human Decision Making in Atari Games
Henrik Krauss and Takehisa Yairi
We study how different visual information sources contribute to human decision making in dynamic visual environments. Using Atari-HEAD, a large-scale Atari gameplay
dataset with synchronized eye-tracking, we introduce a controlled ablation framework as a means to reverse-engineer the contribution of peripheral visual information, explicit gaze information in the form of gaze maps, and past-state information from human behavior. We train action-prediction networks under six settings that selectively include or exclude these information sources. Across 20 games, peripheral information shows by far the strongest contribution, with median prediction-accuracy drops in the range of 35.27-43.90% when removed. Gaze information yields smaller drops of 2.11-2.76%, while past-state information shows a broader range of 1.52-15.51%, with the upper end likely more informative due to reduced peripheral-information leakage. To complement aggregate accuracies, we cluster states by true-action probabilities assigned by the different model configurations. This analysis identifies coarse behavioral regimes, including focus-dominated, periphery-dominated, and more contextual decision situations. These results suggest that human decision making in Atari depends strongly on information beyond the current focus of gaze, while the proposed framework provides a way to estimate such information-source contributions from behavior.
Read the full abstract ▾
Computer Vision
HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance
Green Rosh, Prateek Kukreja, Vishakha SR et al. (+1 more)
The emergence of virtual reality has necessitated the generation of detailed and customizable 3D hand models for interaction in the virtual world. However, the current methods for 3D hand model generation are both expensive and cumbersome, offering very little customizability to the users. While recent advancements in zero-shot text-to-3D synthesis have enabled the generation of diverse and customizable 3D models using Score
Distillation Sampling (SDS), they do not generalize very well to 3D hand model generation, resulting in unnatural hand structures, view-inconsistencies and loss of details. To address these limitations, we introduce HandDreamer, the first method for zero-shot 3D hand model generation from text
prompts. Our findings suggest that view-inconsistencies in SDS is primarily caused due to the ambiguity in the probability landscape described by the text
prompt, resulting in similar views converging to different modes of the distribution. This is particularly aggravated for hands due to the large variations in articulations and poses. To alleviate this, we propose to use MANO hand model based initialization and a hand skeleton guided
diffusion process to provide a strong prior for the hand structure and to ensure view and pose consistency. Further, we propose a novel corrective hand shape guidance loss to ensure that all the views of the 3D hand model converges to view-consistent modes, without leading to
geometric distortions. Extensive evaluations demonstrate the superiority of our method over the state-of-the-art methods, paving a new way forward in 3D hand model generation.
Read the full abstract ▾
All authors Green Rosh, Prateek Kukreja, Vishakha SR, Pawan Prasad B H
Computer Vision
BoxComm: Benchmarking Category-Aware Commentary Generation and Narration Rhythm in Boxing
Kaiwen Wang, Kaili Zheng, Rongrong Deng et al. (+3 more)
Recent
multimodal large
language models (MLLMs) have shown strong capabilities in general
video understanding, driving growing interest in automatic sports commentary generation. However, existing
benchmarks for this task focus exclusively on team sports such as soccer and basketball, leaving combat sports entirely unexplored. Notably, combat sports present distinct challenges: critical actions unfold within milliseconds with visually subtle yet semantically decisive differences, and professional commentary contains a substantially higher proportion of tactical analysis compared to team sports. In this paper, we present BoxComm, a large-scale
dataset comprising 445 World Boxing Championship match
videos with over 52K commentary sentences from professional broadcasts. We propose a structured commentary taxonomy that categorizes each sentence into play-by-play, tactical, or contextual, providing the first category-level annotation for sports commentary
benchmarks. Building on this taxonomy, we introduce two novel and complementary evaluations tailored to sports commentary generation: (1) category-conditioned generation, which evaluates whether models can produce accurate commentary of a specified type given
video context; and (2) commentary rhythm assessment, which measures whether freely generated commentary exhibits appropriate temporal pacing and type distribution over continuous
video segments, capturing a dimension of commentary competence that prior
benchmarks have not addressed. Experiments on multiple state-of-the-art MLLMs reveal that current models struggle on both evaluations. We further propose EIC-Gen, an improved baseline incorporating detected punch events to supply structured action cues, yielding consistent gains and highlighting the importance of perceiving fleeting and subtle events for combat sports commentary.
Read the full abstract ▾
All authors Kaiwen Wang, Kaili Zheng, Rongrong Deng, Yiming Shi, Chenyi Guo, Ji Wu
Computer Vision · Cross-listed from cs.CL
Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding
Haruka Kawasaki, Ryota Tanaka and Kyosuke Nishida
Visual document understanding (VDU) is a challenging task for large
vision language models (LVLMs), requiring the integration of visual perception, text recognition, and
reasoning over structured layouts. Although recent LVLMs have shown progress on VDU
benchmarks, their performance is typically evaluated based on generated responses, which may not necessarily reflect whether the model has actually captured the required information internally. In this paper, we investigate how information required to solve VDU tasks is represented across different layers of LLMs within LVLMs using linear probing. Our study reveals that (1) there is a clear gap between internal representations and generated responses, and (2) information required to solve the task is often encoded more linearly from intermediate layers than from the final layer. Motivated by these findings, we explore fine-tuning strategies that target intermediate layers. Experiments show that fine-tuning intermediate layers improves both linear probing accuracy and response accuracy while narrowing the gap.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
NAIMA: Semantics Aware RGB Guided Depth Super-Resolution
Tayyab Nasir, Daochang Liu and Ajmal Mian
Guided depth super-resolution (GDSR) is a multi-modal approach for depth map super-resolution that relies on a low-resolution depth map and a high-resolution RGB image to restore finer structural details. However, the misleading color and texture cues indicating depth discontinuities in RGB images often lead to artifacts and blurred depth boundaries in the generated depth map. Recent methods counter this by drawing priors from large pretrained models, but these priors enter the network as decoded predictions such as relative depth, surface normal, or
segmentation maps, coupling restoration quality to the accuracy of the decoded prior and requiring auxiliary objectives. We propose a solution that introduces global contextual semantic priors, generated from pretrained
vision transformer token embeddings, injecting them directly into the depth branch. Our Guided
Token Attention (GTA) module lets multi-level depth encodings act as queries of cross-
attention over semantic
tokens drawn from progressively deeper layers of the pretrained visual
transformer (ViT), scaled by a zero-initialized gate that admits semantic evidence only to the extent that it reduces reconstruction error. The resulting
token-to-depth correspondence is aligned implicitly, through learned
attention under a single reconstruction loss, rather than through an explicit
alignment or distribution-matching objective. Building on this, we present Neural
Attention with Implicit Multi-
token Alignment (NAIMA), which, to the best of our knowledge, is the first GDSR framework guided by undecoded semantic
tokens. NAIMA remains competitive in-distribution while achieving the strongest cross-
dataset generalization.
Read the full abstract ▾
Computer Vision
3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image
Ze-Xin Yin, Liu Liu, Xinjie Wang et al. (+4 more)
Compositional 3D scene generation from a single view requires the simultaneous recovery of scene layout and 3D assets. Existing approaches mainly fall into two categories: feed-forward generation methods and per-instance generation methods. The former directly predict 3D assets with explicit 6DoF poses through efficient network inference, but they generalize poorly to complex scenes. The latter improve generalization through a divide-and-conquer strategy, but suffer from time-consuming pose optimization. To bridge this gap, we introduce 3D-Fixer, a novel in-place completion paradigm. Specifically, 3D-Fixer extends 3D object generative priors to generate complete 3D assets conditioned on the partially visible point cloud at the original locations, which are cropped from the fragmented geometry obtained from the geometry estimation methods. Unlike prior works that require explicit pose
alignment, 3D-Fixer uses fragmented geometry as a spatial anchor to preserve layout fidelity. At its core, we propose a coarse-to-fine generation scheme to resolve boundary ambiguity under occlusion, supported by a dual-branch conditioning network and an Occlusion-Robust Feature
Alignment (ORFA) strategy for stable training. Furthermore, to address the data scarcity bottleneck, we present ARSG-110K, the largest scene-level
dataset to date, comprising over 110K diverse scenes and 3M annotated images with high-fidelity 3D ground truth. Extensive experiments show that 3D-Fixer achieves state-of-the-art
geometric accuracy, which significantly outperforms baselines such as MIDI and Gen3DSR, while maintaining the efficiency of the
diffusion process. Code and data will be publicly available at https://zx-yin.github.io/3dfixer.
Read the full abstract ▾
All authors Ze-Xin Yin, Liu Liu, Xinjie Wang, Wei Sui, Zhizhong Su, Jian Yang, Jin Xie
Computer Vision
UENR-600K: A Large-Scale Physically Grounded Dataset for Nighttime Video Deraining
Pei Yang, Hai Ci, Beibei Lin et al. (+2 more)
Nighttime
video deraining is uniquely challenging because raindrops interact with artificial lighting. Unlike daytime white rain, nighttime rain takes on various colors and appears locally illuminated. Existing small-scale synthetic
datasets rely on 2D rain overlays and fail to capture these physical properties, causing models to generalize poorly to real-world night rain. Meanwhile, capturing real paired nighttime
videos remains impractical because rain effects cannot be isolated from other degradations like sensor noise. To bridge this gap, we introduce UENR-600K, a large-scale, physically grounded
dataset containing 600,000 1080p frame pairs. We utilize Unreal Engine to simulate rain as 3D particles within virtual environments. This approach guarantees photorealism and physically real raindrops, capturing correct details like color refractions, scene occlusions, rain curtains. Leveraging this high-quality data, we establish a new state-of-the-art baseline by adapting the Wan 2.2
video generation model. Our baseline treat deraining as a
video-to-
video generation task, exploiting strong generative priors to almost entirely bridge the sim-to-real gap. Extensive benchmarking demonstrates that models trained on our
dataset generalize significantly better to real-world
videos. Project page: https://showlab.github.io/UENR-600K/.
Read the full abstract ▾
All authors Pei Yang, Hai Ci, Beibei Lin, Yiren Song, Mike Zheng Shou
Computer Vision
BiTDiff: Fine-Grained 3D Conducting Motion Generation via BiMamba-Transformer Diffusion
Tianzhi Jia, Kaixing Yang, Xiaole Yang et al. (+4 more)
3D conducting motion generation aims to synthesize fine-grained conductor motions from music, with broad potential in music education, virtual performance, digital human animation, and human-AI co-creation. However, this task remains underexplored due to two major challenges: (1) the lack of large-scale fine-grained 3D conducting
datasets and (2) the absence of effective methods that can jointly support long-sequence generation with high quality and efficiency. To address the data limitation, we develop a quality-oriented 3D conducting motion collection pipeline and construct CM-Data, a fine-grained SMPL-X
dataset with about 10 hours of conducting motion data. To the best of our knowledge, CM-Data is the first and largest public
dataset for 3D conducting motion generation. To address the methodological limitation, we propose BiTDiff, a novel framework for 3D conducting motion generation, built upon a BiMamba-
Transformer hybrid model architecture for efficient long-sequence modeling and a
Diffusion-based generative strategy with human-kinematic decomposition for high-quality motion synthesis. Specifically, BiTDiff introduces auxiliary physical-consistency losses and a hand-/body-specific forward-kinematics design for better fine-grained motion modeling, while leveraging BiMamba for
memory-efficient long-sequence temporal modeling and
Transformer for cross-modal semantic
alignment. In addition, BiTDiff supports training-free joint-level motion editing, enabling downstream human-AI interaction design. Extensive quantitative and qualitative experiments demonstrate that BiTDiff achieves state-of-the-art (SOTA) performance for 3D conducting motion generation on the CM-Data
dataset. Code will be available upon acceptance.
Read the full abstract ▾
All authors Tianzhi Jia, Kaixing Yang, Xiaole Yang, Xulong Tang, Ke Qiu, Shikui Wei, Yao Zhao