Computer Vision
InterLight: Leveraging Intrinsic Illumination Priors for Low-Light Image Enhancement
Ziqi Wang, Xu Zhang, Laibin Chang et al. (+3 more)
Low-Light Image Enhancement (LLIE) has long been a challenging problem in low-level
vision, as insufficient illumination often leads to low contrast, detail loss, and noise. Recent studies show that deep learning-based Retinex theory can effectively decouple illumination and reflectance. However, existing methods frequently suffer from over-enhancement or color distortion, and often assume uniform noise or ideal lighting. To address these limitations, we propose InterLight, a novel framework that systematically excavates and operationalizes intrinsic illumination priors for LLIE.Our core insight is that robust enhancement requires not just estimating illumination, but constructing an illumination-aware pipeline. We first inject sensor-level illumination-response priors via physics-guided augmentation, then represent the degradation through adaptive
prompts conditioned on the scene's latent illumination state. This explicit representation directly guides a luminance-gated intrinsic
memory mechanism to selectively compensate for information loss, prioritizing reconstruction in dark regions while preserving fidelity in bright ones. Finally, the entire process is regularized by a self-supervised consistency objective that distills illumination-invariant features. By deeply exploiting intrinsic illumination priors, our method achieves clearer textures and more visually coherent enhancement results. Extensive experiments across multiple
benchmarks demonstrate the effectiveness of our approach. Code is available at: https://github.com/House-yuyu/InterLight.
Read the full abstract ▾
All authors Ziqi Wang, Xu Zhang, Laibin Chang, Shi Chen, Jiaqi Ma, Huan Zhang
Computer Vision
RECIPE: Procedural Planning via Grounding in Instructional Video
Luigi Seminara, Antonino Furnari and Lorenzo Torresani
Visual planning asks a model to generate the remaining steps of a procedure in natural
language given a partial
video context and a goal. Progress on this task is bottlenecked by annotation: clean labeled
datasets are small, domain-narrow, and encode a single execution trajectory per example, even though many valid orderings exist. Large-scale instructional
video corpora offer orders of magnitude more procedural content, but supervised fine-tuning on pseudo-labels from their noisy ASR narrations propagates
segmentation and
alignment errors and stays single-trajectory. We identify a key asymmetry: extracting clean step labels from noisy
video is hard, but verifying whether a generated step sequence is temporally grounded in ASR transcripts is cheap and scales to millions of
videos via precomputed text
embeddings. We exploit this asymmetry in RECIPE, which uses grounding quality as a reward for GRPO, turning the noisy corpus into a verifier rather than a label source. The framework applies uniformly to two planner input configurations (Socratic, with a textual history extracted by a frozen VLM, and
Video, consuming
video tokens directly) and to annotated and weakly supervised regimes. We evaluate on 7 procedural
benchmarks using a reference-based LLM-as-judge protocol scoring plans across 6 procedural criteria. RECIPE-RL improves over the base checkpoint at all scales (0.5B, 3B, 7B) and every
benchmark, with macro-accuracy gains of +7 to +8 points in-domain and up to +16 points zero-shot. It outperforms supervised fine-tuning on both annotated and pseudo-labeled plans (the latter degrades the base) and remains robust without human annotations. Used as the proposal stage of a prior propose-assess-search planner, it improves over the strongest zero-shot baseline at every horizon on Visual Planning for Assistance, and on COIN it preserves the generation diversity that SFT collapses.
Read the full abstract ▾
Computer Vision
SphericalDreamer: Generating Navigable Immersive 3D Worlds with Panorama Fusion
Antoine Schnepf, Karim Kassab, Flavian Vasile et al. (+1 more)
The generation of immersive and navigable 3D environments is increasingly prevalent with the growing adoption of virtual reality and 3D content. However, recent methods face a fundamental limitation: they cannot produce 3D worlds that simultaneously (i) are navigable over long-range spatial extents and (ii) cover the complete omnidirectional field of view ($360^\circ$ horizontally and $180^\circ$ vertically). To address this challenge, we introduce SphericalDreamer, a method for generating fully immersive and long-range 3D outdoor environments from textual
prompts. Our approach is built on the generation of multiple panoramic images, which are subsequently lifted into 3D and fused together while maintaining visual and
geometric consistency. SphericalDreamer produces highly detailed, fully immersive 3D environments, while substantially improving scale and navigability compared to prior approaches.
Read the full abstract ▾
All authors Antoine Schnepf, Karim Kassab, Flavian Vasile, Andrew Comport
Computer Vision
World-Ego Modeling for Embodied Video Generation in Long-Horizon Navigation-Manipulation Tasks
Zuyao Lin, Jianhui Zhang, Peidong Jia et al. (+4 more)
Embodied video world models typically capture both scene
evolution and the
robot's behavior, which we refer to as the world and the ego, respectively. The world and the ego exhibit different underlying dynamics: world prediction relies primarily on visual history and emphasizes scene stability, whereas ego prediction relies more strongly on the current instruction and emphasizes accurate instruction following. Modeling both components within a single generation stream can entangle these different dependencies, making it difficult to specialize the prediction of either component. Consequently, it becomes difficult to simultaneously maintain scene consistency and accurate instruction following, particularly in long-horizon
navigation-manipulation tasks. In this paper, we propose to decompose an
embodied video into the world and the ego and disentangle their generation processes. Specifically, we define the world as the background and currently unmanipulated objects, and the ego as the
robot and currently manipulated objects. Based on this definition, we develop the World-Ego Model (WEM), which combines a
vision-
language state predictor using role-conditioned
attention (RCA) and asymmetric query budgets with a semantic-routed mixture-of-experts (SR-MoE)
diffusion generator. To enable rigorous evaluation, we further construct HTEWorld, a
dataset and
benchmark for long-horizon
embodied video generation with hybrid
navigation-manipulation tasks, providing 125K training
video clips comprising over 4.5M frames with fine-grained instructions, together with 300 multi-turn evaluation trajectories covering over 2K instructions. Extensive experiments show that WEM achieves state-of-the-art performance on HTEWorld while remaining competitive on existing manipulation-oriented evaluations.
Read the full abstract ▾
All authors Zuyao Lin, Jianhui Zhang, Peidong Jia, Xiaoguang Zhao, Shanghang Zhang, Jingdong Wang, Xingyu Chen
Computer Vision
Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
Jia-Wei Hai, Yijun Wang and Xiu-Shen Wei
Vision-
Language Models (VLMs), such as CLIP, have achieved significant zero-shot performance on downstream tasks with various fine-tuning adaptation methods. However, recent studies have proven that
adversarial attacks can significantly degrade the inference ability of VLMs, posing substantial risks to their practical applications. Prevalent test-time adaptation methods typically rely on multi-view augmentation to implement various fine-tuning strategies, which struggle to identify semantic information and are prone to destroying discriminative regions in fine-grained scenarios. To address these limitations, we propose
Attention-Guided Test-Time
Prompt Tuning (A-TPT), a semantics-preserving method designed for test-time adaptation. We first refine the gradient
attention rollout mechanism to identify semantically meaningful regions surviving under
adversarial attacks. Furthermore, we leverage them to guide the spatially varying augmentation intensities and multi-view ensemble for
prompt tuning and inference. Extensive experiments demonstrate that A-TPT outperforms existing test-time adaptation methods on both
adversarial and clean data. Codes are available at https://github.com/SEU-VIPGroup/A-TPT .
Read the full abstract ▾
Computer Vision
AffectVerse: Emotional World Models for Multimodal Affective Computing
Bo Zhao, Fanghua Ye, Yixin Ji et al. (+3 more)
Humans infer
emotions by integrating observed
multimodal cues with expectations about how affective states may unfold. Existing
multimodal large
language models (MLLMs), however, often treat
emotion recognition as static fusion over complete audiovisual-text inputs, leaving affective dynamics implicit. We propose AffectVerse, a Qwen2.5-Omni-based model equipped with an
Emotion World Module (EWM), an action-free representation-level module for short-horizon latent affective prediction. \rev{EWM contains three modules: 1) Cross-Modal Temporal Imagination predicts future
video/audio representations from past
tokens with multi-step rollout. 2) MAMA(Modality-Aware Multi-step
Attention) Belief Aggregation compresses imagined
tokens into modality-aware belief
tokens. 3) Belief Injection inserts these belief
tokens into the LLM for affective
reasoning.} AffectVerse uses future prediction as a past-conditioned self-supervised signal: it does not replace modeling observed history or require unseen signals at inference, but forces the current belief state to encode transition cues that are predictive of subsequent affective change. Across nine
benchmarks, AffectVerse improves at least 2.57% over other models, while controlled ablations show additive gains from temporal imagination, cross-modal rollout, and belief aggregation. These results suggest predictive belief-state modeling is a practical alternative for affective computing.
Read the full abstract ▾
All authors Bo Zhao, Fanghua Ye, Yixin Ji, Sicheng Zhao, Xiaojiang Peng, Zitong YU
Computer Vision
Feed-Forward Gaussian Splatting from Sparse Aerial Views
Dongli Wu, Zhuoxiao Li, Tongyan Hua et al. (+4 more)
Reconstructing large-scale urban scenes from sparse aerial views is a crucial yet challenging task. Due to biased top-down and shallow-oblique camera poses, sparse aerial captures exhibit strong evidence imbalance: roofs and open regions are repeatedly observed, while facades, distant buildings, and occluded structures receive little multi-view support. Existing feed-forward 3D Gaussian Splatting methods directly regress a deterministic representation from sparse inputs, but this often leads to ghosting, melted facades, and stretched textures. Recent pseudo-view and
video-based generative reconstruction methods use additional supervision or generative priors. However, they often lack a clear separation between observed geometry and prior-driven content, which can lead to plausible but inconsistent structures. We propose AnyCity, an observation-grounded generative reconstruction framework for sparse aerial urban scenes. AnyCity first predicts an observation-supported geometry latent to anchor reliable structures, and then uses scaffold-conditioned aerial completion
tokens to predict a gated residual update for weakly constrained content before Gaussian decoding. During training, dense-to-sparse
distillation transfers structural cues from dense-view reconstruction, while an aerial-adapted
video diffusion prior provides fine-grained urban appearance cues through gated
token conditioning. Observation-preserving objectives keep the refined representation consistent with input-supported geometry. At inference time, AnyCity reconstructs the final 3D Gaussian scene from sparse aerial views in a single feed-forward pass, achieving coherent urban novel-view synthesis with second-level inference. Experiments on synthetic, aerial-domain,
UAV-textured, and real-world scenes show consistent improvements over feed-forward baselines.
Read the full abstract ▾
All authors Dongli Wu, Zhuoxiao Li, Tongyan Hua, Yinrui Ren, Xiaobao Wei, Rongjun Qin, Wufan Zhao
Computer Vision
StruMPL: Multi-task Dense Regression under Disjoint Partial Supervision and MNAR Labels
Reza M. Asiyabi, Juan Alberto Molina-Valero, The SEOSAW Partnership et al. (+2 more)
Estimating forest aboveground biomass (AGB) from Earth observation combines two structurally incompatible label sources: spaceborne
lidar provides canopy structure at millions of locations but no biomass estimate, and ground-based plots provide biomass at thousands of biased locations but no metrics of structure. No single training sample carries labels for all target variables, plot labels are missing not at random (MNAR), and biomass is linked to the structural variables by known but biome-specific allometric laws. We formalise this as multi-task dense regression under heterogeneous disjoint partial supervision with MNAR labels and inter-task physical constraints, and propose StruMPL to address it jointly. A shared encoder feeds per-variable regression, imputation, and propensity heads for spatial MNAR correction, and a learnable physics module that evaluates the inter-task constraint on the model's own predictions at every pixel. The supervised loss uses an Augmented IPW (AIPW) pseudo-outcome with stop-gradients on the propensity and on the imputation baseline; we show analytically and empirically that both are necessary for joint optimisation to recover IPW-weighted stationary points while keeping the loss bounded. On two ecologically distinct biomes, StruMPL outperforms ablation variants and the closest published method on AGB RMSE and bias, with a stratified analysis showing AIPW reduces high-AGB bias by ~54%.
Read the full abstract ▾
All authors Reza M. Asiyabi, Juan Alberto Molina-Valero, The SEOSAW Partnership, Steven Hancock, Casey M. Ryan
Computer Vision
Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
Yi Zhong, Haotong Qin, Xindong Zhang et al. (+2 more)
Low-bit post-training quantization (PTQ) is a pivotal technique for deploying
Vision-
Language Models (VLMs) on resource-constrained devices. However, existing PTQ methods often degrade VLMs' accuracy due to the heterogeneous activation distributions of text and
vision modalities during quantization. We find that this cross-modal heterogeneity is distributed unevenly across channels: a small subset of channels contains most modality-specific outliers, and these outliers typically reside in different channels for each modality. Motivated by this, we propose SplitQ, a channel-Splitting-driven post-training Quantization framework. At its core, SplitQ introduces a novel Modality-specific Outlier Channel Decoupling (MOCD) module that effectively isolates salient modality-specific outlier channels with minimal overhead. To further address the remaining cross-modal distribution discrepancies, we design an Adaptive Cross-Modal Calibration (ACC) module that employs dual lightweight learnable branches to dynamically mitigate modality-induced quantization errors. Extensive experiments on popular VLMs demonstrate that SplitQ significantly outperforms existing approaches across 6 popular multi-modal
datasets under all evaluated quantization settings, including W4A8, W4A4, W3A3, and W3A2. Notably, SplitQ preserves 93.5% of FP16 performance under the challenging W3A3 setting (69.5 vs. 74.3), pushing the efficiency frontier for deploying advanced VLMs. Our code is available at https://github.com/EMVision-NK/SplitQ
Read the full abstract ▾
All authors Yi Zhong, Haotong Qin, Xindong Zhang, Lei Zhang, Guolei Sun
Computer Vision
GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation
Shyma Alhuwaider, Yasmeen Alsaedy, Merey Ramazanova et al. (+2 more)
Test-time adaptation (TTA) enables a pre-trained model to adapt online to an unlabeled test stream under distribution shift. While most TTA research focuses on the adaptation objective, practical streams also depend critically on the
memory used to select which test samples drive adaptation. Existing
memory mechanisms are usually evaluated as components of specific TTA
algorithms, making it difficult to isolate which
memory design choices matter and when they matter. In this work, we provide a systematic
benchmark that decouples
memory from the adaptation
algorithm and evaluates
memory policies under unified conditions across i.i.d., non-i.i.d., continual, and practical test streams. Our study shows that effective
memory management requires more than retaining recent or class-balanced samples. In particular, intra-class diversity is a key factor for avoiding redundant buffers and maintaining representative adaptation signals under temporally correlated and label-skewed streams. Motivated by this finding, we introduce Guided Observational Test-Time Adaptation (GOTTA), a family of diversity-aware
memory policies that combine class-balanced allocation with feature-space diversity. GOTTA
memories act as drop-in replacements for existing buffers and can be paired with different TTA objectives. Across corruption
benchmarks and
video-stream settings, diversity-aware
memory improves adaptation most clearly under constrained
memory budgets and challenging non-i.i.d. streams, while remaining competitive as
memory capacity increases. These results highlight
memory management as a first-class component of robust test-time adaptation and identify diversity as a central principle for practical TTA.
Read the full abstract ▾
All authors Shyma Alhuwaider, Yasmeen Alsaedy, Merey Ramazanova, Silvio Giancola, Bernard Ghanem
Computer Vision · Cross-listed from cs.GR
GLUT: 3D Gaussian Lookup Table for Continuous Color Transformation
Danna Xue, David Serrano-Lozano, Shaolin Su et al. (+1 more)
3D Lookup Tables (3D LUTs) are widely used for color mapping, but their grid-based representation requires discretizing the RGB space, leading to a capacity-
memory trade-off that becomes prohibitive when storing large numbers of LUTs. Recent approaches adopt implicit neural representations to improve scalability, yet their black-box nature limits interpretability and hinders intuitive, localized editing. In this paper, we propose Gaussian LUT (GLUT), a continuous and explicit color representation that models color transformations using a set of learnable 3D Gaussian primitives. By avoiding fixed-resolution grids, GLUT achieves flexible representational capacity while maintaining a compact
memory footprint. Its explicit, spatially localized formulation further enables both accurate modeling and interpretability. Building on this representation, we introduce a compact conditional generator (CGLUT) that predicts GLUT parameters for multiple LUT instances, encoding diverse color styles in a single framework to enable smooth and controllable LUT style blending. Moreover, GLUT supports efficient, user-friendly editing by allowing localized adjustments to specific color regions without global retraining. Experimental results demonstrate that our approach outperforms prior neural LUT representations in both accuracy and efficiency, while offering improved interpretability and interactive control.
Read the full abstract ▾
All authors Danna Xue, David Serrano-Lozano, Shaolin Su, Javier Vazquez-Corral
Computer Vision
Structural Energy Guidance for View-Consistent Text-to-3D Generation
Qing Zhang, Jinguang Tong, Jing Zhang et al. (+2 more)
Text-to-3D generation based on
diffusion models often suffers from the Janus problem, leading to inconsistent geometry across viewpoints. This work identifies viewpoint bias in 2D
diffusion priors as the main cause and proposes Structural Energy-Guided Sampling (SEGS), a training-free and plug-and-play framework to improve multi-view consistency. SEGS constructs a structural energy in the PCA subspace of U-Net features and injects its gradient into the denoising process. It can be easily integrated into SDS/VSD pipelines without retraining. Experiments show that SEGS reduces the Janus Rate by about 10% on average and improves View-CS scores across multiple baselines, including DreamFusion, Magic3D, and LucidDreamer. This method effectively alleviates viewpoint artifacts while preserving appearance fidelity, providing a flexible solution for high-quality text-to-3D content generation.
Read the full abstract ▾
All authors Qing Zhang, Jinguang Tong, Jing Zhang, Jie Hong, Xuesong Li
Computer Vision
Passive Construction Site Safety Monitoring via Persona-Scaffolded Adversarial Chain-of-Thought VLM Verification
Ananth Sriram, Neel Mokaria and Rajveer Singh
Construction remains the deadliest industry sector in the United States, with 1,055 fatal worker injuries recorded in 2023, and the majority preventable. Existing monitoring approaches are expensive, require real-time human operators, or address only a narrow subset of violations. This paper presents a passive, end-of-shift construction safety monitoring pipeline processing
video from POV body-worn and fixed wall-mounted cameras through a three-stage architecture: (1) fine-tuned YOLO11 for primary PPE and hazard detection, (2) SAM 3 for
segmentation refinement and worker deduplication, and (3) Qwen3-VL-8B-Instruct with a method-prompted, persona-scaffolded three-pass
adversarial chain-of-thought protocol for compliance verification and
hallucination control. The principal contribution is the Stage 3
prompt design: professional persona backstories following the method-actor framing drive an observed 12% precision improvement over single-pass prompting in an informal three-author review of the 12-
video Ironsite development corpus, with the largest gains on
hallucination-prone violation categories. Structural message isolation enforces observational independence between a generator, discriminator, and reconciliation pass governed by asymmetric rules encoding priors about human observation versus automated detection reliability. The system maps violations to OSHA standards, performs REBA-inspired ergonomic risk scoring from pose keypoints, and produces per-worker safety reports with timestamped evidence. An evaluation harness is released for future reproduction.
Read the full abstract ▾
Computer Vision
WoundFormer: Multi-Scale Spatial Feature Fusion for Multi-Class Wound Tissue Segmentation
Muhammad Ashad Kabir and Rabin Dulal
Chronic wounds such as diabetic foot ulcers and pressure injuries require accurate tissue-level assessment to guide treatment planning and monitor healing progression. While deep learning methods have advanced automated wound analysis, most existing approaches focus on binary
segmentation and inadequately model heterogeneous tissue composition due to high intra-class variability and limited annotated data. Multi-class wound tissue
segmentation, therefore, remains a challenging and clinically relevant problem. We propose WoundFormer, a
transformer-based framework that enhances hierarchical spatial feature fusion for multi-class wound tissue
segmentation. Specifically, we replace the standard SegFormer decoder with a spatially-preserving multi-scale aggregation head that maintains feature
topology during cross-scale integration and strengthens contextual interactions through convolutional fusion. This design improves boundary localization and discrimination between visually similar tissue categories while preserving
transformer efficiency. We evaluate WoundFormer on the WoundTissueSeg
dataset (147 images, six tissue classes) and a second
benchmark (DFUTissue
dataset). The proposed method achieves an overall Dice score of 81.9%, outperforming strong CNN- and
transformer-based baselines by up to 4.3 Dice points on the WoundTissueSeg
benchmark, with consistent improvements across minority tissue classes. These results indicate that explicit modeling of hierarchical spatial interactions enhances
transformer representations for heterogeneous wound tissue
segmentation and supports more reliable quantitative wound assessment.
Read the full abstract ▾
Computer Vision
Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding
Peter El Hachem, Ahmed Nassar, A. Said Gurbuz et al. (+2 more)
Vision-
Language Models (VLMs) parse documents end-to-end but frequently break down on layouts unlike those seen in training. We attribute this to a two-hop bottleneck: before the decoder can extract content (Hop 2), it must first classify and localize the enclosing layout entity (Hop 1), and when the first hop fails the second collapses into omissions, malformed structure, or autoregressive repetition. We pre-resolve Hop 1 outside the decoder by running a lightweight RT-DETR detector, serializing its outputs in the parser's native DocTags vocabulary, and injecting them into the
prompt alongside the full page image. Unlike analyze-then-parse approaches that crop the page, or prior
prompt-level priors written in plain text, our prior shares the decoder's generation space and leaves the global image in view as a fallback when detections are noisy. On a 10k-page structural out-of-distribution
benchmark, markdown F1 rises from $0.37$ to $0.92$; on the Chinese subset of OmniDocBench, table TEDS rises from $0.01$ to $0.36$; and on the 26k-page ViDoRe V3
benchmark, infinite-loop decoding failures drop across every industrial domain tested. These gains cost $15\%$ wall-clock latency and a median of $74$
prompt tokens, with no architectural change to the base VLM. An
attention-level analysis further reveals a bimodal phase shift in which the decoder attends to injected layout
tokens when emitting structure and to image patches when emitting content, consistent with the two-hop bottleneck being alleviated. Model weights will be released to support reproducibility.
Read the full abstract ▾
All authors Peter El Hachem, Ahmed Nassar, A. Said Gurbuz, Christoph Auer, Peter W. J. Staar
Computer Vision
Yan-Ting Chen, Hao-Wei Chen, Tsu-Ching Hsiao et al. (+1 more)
Accurate camera viewpoint estimation under sparse-view conditions remains challenging, particularly in two-view scenarios. Recent approaches leverage
diffusion models such as Zero123 to synthesize novel views conditioned on relative viewpoint, showing promising results when repurposed for viewpoint estimation via optimization with MSE loss. However, existing methods often suffer from nonconvex loss landscape with numerous local minima, making them sensitive to initialization and reliant on naive multistart strategies. We analyze these optimization challenges and visualize failure cases, showing that
geometric ambiguities, such as
symmetry and self-similarity, can mislead gradient-based updates toward incorrect viewpoints. To address these limitations, we propose a score-based method that reshapes the optimization landscape to guide updates toward the ground-truth viewpoint, followed by a refinement stage using a viewpoint-conditioned
diffusion model. Experiments show that our method improves convergence, reduces reliance on brute-force sampling, and achieves competitive accuracy with higher sample-efficiency.
Read the full abstract ▾
All authors Yan-Ting Chen, Hao-Wei Chen, Tsu-Ching Hsiao, Chun-Yi Lee
Computer Vision
Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models
Hengfei Wang, Anshul Gupta, Pierre Vuillecard et al. (+1 more)
Vision-
language models (VLMs) have rapidly evolved into general-purpose
multimodal reasoners with strong zero-shot generalization. In this context, VLMs could greatly benefit the analysis of human gaze and
attention, a central task in human behavior understanding that requires
reasoning about the physical scene as well as the activity, interactions, and social context. However, the extent to which VLMs can reliably understand human gaze and related attentional behaviors remains largely unexplored. In this work, we present EyeVLM, a systematic evaluation framework for gaze understanding in VLMs across two complementary dimensions: tasks and models. To assess gaze understanding capabilities, we focus on two core tasks. The first, gaze following, i.e., predicting the 2D location where a person is looking, has a
geometric and visual processing focus, requiring a precise understanding of the human face,
attention direction, 3D scene structure, and spatial grounding of attended targets. The second, social gaze prediction, requires social and relational
reasoning over multi-person interactions (e.g., mutual gaze and shared
attention), and may benefit more from the LLM semantic
reasoning capabilities within VLMs. Regarding models, EyeVLM evaluates these tasks in two ways: a zero-shot setting with a diverse set of state-of-the-art open- and closed-source VLMs, exploring different prompting strategies; and a fine-tuning approach based on task-specific QA pairs, studying the impact of model scale and data scale. As
benchmarks, we rely on existing gaze understanding
datasets and perform a systematic comparison with state-of-the-art purely visual models. Overall, our results show that current VLMs lack precise gaze understanding capabilities. While standard training helps reduce the gap with visual models, significant improvements are still needed.
Read the full abstract ▾
All authors Hengfei Wang, Anshul Gupta, Pierre Vuillecard, Jean-Marc Odobez
Computer Vision
A Framework for Evaluating Zero-Shot Image Generation in Concept-based Explainability
Giacomo Astolfi, Matteo Bianchi, Riccardo Campi et al. (+2 more)
Concept-based Explainable Artificial Intelligence (XAI) interprets deep learning models using human-understandable visual features (e.g., textures or object parts) by linking internal representations to class predictions, thereby bridging the gap between low-level image data and high-level semantics. A major challenge, however, is the reliance on large sets of labeled images to represent each concept, which limits scalability. In this work, we investigate the use of zero-shot Text-to-Image (T2I) generative models as a source of synthetic concept
datasets for concept-based XAI methods. Specifically, we generate concepts using predefined
prompts and evaluate their faithfulness to real ones through four complementary analyses: (1) comparing synthetic vs. real concept images via concept representation similarity; (2) evaluating their intra-similarity by comparing pairs of subsets of the same concept with progressively increasing size; (3) evaluating their performance for downstream explanation tasks using relevant class images; (4) evaluating how removing a concept from tested class images affects explanations of generated concepts. While current T2I generative models promise a shortcut to concept-based XAI, our study highlights challenges and raises open questions about the use of synthetic data generated by zero-shot pipelines in model analyses. The resulting
dataset is available at https://github.com/DataSciencePolimi/ZeroShot-T2I-Concepts.
Read the full abstract ▾
All authors Giacomo Astolfi, Matteo Bianchi, Riccardo Campi, Antonio De Santis, Marco Brambilla
Computer Vision
FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding
Gueter Josmy Faure, Min-Hung Chen, Jia-Fong Yeh et al. (+2 more)
Vision-
Language Models (VLMs) have demonstrated remarkable capabilities in general
video understanding, yet they often struggle with the fine-grained comprehension crucial for real-world applications requiring nuanced interpretation of human actions and interactions. While some recent human-centric
benchmarks evaluate aspects of model behaviour such as
fairness/ethics,
emotion perception, and broader human-centric metrics, they do not combine long-form
videos, very dense QA coverage, and frame-level spatial/temporal grounding at scale. To bridge this gap, we introduce FineBench, a human-centric
video question answering (VQA)
benchmark specifically designed to assess fine-grained understanding. FineBench comprises 199,420 multiple-choice QA pairs densely annotated across 64 long-form
videos (15 minutes each), focusing on detailed person movement, person interaction, and object manipulation, including compositional actions. Our extensive evaluation reveals that while proprietary models like GPT-5 achieve respectable performance, current open-source VLMs significantly underperform, struggling particularly with spatial
reasoning in multi-person scenes and distinguishing subtle differences in human movements and interactions. To address these identified weaknesses, we propose FineAgent, a modular framework that enhances VLMs by leveraging a Localizer and a Descriptor. Experiments show that FineAgent consistently improves the performance of various open VLMs on FineBench. FineBench provides a rigorous testbed for future research into fine-grained human-centric
video understanding, while FineAgent offers a practical approach to enhance such
reasoning in current VLMs. Project page and code at https://joslefaure.github.io/assets/html/finebench.html.
Read the full abstract ▾
All authors Gueter Josmy Faure, Min-Hung Chen, Jia-Fong Yeh, Hung-Ting Su, Winston H. Hsu
Computer Vision
When Preference Labels Fall Short: Aligning Diffusion Models from Real Data
Weiyan Chen, Weijian Deng, Yao Xiao et al. (+5 more)
Preference
alignment aims to guide generative models by learning from comparisons between preferred and non-preferred samples. In practice, most existing approaches rely on preference pairs constructed from model-generated images. Such supervision is inherently relative and can be ambiguous when both samples exhibit artifacts or limited visual quality, making it difficult to infer what constitutes a truly desirable output. In this work, we investigate whether real data can serve as an alternative source of supervision for preference
alignment. We adopt a data-centric perspective and study a curation strategy that treats real images as reference points and constructs preference signals by contrasting them with generated or perturbed samples, without requiring manually annotated preference pairs. Through empirical analysis, we show that real-data-based supervision provides effective guidance for aligning
diffusion models and achieves performance comparable to existing preference-based methods. Our results suggest that real data offers a practical and complementary source of supervision for preference
alignment and highlight directions of label-efficient
alignment strategies. Code and models are available at https://cwyxx.github.io/RealAlign.
Read the full abstract ▾
All authors Weiyan Chen, Weijian Deng, Yao Xiao, Weijie Tu, ZiYi Dong, Ibrahim Radwan, Liang Lin, Pengxu Wei
Computer Vision
CADENet: Condition-Adaptive Asynchronous Dual-Stream Enhancement Network for Adverse Weather Perception in Autonomous Driving
Sherif Khairy and Catherine M. Elias
Adverse weather (rain, fog, sand, and snow) degrades camera-based object detection in autonomous vehicles. Existing enhancement-then-detect approaches stall the safety-critical perception loop, violating hard real-time requirements. Progress on this problem is also constrained by an under-recognized evaluation ceiling: ground truth annotated on degraded images cannot credit a detector that recovers objects the annotators themselves could not see, so a genuinely useful enhancement can register as a near-flat F1 gain. This paper presents CADENet (Condition-Adaptive Asynchronous Dual-stream Enhancement Network), a training-free three-thread system: Thread S (YOLOv11n) delivers detections at full frame rate with zero added latency; Thread Q applies condition-adaptive enhancement (CAPE) and fuses results via
entropy-guided NMS (EG-NMS) without blocking Thread S; Thread E provides CLIP zero-shot weather classification, so new weather categories require only a new text
prompt, with no labeled data and no retraining. Evaluated on 1327 DAWN images (YOLOv11m, IoU = 0.5, confidence = 0.25), CADENet achieves Recall = 0.0103 (micro), F1 = 0.0230 on snow, and F1 = 0.0038 on rain. We formalize the annotation completeness bias on DAWN-class data, so the reported F1 values are lower bounds on the true gain; recall is the annotation-gap-immune headline metric. Thread S sustains approximately 44 FPS regardless of enhancement load. No model retraining or additional sensor hardware is required.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
From Prompts to Pavement Through Time: Temporal Grounding in Agentic Scene-to-Plan Reasoning
Ahmed Y. Gado, Omar Y. Goba, Alaa Hassanein et al. (+2 more)
Recent attempts to support high-level scene interpretation and planning in Autonomous Vehicles (AVs) using ensembles of Large
Language Models (LLMs) and Large
Multimodal Models (LMMs) continue to treat time as a secondary property. This lack of temporal grounding leads to inconsistencies in
reasoning about continuous actions, undermining both safety and interpretability. This work explores whether temporal conditioning within inter-
agent communication can preserve or enhance coherence without introducing degradation in semantic or logical consistency. To investigate this, we introduce three planner architectures with progressively increasing temporal integration and evaluate them on curated subsets of the BDD-X
dataset using semantic, syntactic, and logical metrics. Results show that while temporal conditioning reshapes
reasoning style, it yields no statistically significant improvements in standard NLP-based correctness metrics. However, qualitative analysis reveals predictive hazard
reasoning, stable corrective behavior, and strategic divergence in the Sentinel. These findings clarify the limits of
prompt-based temporal grounding and establish the first empirical
benchmark for temporal scene-to-plan
reasoning.
Read the full abstract ▾
All authors Ahmed Y. Gado, Omar Y. Goba, Alaa Hassanein, Catherine M. Elias, Ahmed Hussein
Computer Vision
LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition
Jiaxin Wang, Muwei Jian, Hui Yu et al. (+2 more)
Facial Expression Recognition (FER) in the wild requires models to identify subtle expression cues under large variations in pose, occlusion, illumination, and identity. Recent FER methods improve robustness by introducing visual
attention, facial landmarks, or
vision-
language models as auxiliary priors. However, these priors are typically integrated in a static manner, failing to capture instance-specific facial variations, thereby resulting in severe
attention redundancy and representation instability. To address this issue, we propose LaCoVL-FER, a landmark-guided contrastive learning network with
vision-
language enhancement for FER, which shifts FER from static prior injection to sample-adaptive prior refinement. Specifically, a Landmark-Guided Adaptive Encoder (LGAE) calibrates regional appearance features with landmark geometry through Bi-branch Gated Cross
Attention (BGCA), suppressing noisy responses and producing expression-relevant representations. In parallel, a
Vision-
Language Enhancement Strategy (VLES) refines the generalizable visual features from a frozen CLIP image encoder into expression-specific visual representations. Based on them, an Expression-Conditioned Prompting (ECP) mechanism adapts fixed class-level textual
prompts from the frozen CLIP text encoder into instance-aware textual representations. The resulting visual-textual representations are aligned as adaptive semantic priors to enhance robustness and generalization. Quantitative and qualitative experiments show that LaCoVL-FER outperforms state-of-the-art methods on RAF-DB, FERPlus, and AffectNet. The code is available at https://github.com/ylin06804/LaCoVL-FER.
Read the full abstract ▾
All authors Jiaxin Wang, Muwei Jian, Hui Yu, Junyu Dong, Yifan Xia
Computer Vision
Hyojun Go, Hyungjin Chung, Prune Truong et al. (+8 more)
For practical use,
diffusion- or flow-based generative models must be aligned with task-specific rewards, such as
prompt fidelity or aesthetic preference. That
alignment is challenging because the reward is defined for clean output images, but the
alignment procedure requires value function estimates at noisy intermediate latents. Existing methods resort to Tweedie-style or Monte Carlo approximations, trading off
estimator bias against computational cost: Tweedie estimates are efficient but biased, while Monte Carlo estimates are more accurate but require expensive rollouts. A natural alternative would be a learned value function, but it remains an open question how to effectively train a strong and general value model specifically for noisy latents. Here, we propose StitchVM, a model stitching framework that efficiently transfers reward models pretrained for clean images to the noisy latent regime. StitchVM starts from an existing, truncated pixel-space reward model and attaches a frozen
diffusion backbone to it as its head. From the pixel-space model, the resulting hybrid retains a carefully pretrained, robust reward capability; from the
diffusion backbone, it inherits its native ability to handle noisy latents. The stitching procedure is exceptionally lightweight, e.g., stitching and finetuning CLIP ViT-L and SD 3.5 Medium takes only 10 GPU-hours. By lifting powerful pixel-space reward models to latent space, StitchVM opens up a new style of
diffusion alignment: instead of rough, yet costly per-sample approximation of the value function, the correct function for the actual, noisy latents is constructed once and then amortized over many samples and iterations. We show that this approach yields improvements across a broad range of downstream steering and post-training methods: DPS becomes $3.2\times$ faster while halving peak GPU
memory, and DiffusionNFT becomes $2.3\times$ faster.
Read the full abstract ▾
All authors Hyojun Go, Hyungjin Chung, Prune Truong, Goutam Bhat, Li Mi, Zhaochong An, Zixiang Zhao, Dominik Narnhofer, Serge Belongie, Federico Tombari, Konrad Schindler
Computer Vision
Synergistic Foundation Models for Semi-Supervised Fetal Cardiac Ultrasound Analysis: SAM-Med2D Boundary Refinement and DINOv3 Semantic Enhancement
Tonghao Zhuang, Shanglong Hu, Yongsheng Luo et al. (+2 more)
We present a semi-supervised framework for joint
segmentation and classification of fetal cardiac ultrasound images. Built upon the EchoCare multi-task backbone, our method integrates SAM-Med2D for boundary refinement and leverages DINOv3 to enhance pseudo-label quality. We introduce view-specific hard masking along with a two-stage optimization strategy: an EMA phase to consolidate
segmentation capabilities, followed by a Classification Fine-Tuning phase that freezes
segmentation parameters and resets the classification head to recover classification performance without compromising
segmentation gains. Evaluated on the FETUS 2026 leaderboard, our method achieves a Dice Similarity Coefficient at 79.99%, Normalized Surface Distance at 61.62%, and F1-score at 41.20%, validating the effectiveness of our approach for prenatal congenital heart disease screening. Source code is publicly available at: https://github.com/2826056177/zcst_fetus2026.
Read the full abstract ▾
All authors Tonghao Zhuang, Shanglong Hu, Yongsheng Luo, Zhiqi Zhang, Yu Li
Computer Vision
Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth
Viktor Kocur, Sithu Aung, Gabrielle Flood et al. (+4 more)
Monocular depth estimation has improved significantly in recent years, driven by increasingly powerful models and large-scale training data. Predicted depth is increasingly used as an input signal for downstream tasks such as Structure-from-Motion (SfM), visual localization, and SLAM. However,
monocular depth
estimators (MDEs) are still primarily evaluated in terms of depth accuracy. Standard metrics aggregate errors globally and may not reflect the usefulness of depth for downstream
geometric tasks. We therefore propose Depth2Pose, a framework for evaluating MDEs in the context of downstream tasks. By combining depth predictions with feature correspondences in depth-aware
geometric solvers, we use relative camera pose estimation accuracy as a task-driven proxy for depth quality. Traditional
benchmarks require dense ground truth in the form of per-pixel depth, which is expensive to obtain. In contrast, our formulation requires only camera poses, which can be estimated efficiently, e.g., using Structure-from-Motion pipelines. As a result, our framework can be applied to scenes where ground-truth depth is difficult to obtain, for example due to large scene scale or heavy occlusions (e.g., vegetated environments). Leveraging this, we introduce the D2P
dataset, which contains challenging scenes outside the distribution of commonly used training data. We show that methods performing well under standard depth error metrics on existing
benchmarks also perform well under our pose-based metric when evaluated on the same
datasets, but do not necessarily generalize to our more challenging
dataset. Finally, we provide a simple and extensible evaluation framework. The
dataset and code are available at kocurvik.github.io/depth2pose.
Read the full abstract ▾
All authors Viktor Kocur, Sithu Aung, Gabrielle Flood, Yaqing Ding, Lukas Bujnak, Torsten Sattler, Zuzana Kukelova
Computer Vision
Mechanisms of Object Localization in Vision-Language Models
Timothy Schaumlöffel, Martina G. Vilas and Gemma Roig
Visually-grounded
language models (VLMs) are highly effective in linking visual and textual information, yet they often struggle with basic classification and localization tasks. While classification mechanisms have been studied more extensively, the processes that support object localization remain poorly understood. In this work, we investigate two representative families, LLaVA-1.5 and InternVL-3.5, using a suite of mechanistic interpretability tools, including
token ablations,
attention knockout, and causal mediation analysis. We find that localization is driven by a containerization mechanism in which object-aligned
tokens define the spatial extent of the object, while the semantic arrangement of
tokens within those boundaries is largely irrelevant to the predicted box. Only a very small set of
attention heads mediates the causal effect for both classification and localization, concentrating in early-mid layers for LLaVA and mid-late layers for InternVL. The two tasks share some early processing but ultimately depend on largely distinct specialized heads. Overall, we provide the first layer- and head-level account of localization in VLMs, revealing narrow computational pathways that can guide future model design and grounding objectives.
Read the full abstract ▾
Computer Vision
Fast 4D Mesh Generation by Spatio-Temporal Attention Chains
Dvir Samuel, Yuval Atzmon, Gal Chechik et al. (+1 more)
4D mesh generation has recently emerged as a powerful paradigm for recovering dynamic 3D structure from
videos, but existing methods remain slow, computationally expensive, and difficult to scale to longer sequences. We introduce a training-free approach that accelerates 4D mesh generation while improving temporal correspondence quality. Our key observation is that temporal correspondences emerge inside a 4D backbone long before its generated meshes become visually accurate. We exploit this with a general framework we call Spatio-Temporal
Attention Chain which propagates information across space and time. Starting from vertices on an anchor mesh, the chain maps vertices to latent
tokens. It then follows temporal correspondences in latent space, and recovers frame-specific vertices through latent-to-vertex
attention. This design avoids expensive explicit matching while preserving anchor mesh details and thereby improving dynamic mesh geometry and temporal consistency. Compared to state-of-the-art, our method generates a 4D mesh in 9 seconds, achieving a $13\times$ speedup while producing higher-quality results. Moreover, our approach scales to
videos up to $16\times$ longer without degrading mesh quality. Beyond generation, the improved correspondences enable competitive zero-shot performance on two downstream tasks: 2D object tracking and 4D tracking. We further show that our framework enables reliable camera estimation, a capability not supported by prior 4D mesh generation methods.
Read the full abstract ▾
All authors Dvir Samuel, Yuval Atzmon, Gal Chechik, Yoni Kasten
Computer Vision
Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation
Yuanpei Zhao, Jie Lin, Chao Zhang et al. (+5 more)
Pairwise preferences and pointwise ratings are the two dominant annotation protocols in image aesthetic assessment (IAA), yet existing
benchmarks adopt only one, leaving their complementarity unmeasured under controlled conditions. We introduce PPaint, a matched dual-protocol
benchmark in which 15 domain experts, 5 per category, annotate 150 Chinese paintings under both protocols across five aesthetic dimensions, collecting 45,900 pairwise expert judgments through a locally dense preference design alongside the matched ratings. The matched design reveals complementary strengths: preferences yield more consistent ordinal rankings, while ratings anchor the absolute score scale. Fusing both signals via two independent preference-to-score methods yields a fused expert ground truth on which the two constructions converge to nearly identical scores. The same preference-to-score principle extends to label-free VLM training. PSDistill converts VLM pairwise judgments into calibrated pseudo-scores via an Elo reference pool, and trains the same VLM with confidence-weighted ranking optimization to produce a single-pass aesthetic scorer. Trained on a single painting category, the distilled Qwen3-VL-8B improves mean SRCC from 0.504 to 0.709 across all three categories, outperforming all open-source baselines including the dedicated aesthetic model ArtiMuse and matching closed-source Gemini-3.1-Pro within 0.04 SRCC at single-pass inference cost, with cross-domain transfer further validated on APDDv2. We will release the full PPaint
dataset and training code.
Read the full abstract ▾
All authors Yuanpei Zhao, Jie Lin, Chao Zhang, Yilin Wang, Mao Li, Chenhui Li, Jie Hou, Tangjie Lv
Computer Vision · Cross-listed from cs.RO
Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negatives
Junli Wang, Zhihua Hua, Xueyi Liu et al. (+7 more)
Existing imitation learning methods for end-to-end autonomous driving predominantly learn from successful demonstrations by minimizing
geometric deviations from expert trajectories. This paradigm implicitly assumes that spatial proximity implies behavioral safety, leading to a critical objective mismatch: trajectories with nearly identical imitation losses may exhibit drastically different safety outcomes, where one remains recoverable while the other results in collision. To address this limitation, we propose BeyondDrive, a failure-aware imitation learning framework that jointly learns from successful and failed driving behaviors. First, we introduce a flow matching-based negative trajectory generator that synthesizes safety-critical yet expert-proximate trajectories, enabling explicit modeling of safety asymmetry. Second, we develop a diversity-aware sampling strategy that mitigates mode collapse and improves coverage of diverse failure modes during negative trajectory generation. Third, we propose a Repulsive Distance Loss that simultaneously attracts predictions toward expert demonstrations while repelling them from hard negative trajectories, thereby establishing discriminative safety boundaries in trajectory space. Applied to the uni-modal baseline Latent TransFuser, BeyondDrive achieves 89.7 PDMS on the NAVSIMv1 closed-loop
benchmark, outperforming prior state-of-the-art methods. Moreover, BeyondDrive generalizes effectively across different autonomous driving architectures, including multi-modal planners, and further demonstrates strong zero-shot transferability on the HUGSIM
benchmark.
Read the full abstract ▾
All authors Junli Wang, Zhihua Hua, Xueyi Liu, Zebin Xing, Haochen Tian, Kun Ma, Hangjun Ye, Guang Chen, Long Chen, Qichao Zhang