Computer Vision
Disentangling Model and Human Data Uncertainty in Apparent Facial Age Estimation
Andrei Foitos, Ivo Pascal de Jong and Matias Valdenegro-Toro
Estimating the apparent age of individuals from facial images is challenging due to the subjective nature of perception and the inherent variability of the data. We investigate the role of uncertainty estimation, attributing uncertainty jointly to a lack of knowledge (epistemic) or inherent noise/chance (aleatoric). Leveraging the APPA-REAL
dataset, we train
Bayesian Neural Networks on
datasets of varying sizes using three BNN approximations: MC-DropConnect, Flipout, and Deep Ensembles using supervision on human aleatoric uncertainty available in the APPA-REAL
dataset. Each model outputs both the predicted apparent age and the amount of aleatoric and epistemic uncertainty. Our results confirm the hypothesis that the inherent aleatoric uncertainty remains stable across
dataset sizes, while epistemic uncertainty increases as training data decreases. These findings demonstrate that different sources of uncertainty can be quantified in face age estimation.
Read the full abstract ▾
Computer Vision
OmniStyle-INR: Universal and Multimodal Style Transfer for INRs
Rafał Kajca, Michał Miziołek, Kornel Howil et al. (+2 more)
Style transfer remains a fundamental and highly important task across various data modalities, enabling creative manipulation conditioned by both reference images and textual descriptions. Recently, methods utilizing Gaussian Splatting have emerged as a unified representation for 2D images,
video, 3D scenes, and 4D dynamics. However, representing
videos and 2D images with Gaussian Splatting is structurally sub-optimal for dense continuous domains. The number of required Gaussians often approaches the total number of pixels, raising questions about the actual utility of such a representation for these specific modalities. In contrast, Implicit Neural Representations have established themselves as a much more popular and natural choice across all these data domains. Implicit Neural Representations naturally provide significant advantages, including data compression, inherent capabilities for super resolution, and seamless integration with deep generative models. To this end, we introduce OmniStyle-INR, a novel framework that leverages network-based continuous representations as a truly universal domain. Our approach successfully performs high-quality style transfer across all visual modalities, guided seamlessly by both text
prompts and visual exemplars.
Read the full abstract ▾
All authors Rafał Kajca, Michał Miziołek, Kornel Howil, Rafał Tobiasz, Przemysław Spurek
Computer Vision
Spatial Transport of Integration Error in Generative ODEs
Songheng Yin
A trained flow or
diffusion model is usually run with only a handful of solver steps, and the integration error this leaves behind is unevenly distributed across the image. We ask where that error is injected and how it reaches the endpoint, and answer with a signed source-and-transport accounting of few-step integration error, tested to first order. A perturbation experiment on five models at 256^2 resolution shows the learned dynamics spread local disturbances widely: near the start of sampling, under 10% of the summed endpoint response remains at the source. Signed one-step truncation residuals, propagated through the model's own linearized dynamics, reconstruct much of the endpoint error's direction and regional structure (cosine 0.81-0.87), and a region's error owes more to what arrives from elsewhere than to its own injection. Structure-destroying nulls, with protocols frozen before evaluation, locate what carries the account: randomizing contribution signs halves it, and reassigning which region receives each contribution, with content, norms, and signs intact, destroys it entirely. Where the injections land is readable from the model itself. The variation of its velocity or prediction field along the trajectory, a structure that emerges during training, predicts the final per-region gap (within-image rho of 0.57-0.70 on fine trajectories, weaker from the cheap solve alone). The prediction is partial because endpoint error depends not only on injected magnitude but on its sign, timing, and transport through the learned dynamics. A training penalty on the injected variation lowers few-step error, so the structure is one a model can be trained to change.
Read the full abstract ▾
Computer Vision
PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation
Qirui Li, Jinkun Hao, Yibo Li et al. (+3 more)
Recent advances in physics-grounded
video generation leverage physics simulation as a physical prior to guide
video synthesis toward physically plausible outcomes. The simulation process is controlled by physical specifications, which are typically generated by a
vision-
language model in a single pass. Such one-shot prediction often fails to accurately translate user intent into executable simulations, particularly for fine-grained object dynamics, complex motion trajectories, and temporally structured interactions. In this paper, we propose PhysAgent, a reflective agentic framework that closes the loop among physical program generation, physics simulation, stage-specific verification, and targeted program repair. Beyond improving the control of coupled physical parameters, our framework enables the
agent to progressively realize complex trajectories, multi-stage interactions, and precise event outcomes by treating each physical program as an executable hypothesis. In addition, we design a set of physics-control APIs to support more stable and complex motion behaviors. Extensive experiments demonstrate that PhysAgent produces more physically plausible
videos, achieves better
prompt alignment, and generalizes more effectively across diverse physical scenarios.
Read the full abstract ▾
All authors Qirui Li, Jinkun Hao, Yibo Li, Ran Yi, Paul L. Rosin, Yu-Kun Lai
Computer Vision
Clarify Before Executing: A Self-Evolving Agent for Resolving Intent Asymmetry in 3D Tool Orchestration
Xiaoye Zhu, Weixin Li, Junan Huo et al. (+5 more)
A fundamental intent asymmetry plagues modern 3D asset creation: while state-of-the-art 3D toolchains demand precise, executable parameters, ordinary users typically provide vague, underspecified instructions. Current 3D
agents treat this ambiguity as noise, defaulting to blind execution under a single-turn assumption. To address this limitation, we introduce CLARE, a clarification-aware and evolutionary 3D
agent that treats intent asymmetry not as an execution error, but as an opportunity for strategic dialogue. By decoupling the generation pipeline into four specialized cognitive roles, CLARE intercepts and resolves underspecified instructions before invoking computationally expensive 3D tools to seamlessly execute tasks across five diverse domains: text-to-3D generation, single-view reconstruction, multi-view reconstruction, point cloud editing, and post-processing. Crucially, rather than relying on rigid manual rules, CLARE self-evolves its clarification policy via simulated multi-turn interactions. By optimizing a Multi-turn Reward, the
agent internalizes the delicate balance between interaction efficiency and task completion. To rigorously test this, we construct 3D-Clarify, a comprehensive
benchmark comprising 620 interaction scenarios with systematically injected ambiguity, missing information, and mistaken details. CLARE achieves state-of-the-art performance, with 60.40% and 43.34% success rates on single-step and multi-step tasks, respectively, more than doubling existing baselines. Both quantitative and qualitative results demonstrate that proactive clarification is the missing key to robust 3D execution. Code is available at https://github.com/xyzhu1225/CLARE.
Read the full abstract ▾
All authors Xiaoye Zhu, Weixin Li, Junan Huo, Bozhong Wang, Jia Zeng, Yi Yang, Cen Chen, Qi Liu
Computer Vision
Privacy-Aware Synthetic Video Benchmarking and Relational Evaluation for Worker-Under-Suspended-Load Detection
Anshu Singh and Alejandro Seif
Publicly shareable construction-
video benchmarks remain scarce, especially for safety-critical hazards that are rare, dangerous to stage, and difficult to release. We study worker under suspended load, a relational hazard that depends on worker-load geometry and temporal persistence rather than object detection alone. We introduce SynthSite, a focused synthetic
video benchmark of 55 clips spanning varied load configurations, viewpoints, clutter, occlusions, and surveillance conditions, together with a
privacy-aware hybrid generation workflow that supports both publicly shareable
benchmark creation and
privacy-constrained synthetic
video generation. We then ask whether worker appearance can be suppressed without undermining downstream hazard recognition. Under five whole-body
privacy conditions, we evaluate worker and load retention, localization stability, and clip-level hazard recognition. We find that structure-preserving obfuscations retain substantially more downstream utility than appearance-smoothing baselines, and that preserving a raw visual reference alone does not guarantee the strongest agreement with human hazard labels. These findings suggest that
privacy evaluation for construction safety analytics should assess not only appearance suppression, but also preservation of the
geometric cues required for hazard
reasoning. Our
dataset and code are available at https://huggingface.co/
datasets/govtech/SynthSite .
Read the full abstract ▾
Computer Vision
Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs
Like Liu, Zhengzheng Xu, Haitao He et al. (+3 more)
Multimodal large
language models have achieved strong performance across diverse
vision-
language tasks, yet their capabilities in
UAV scenarios remain insufficiently explored. Recent
UAV-oriented
benchmarks have begun to evaluate MLLMs in aerial scenarios, but they typically focus on scene understanding, event recognition, or
navigation completion, rather than jointly assessing the dual-cognition capability required for
UAV agents:
reasoning about both the
UAV's own state and the external environment in multiview spatio-temporal contexts. To address this gap, we present
UAV-DualCog, a
benchmark for aerial multiview spatio-temporal
reasoning built on this dual-cognition perspective.
UAV-DualCog includes both image and
video tasks to jointly evaluate self-state and environment-state
reasoning, while requiring spatial or temporal grounding beyond discrete answer prediction. We also develop an automated pipeline that constructs data from scene-level semantic point clouds, yielding a scalable
benchmark with diverse scenes, hundreds of landmarks, and thousands of QA samples. Extensive evaluations show that current MLLMs remain far from reliable in
UAV dual cognition. Self-state
reasoning, viewpoint transformation, precise spatial grounding, and temporal interval localization are persistent bottlenecks, and additional validation with thinking/frontier models and a human baseline confirms that the
benchmark is understandable to humans but challenging for existing models. We further construct
UAV-DualCog-Train from disjoint scenes and show through a lightweight optimization probe that it provides useful structured supervision, suggesting its value not only as an evaluation
benchmark but also as a data resource for advancing MLLM-based
UAV agents. Project website and supplementary materials: https://
uav-dualcog.lozumi.com
Read the full abstract ▾
All authors Like Liu, Zhengzheng Xu, Haitao He, Hongzhe Li, Shuchang Zhang, Dian Shao
Computer Vision
MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction
Homanga Bharadhwaj and Yash Jangir
Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary
monocular videos of human-object interaction. Given a short observed
video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion
forecasting without any assumptions on the object properties. Our key insight is that
video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained
video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large
video and tracking components. Using just 40k human
videos and no auxiliary inputs such as
language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training
videos. These results show that we can efficiently re-purpose
video priors into explicit
geometric forecasts for
embodied intelligence. https://motionforesight.github.io/
Read the full abstract ▾
Computer Vision
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
Hao Liu, Chenghuan Huang, Ye Huang et al. (+7 more)
Video Diffusion Transformers process long spatio-temporal sequences, making self-
attention the main bottleneck in high-resolution
video generation. Training-free sparse
attention reduces this cost, but adaptive Top-$p$ routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse
attention into a rank-level straggler problem. We present \method{}, a training-free sparse-
attention system that improves the distributed execution efficiency of adaptive sparse
attention under multi-GPU sequence parallelism. \method{} uses Top-$p$ routing, a Top-$k$ safety floor, and
video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, \method{} reduces average load imbalance from 1.34 to 1.08 and delivers a $4.41\times$
attention speedup over FlashAttention, while achieving a $2.02$–$2.11\times$ DiT inference speedup with competitive
video quality.
Read the full abstract ▾
All authors Hao Liu, Chenghuan Huang, Ye Huang, Zhiying Wen, Hao Liu, Mohan Zhang, Chen Li, Ziyang Ma, Jing Lyu, Jiangsu Du
Computer Vision
Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA
Ce Zhang, Ziyang Wang, Yulu Pan et al. (+6 more)
Grounded long-
video question answering (Grounded LVQA) requires answering a question about a long
video while localizing the short evidence interval that supports the answer. Recent agentic methods frame this task as multi-turn exploration with a single crop_
video(start, end) action, which supports coarse-to-fine narrowing but provides no primitive for fine-to-coarse backtracking. As a result, these
agents typically converge prematurely and cannot recover from an early mistake. We propose VideoTreeSearch (VTS), a framework that casts grounded LVQA as iterative self-correcting search over an adaptive temporal tree. VTS constructs a non-uniform tree from visual scene boundaries so that each node corresponds to a semantically coherent segment, and trains an
agent to navigate the tree through four discrete operations: zoom_in, zoom_out, shift, and answer. These operations expose backtracking and recovery as explicit, learnable primitives rather than implicit behaviors. To train this
navigation, we introduce a trajectory synthesis pipeline that produces multi-step paths through the tree, including deliberate detours into incorrect branches followed by recovery. We use these trajectories for supervised fine-tuning, followed by reinforcement learning with grounding and answer-accuracy rewards. On three Grounded LVQA
benchmarks (CG-Bench, Haystack-LVBench, Haystack-Ego4D), VTS outperforms the strongest prior agentic methods by +12.5 mIoU on CG-Bench and +7.4 T-F1 on Haystack-Ego4D. The learned policy also transfers to general long-
video QA, surpassing all prior agentic baselines on
Video-MME, MLVU, and LVBench by up to +7.1 accuracy points. Ablations confirm that self-correcting hierarchical search is the central mechanism behind these gains: removing either adaptive descent or explicit backtracking substantially degrades performance. Code is available at https://github.com/CeeZh/VTS.
Read the full abstract ▾
All authors Ce Zhang, Ziyang Wang, Yulu Pan, Oluwatumininu Oguntola, Pranav Wagh, Qiyu Wu, Hiromi Wakaki, Mohit Bansal, Gedas Bertasius
Computer Vision
Vision-Language Assistant for Emotional Reactions to Risky Driving
Harine Choi, Eun Hak Lee and Zhengzhong Tu
This study introduces a
vision-
language pipeline that detects risky driving behaviors and generates emotionally expressive responses to support driver awareness and comfort. Although
vision-
language models have advanced perception and
reasoning in autonomous driving, existing systems rarely consider the emotional dimension or real-world user experience. Keep Yelling Assistant (KYA) detects high-risk driving maneuvers in real time, such as sudden cut-ins. It then produces emotional responses through a large
language model tailored to driver preferences. The framework comprises two core modules. The
vision module uses YOLOv8 variants to detect nearby vehicles and identify risky behaviors such as sudden cut-ins. Key driving metrics, including relative distance, speed, and projected reach time, are extracted and normalized to produce a structured behavior log. The
language module processes this log with user-defined emotional tone settings, such as neutral, humorous, and analytical, and generates verbal reactions using state-of-the-art large
language models, including ChatGPT-4o, Claude 3, Gemini 2.5, and Copilot. We evaluated the proposed system using dashcam
videos containing risky driving behaviors and a user study involving 108 participants. Participants selected preferred response styles, and the large
language models were evaluated based on emotional
alignment. All models received favorable ratings, although preferences varied across personas. Notably, the combination of YOLOv8s and ChatGPT-4o achieved the highest score of 4.29 out of 5.00. By integrating real-world perception with emotionally adaptive dialogue, KYA introduces a new paradigm for emotionally intelligent in-vehicle artificial intelligence. It offers promising directions for improving safety, trust, and emotional well-being in both conventional and autonomous vehicles.
Read the full abstract ▾
Computer Vision
An Exam for Active Observers
Jiarui Zhang, Muzi Tao, Shangshang Wang et al. (+3 more)
Human
vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's
multimodal large
language models (MLLMs) exercise active observation is an empirical question that current
vision-
language benchmarks do not answer. We introduce ActiveVision, a
benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception rather than a single static description. Frontier MLLMs collapse on ActiveVision: the highest-scoring model we evaluate, GPT-5.5 at the highest exposed
reasoning-effort tier, solves only 10.6% of items and scores zero on 11 of the 17 tasks, and even Claude Fable 5, despite topping most
reasoning and coding leaderboards, solves just 3.5%, far behind three human participants who average 96.1%. Furthermore, much of the gap persists even when models write and run their own
vision code: such code is unreliable on realistic imagery, and catching its failures itself requires the active perception the models lack. Together, these results indicate that current MLLMs lack robust active visual observation, motivating architectures and training objectives that close the perception-
reasoning loop.
Read the full abstract ▾
All authors Jiarui Zhang, Muzi Tao, Shangshang Wang, Ollie Liu, Xuezhe Ma, Willie Neiswanger
Computer Vision
CLIFE: Camera-LiDAR Fusion Framework for Edge-Deployable Roadside VRU Perception
Tam Bang, Hoang H. Nguyen, Lei Cheng et al. (+6 more)
Reliable roadside perception of vulnerable road users (VRUs) remains challenging under occlusions, variable lighting, and diverse weather conditions, particularly under strict edge-computing and latency constraints. Existing multi-sensor fusion systems rely on cloud or server-grade infrastructure, creating a deployment gap at real-world intersections. We present CLIFE, an edge-native camera-
LiDAR fusion framework that integrates targetless online calibration and lightweight late-fusion tracking entirely on a single embedded device, without cloud offloading. CLIFE adaptively refines camera-
LiDAR alignment on demand and performs multi-sensor fusion and track association with O(N log N) per-frame cost. We deploy CLIFE across 12 signalized intersections in Chattanooga and conduct an in-depth evaluation at a representative intersection using synchronized camera-
LiDAR data that spans diverse daytime, nighttime, and weather conditions. Our experiments demonstrate that the fusion architecture substantially enhances the perceptual range and robustness of the individual sensors under varied environmental and traffic conditions. The late-fusion core operates at 53.2 FPS on the Jetson AGX Thor, ensuring high throughput for real-time intersection-scale applications. By centering perception at the edge, CLIFE provides a deployable foundation for downstream safety applications, while reducing bandwidth and calibration overhead for agencies operating multi-intersection corridors.
Read the full abstract ▾
All authors Tam Bang, Hoang H. Nguyen, Lei Cheng, Lihao Guo, Siyang Cao, Hussam Abubakr, Tianya Zhang, Austin Harris, Mina Sartipi
Computer Vision · Cross-listed from cs.RO
VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds
Zhiyuan Wu, Zhuo Chen and Shan Luo
Vision and touch are complementary modalities essential for robotic perception and manipulation. While
vision provides global object context, touch offers precise local information at contact points. Integrating these modalities for contact localization, i.e., predicting the location of touch on an object's surface, poses significant challenges due to the need for accurate spatial
alignment between tactile data and visual geometry. To address this challenge, we propose VTLoc, a novel visual-tactile framework that localizes contact points from tactile readings using a 3D point cloud as visual input. VTLoc introduces two key components: a
geometric multi-modal
alignment module, which reconstructs a pseudo-point cloud from fused visual-tactile features and aligns it with the visual point cloud to enforce spatial consistencies across modalities; and an iterative localizing updater, which iteratively refines the predicted contact location using fused visual-tactile features. Evaluated on a new
benchmark of 100 real-world objects, VTLoc improves single-touch contact localization by reducing local-to-global correspondence ambiguity.
Read the full abstract ▾
Computer Vision
Toward Semantic Communication for Real-time Mobile 3D Reconstruction
Fangzhou Zhao, Yao Sun, Xuesong Liu et al. (+3 more)
Real-time mobile 3D reconstruction is fundamental to many emerging applications such as autonomous
navigation and digital twin construction, where a moving platform continuously captures an image stream and transmit to a computing server for scene understanding. Unlike offline reconstruction, camera poses and scene geometry are estimated on-the-fly during acquisition, making multi-view consistency a real-time requirement and rendering
geometric estimation highly sensitive to communication-induced distortions. Semantic communication (SemCom) transmits compact semantic information, offering a promising way to preserve task-critical data over unreliable links. However, existing designs are optimized at the image or single-view level and without providing explicit reliability information for
geometric estimation, limiting their applicability to real-time mobile 3D reconstruction. In this context, we propose a SemCom framework for real-time mobile 3D reconstruction. The framework includes a semantic transceiver that outputs a reconstructed image alongside a pixel-wise confidence map, quantifying the reliability of each region. We further introduce a confidence-guided
geometric estimation method, incorporating confidence into RANSAC-based pose initialization and bundle adjustment to reduce the influence of unreliable regions and enhance robustness under noisy channels. Simulations show that, compared to existing SemCom and traditional seperate source and channel coding, our framework maintains high image quality while significantly improving pose estimation accuracy and 3D structural consistency.
Read the full abstract ▾
All authors Fangzhou Zhao, Yao Sun, Xuesong Liu, Runze Cheng, Shang Kai, Yi Sun
Computer Vision · Cross-listed from eess.AS
Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos
Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar et al. (+19 more)
We present Audio-Visual Flamingo (AV-Flamingo), a fully open state-of-the-art audio-visual large
language model (AV-LLM) for joint understanding and
reasoning over audio, images, and long-form
videos. Unlike prior AV-LLMs that primarily focus on short clips, AV-Flamingo is designed for understanding and
reasoning over long and complex real-world (audio-visual)
videos. To support this, we make three key contributions: (i) Audio-Visual-Skills, a large-scale collection of real-world
videos with ~7M caption and question-answer training instances designed to emphasize temporal, compositional, and cross-modal audio-visual
reasoning; (ii) a novel three-stage curriculum that progressively trains the model from short-range perception to long-horizon multi-event
reasoning; and (iii) Temporal Audio-Visual Interleaved Chain-of-Thought, a
reasoning framework that explicitly grounds intermediate
reasoning steps to timestamps in long audio-visual streams, improving temporal
alignment and interpretability. Extensive experiments across 15+ audio-visual, omni-modal, audio, and
vision benchmarks show that AV-Flamingo outperforms similarly sized open models by clear margins and remains highly competitive with, and in some cases surpasses, much larger open-weight and closed models, particularly on long and complex real-world audio-visual understanding and
reasoning tasks. Beyond
benchmark performance, AV-Flamingo exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability.
Read the full abstract ▾
All authors Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar, Lasha Koroshinadze, Nishit Anand, Siddharth Gururani, Hanrong Ye, Pritam Biswas, Yuanhang Su, Ehsan Hosseini-Asl, Sang-gil Lee, Zhifeng Kong, Jaehyeon Kim, Sungwon Kim, S Sakshi, Ramani Duraiswami, Dinesh Manocha, Andrew Tao, Mohammad Shoeybi, Bryan Catanzaro, Ming-Yu Liu, Wei Ping
Computer Vision
Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs
Maeve Hutchinson, Abderrahmane Wassim Mehdaoui and Pranava Madhyastha
Understanding how
vision-
language models (VLMs) interpret data visualizations remains an open problem, and is increasingly important as these models are used for analytical tasks where reliable
reasoning is essential. We introduce a lightweight, diagnostic saliency map method tailored for text generation over images using
transformer models, the current state-of-the-art models in visualization interpretation. Our approach aggregates the
language model's
attention over the visual
tokens across all heads and layers, then maps this
attention back onto the
vision encoder's patch grid to localise it over the image, producing a direct correspondence between each generated answer
token and the image regions it attended to. This yields fast, gradient-free saliency maps that expose how VLMs allocate focus across visual elements during answer generation, enabling inspection of whether model
attention aligns with semantically relevant components. We evaluate our approach using a deletion metric which validates the causal faithfulness of our saliency maps to the model's behavior.
Read the full abstract ▾
Computer Vision
How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA
Navya Gupta, Bingjie Xu, Avinash Anand et al. (+2 more)
Compositional visual question answering requires
Vision-
Language Models (VLMs) to execute multiple
reasoning operations like object selection, spatial relation resolution, and attribute verification. Despite strong aggregate performance, the mechanistic basis of VLM failures on this task remains underexplored. To address this gap, we analyze
vision-operation misalignment in VLMs by examining how failures relate to specific
reasoning operations and the internal computational pathways through which they arise and propagate. We introduce an Operation-centric mechanistic framework that decomposes VLM failures by both the
reasoning operation where they originate and the internal computational pathway through which they propagate. Our analysis reveals four dominant failure modes: grounding failure,
reasoning failure, attribute extraction failure, and
language-prior dominance, each characterized by a distinct relationship between visual grounding strength and answer correctness. Through three complementary causal interventions applied across all
transformer layers, we find that object-selection failures are associated primarily with feedforward computation, multi-step relational failures with late-layer direct
attention, and attribute-extraction failures with answer-position feedforward computation. Validation on VSR further shows that single-step spatial failures are concentrated at object-position encoding, distinguishing them from multi-step relational composition. These findings reveal distinct computational bottlenecks across operation types and provide a principled basis for targeted diagnosis of VLM failures in multimedia
reasoning.
Read the full abstract ▾
All authors Navya Gupta, Bingjie Xu, Avinash Anand, Timothy Liu, Zhengchen Zhang
Computer Vision
Adaptive Contrast Enhancement and Optimised Feature Matching for RootSIFT-Based Palm-Vein Recognition
Kaveen Perera, Fouad Khelifi and Ammar Belatreche
Palm-vein recognition is a highly secure biometric modality due to the uniqueness and subcutaneous nature of vein patterns. However, low contrast in palm-vein images, caused by NIR light scattering and sensor limitations, remains a significant challenge. To address this, we propose the Intensity-Limited Adaptive Contrast Stretching with Bidirectional Gaussian-weighted Overlapping Tiles (ILACS-BGOT) method, an enhancement of the previously developed ILACS with Layered Gaussian-weighted Overlapping Tiles (ILACS-LGOT) technique. ILACS enhances local contrast, while BGOT mitigates blocky artefacts. This study further integrates RootSIFT features with KNN+RT and incorporates the previously introduced Mean and Median Distance (MMD) filter to investigate the parameter variations of both MMD and RT, and their impact on recognition performance. A comprehensive analysis was conducted across three
benchmark datasets (CASIA, PolyU, and PUT), using 42 combinations of MMD filter thresholds and RT values. Results were evaluated using EER and Accuracy. Findings reveal that higher template sizes improve performance, while varying MMD thresholds reflect
dataset-specific rotational variations. The proposed system demonstrates superior generalisability, achieving significant improvements in both EER and Accuracy over existing methods. Furthermore, the underlying ILACS-BGOT mechanism suggests potential applicability beyond palm vein recognition to other biometric modalities such as finger vein and palmprint recognition, and more generally to low-contrast image enhancement across computer
vision applications.
Read the full abstract ▾
Computer Vision
HCIG: A Hierarchical Cross-Modal Incongruity Graph Network for Multimodal Sarcasm and Cyberbullying Detection
Bhavana Verma, Priyanka Meel and Dinesh Kumar Vishwakarma
Multimodal sarcasm and cyberbullying detection remain challenging because the intended meaning often emerges from incongruity between textual and visual information rather than from either modality alone. Existing
multimodal approaches primarily rely on feature fusion or cross-modal
attention, which may not effectively capture hierarchical semantic inconsistencies across different levels of representation. To address this limitation, this paper proposes HCIG (Hierarchical Cross-modal Incongruity Graph Network), a novel framework that models cross-modal incongruity at
token, phrase, and global levels using graph
attention networks and adaptively integrates these representations through a learned hierarchical
attention mechanism. As a complementary architecture, we also introduce GCCN (Graph-based Cross-modal Contradiction Network), which performs graph-based
reasoning using contradiction-aware pooling for efficient
multimodal interaction learning. The proposed models are evaluated on the MMSD sarcasm
benchmark and the MultiBully cyberbullying
dataset, together with comprehensive ablation studies and cross-task transfer experiments. Experimental results demonstrate that HCIG achieves the best performance on MMSD with 85.74% accuracy and 85.29% macro-F1, while GCCN attains the highest macro-F1 (68.66%) on MultiBully and HCIG achieves the highest accuracy (69.62%) and bullying-class F1 (74.90%). The findings demonstrate that hierarchical multi-granularity incongruity modeling provides more effective
multimodal reasoning than conventional fusion strategies, offering a robust framework for sarcasm and cyberbullying detection in social media.
Read the full abstract ▾
Computer Vision
Spatial Normalization for Cross-Domain Retinal Layer Segmentation in Optical Coherence Tomography
Iker Moran-Cavero, Monica Hernandez, Elvira Mayordomo et al. (+4 more)
Retinal layer
segmentation in Optical Coherence
Tomography (OCT) is a fundamental step for extracting quantitative biomarkers of retinal structure. Indeed, there is a growing interest in the analysis of OCTs in the context of neurodegenerative diseases. However,
segmentation remains challenging due to speckle noise, shadowing artifacts, low contrast between adjacent layers, anatomical variability across subjects, and domain shifts arising from different acquisition protocols and clinical populations. While deep learning methods have achieved remarkable performance, their robustness and generalization across heterogeneous
datasets remain limited. In this work, we investigate the role of spatial normalization as a preprocessing strategy to mitigate
geometric domain shifts and improve the consistency of retinal layer
segmentation. Inspired by standard practices in neuroimaging, we introduce a fovea-centered normalization framework that aligns OCT volumes into a common anatomical reference. We perform a comprehensive evaluation of state-of-the-art deep learning architectures. To provide a comprehensive assessment of
segmentation quality, we combine conventional overlap-based metrics at B-scan level with
topology-aware metrics at A-scan level and thickness-based measures at the en-face level. In cases where a ground truth is not available, we propose
topology violation quantitative metrics that do not require ground truth annotations and a thickness-based qualitative assessment that captures structural consistency and clinically relevant patterns at the en-face level. The results demonstrate the importance of spatial normalization in OCT
segmentation pipelines toward the development of robust and clinically meaningful retinal analysis tools, enabling reliable biomarker extraction and downstream computational analysis in neurodegenerative research.
Read the full abstract ▾
All authors Iker Moran-Cavero, Monica Hernandez, Elvira Mayordomo, Naiara Artiaga, Beatriz Pardiñas, Beatriz Cordon, Elena Garcia-Martin
Computer Vision
ArtChart: Faithful Artistic Chart Generation with Integrated Text Rendering
Meijia Huang, Yingjie Yin, Shihao Wang et al. (+1 more)
Artistic charts combine data visualization with expressive marks, textures, and typography, but they are difficult for image generators: an output is useful only when its stylization preserves chart geometry, exact in-image text, and the semantic binding between labels and marks. We introduce ArtChart, a framework for faithful artistic chart generation with integrated text rendering. Given a structured chart specification and an artistic
prompt, ArtChart first renders a text-free grayscale layout that encodes the target chart geometry, then trains a chart-specific control module to preserve mathematical structure. To address the remaining text and layout errors, we further refine the generation policy through GRPO-based reinforcement learning with OCR-based text rewards, VLM-based layout rewards, and aesthetic rewards. A multi-expert
distillation stage reconciles these objectives by distilling single-reward experts into one balanced generation policy. We also construct ArtChart-Bench, a bilingual 2K-
prompt benchmark covering four chart types, controlled value distributions, diverse label/value formats, and 15 artistic styles, together with ArtChart-Eval, a six-axis evaluation protocol measuring mathematical logic, text accuracy, text layout, aesthetics, instruction following, and readability. Experiments on ArtChart-Bench show that ArtChart consistently outperforms
prompt-only, image-editing, and generic ControlNet baselines, with the largest gains on mathematical fidelity and label-layout binding while maintaining competitive visual quality. These results suggest that artistic chart generation should be evaluated as reliable visual communication rather than as generic stylized image synthesis.
Read the full abstract ▾
All authors Meijia Huang, Yingjie Yin, Shihao Wang, Chenguang Ma
Computer Vision
Multi-Modal Semantic Segmentation of Electrolyzer Components for Sustainable Hydrogen Technologies: A Dual-Branch Deep Learning Approach
Wasimul Karim, Nur Mohammad Fahad, Abdul Hasib Siddique et al. (+4 more)
Accurate
segmentation of electrolyzer materials is essential for automated disassembly, sustainable recycling, and circular manufacturing in hydrogen technologies. However, this task is challenging due to strong visual similarity between materials, spectral overlap, irregular shapes, and severe class imbalance. To address these challenges, we propose an AI-driven dual-branch framework, Hyperspectral-RGB Electrolyzer Materials Network (HREM-Net), that combines hyperspectral imaging (HSI) and RGB images for electrolyzer material
segmentation. We implemented several innovative modules, including Efficient Channel
Attention, Coordinate
Attention, Mobile Inverted Bottleneck blocks, and Atrous Spatial Pyramid Pooling to capture spectral and spatial features from HSI, and RGB images. With an adaptive gated cross-modal fusion module and composite loss function, HREM-Net achieves a mean class accuracy of 91.66% and a mean Intersection over Union (mIoU) of 0.82 on the Electrolyzers-HSI
dataset, outperforming baseline
segmentation models. Cross-
dataset validation on the PCB-
Vision dataset demonstrates strong generalization with 96.91% accuracy and 0.93 mIoU. This work poses its potential as an industrial application to improve electrolyzer efficiency, thereby improving the predictive maintenance of hydrogen production.
Read the full abstract ▾
All authors Wasimul Karim, Nur Mohammad Fahad, Abdul Hasib Siddique, Md Rafiqul Islam, Hooman Mehdizadeh-Rad, Asif Karim, Sami Azam
Computer Vision
PIXIE: A Zero-Shot texture-invariant 6D pose estimation framework for unseen objects with assembly defects
Leon Jungemeyer, Alejandro Magaña, Gautham Mohan et al. (+2 more)
6D pose estimation remains a key challenge in robotics and computer
vision, particularly in industrial environments. The deployment of currently available data-driven methods is often limited by resource-intensive data pipelines, reliance on textured 3D models, and sensitivity to
geometric deviations caused by damages or assembly defects. We present PIXIE, a zero-shot framework that estimates the 6D pose of an object from an RGB image using only an untextured 3D model. Synthetic depth and normal maps are rendered from sampled reference viewpoints and matched to the query image via a pretrained cross-modality feature matcher. Matched keypoints are back-projected to obtain 2D–3D correspondences for PnP-based pose estimation. Relying exclusively on geometry makes the method inherently robust to lighting and texture variation, while correspondence filtering handles
geometric deviations between the model and physical object. We evaluate on widely-used public
benchmarks, reporting state-of-the-art results on texture-less objects without object-specific training, and introduce a novel
dataset with assembly defects, texture variations, and occlusion to demonstrate real-world applicability.
Read the full abstract ▾
All authors Leon Jungemeyer, Alejandro Magaña, Gautham Mohan, Matthias Karl, Daniel Werdehausen
Computer Vision
DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction
Jehun Kang, Jungha Wang, Youngjun Hwang et al. (+1 more)
Multi-Task Learning (MTL) in robotics perception systems supports comprehensive 3D spatial scene understanding by integrating semantic
segmentation and depth estimation. While
Vision Foundation Models (VFMs) are increasingly adopted as robust feature encoders, existing decoding strategies present a critical bottleneck. To address this, we propose DPNeXt, a streamlined multi-scale feature fusion decoder and efficient alternative to the standard Dense Prediction
Transformer (DPT). DPNeXt uses dual depthwise separable inverted bottlenecks to improve frozen VFM utilization through fusion-centric decoding and independent task modularization. To further mitigate negative inductive transfer between tasks, we introduce the Multi-Task Boundary Guidance (MTBG) strategy. Unlike prior boundary-aware methods that add fusion modules or gating, MTBG applies symmetric boundary-focused supervision to encourage
geometric consistency without extra annotation or inference cost. Experiments on Cityscapes show that DPNeXt-S outperforms prior state-of-the-art (SOTA) MTL models, while DPNeXt-B further improves the overall performance and achieves the best results among the compared methods. On NYUv2, DPNeXt-B also achieves the best semantic
segmentation and depth estimation results among the compared methods while requiring substantially fewer trainable parameters than prior large-scale MTL models. Compared with the standard DPT, DPNeXt-S reduces trainable parameters by 78.6% and achieves the fastest inference speed among the compared models on resource-constrained laptop hardware. The source code, model checkpoints, and a demo
video will be made available at https://github.com/kangjehun/DPNeXt.
Read the full abstract ▾
All authors Jehun Kang, Jungha Wang, Youngjun Hwang, David Hyunchul Shim
Computer Vision
Beyond Unfolding: 60x Faster One-Stage Unmixing for Closely-Spaced Infrared Small Targets
Ximeng Zhai, Zheng Wang, Yaohong Chen et al. (+3 more)
Due to the optical diffraction limit and long imaging distances, Closely-Spaced Infrared Small Targets (CSIST) typically exhibit energy overlap, manifesting as indistinguishable blobs in infrared images. This ambiguity invalidates the one-to-one mapping assumption of traditional detection, thereby necessitating a paradigm shift towards CSIST Unmixing, which decomposes these blobs into discrete sub-targets. However, the dominant paradigm deep unfolding networks are shackled by the high latency and structural inflexibility intrinsic to their repetitively iterative architecture. To this end, we propose the Fast One-stage CSIST Unmixing Scheme (FOCUS), a one-stage lightweight paradigm which demonstrates that deep unfolding is not necessary. Motivated by the key observation that image super-resolution (SR) and CSIST Unmixing share an isomorphic degradation model, our insight is that it is possible to achieve a paradigm shift from image SR to CSIST Unmixing via completely transforming the label space, loss functions, and evaluation criteria. Specifically, to avoid entangling
geometric recovery with artifact suppression, FOCUS adopts a single pass mapping with an internal coarse-to-fine flow that progressively refines target localization from coarse spatial distributions to finer sub-pixel precision. While sparsity regularization suppresses background clutter, it also attenuates target intensities. To compensate for this attenuation of valid signals, flux conservation is introduced as a competing constraint that restores signal energy back to target centers. To the best of our knowledge, this work is the first attempt to address this task via a lightweight one-stage framework without the DUN paradigm. Experiments demonstrate that our method matches or surpasses the state-of-the-art unfolding approaches in both localization and unmixing accuracy, while boosting the inference speed by 60x.
Read the full abstract ▾
All authors Ximeng Zhai, Zheng Wang, Yaohong Chen, Hao Wang, Ming-Ming Cheng, Yimian Dai
Computer Vision
CanonicalPhys: Pose-Robust Remote Photoplethysmography via Canonical-Space Priors
Hui Wei, Seyedata Jodeiri Seyedian, Xiaobai Li et al. (+1 more)
Deep remote photoplethysmography (rPPG) attains sub-bpm heart-rate error on frontal, stationary faces yet degrades sharply under head pose: on MMPD, the state-of-the-art FactorizePhys backbone's MAE grows $1.60\times$ from frontal ($|\text{yaw}|{<}15^\circ$) to large-yaw ($|\text{yaw}|{\geq}45^\circ$) frames. We argue that pose is a coordinate-structural nuisance rather than a data-augmentation problem: in image coordinates the same pixel maps to different anatomy at different poses, blocking three priors otherwise natural for rPPG, namely the dichromatic reflection model, pulse-phase invariance across skin regions, and the POS/CHROM chromaticity projection, each of which presumes a stable anatomy-to-pixel mapping. We introduce CanonicalPhys, which prepends a differentiable four-point homography that fixes four facial anchors at canonical positions; in this canonical frame the three priors become expressible as a per-pixel Lambertian weight, a cross-ROI temporal consistency loss, and knowledge
distillation from windowed POS, none of which adds trainable parameters over the backbone. At an identical parameter count, CanonicalPhys reduces MMPD's frontal-to-large-yaw MAE degradation from $1.60\times$ to $1.33\times$ and flattens the mild-yaw bin from $1.32\times$ to $1.07\times$ (across CanonicalPhys variants), with matched cross-
dataset MAE reductions of up to $32\%$ on pose-rich targets. Code: https://github.com/infraface/CanonicalPhys
Read the full abstract ▾
All authors Hui Wei, Seyedata Jodeiri Seyedian, Xiaobai Li, Guoying Zhao
Computer Vision · Cross-listed from cs.LG
DebrisTracer: Reliable Tracking in Hypervelocity Impact Fast Imaging
Théophane Loloum, Fabien Vivodtzev, David Hébert et al. (+3 more)
This application paper presents DebrisTracer, a framework for the reliable tracking of debris in hypervelocity impact fast imaging. These noisy and highly specific
datasets capture the ejection of a large number of debris fragments after the impact of a projectile launched at hypervelocity into a target material. The reliable estimation of debris mass and speed distributions is of major importance in aerospace applications. We document how to extend an off-the-shelf
topology tracking framework based on critical point extraction and matching, in order to incorporate domain knowledge and physical assumptions. Our approach automatically produces an accurate and reliable debris tracking, enabling an interpretable visual analysis of this complex space-time phenomenon. Extensive experiments demonstrate the accuracy improvements provided by our approach over established tools used by domain experts in terms of physical validation, specifically via the prediction of the experimental ejected mass and crater depth profiles. We illustrate the utility of our approach across several use cases (with varying impact angles and physics). We show that our statistical summaries enable the visual identification of distinct regimes within the debris population, corroborating and refining prior expectations of domain experts. Our database and our C++ implementation are available at this address: https://github.com/tloloum/DebrisTracer.
Read the full abstract ▾
All authors Théophane Loloum, Fabien Vivodtzev, David Hébert, Baptiste Reynier, Michel Arrigoni, Julien Tierny
Computer Vision · Cross-listed from cs.RO
Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection
Hadrien Crassous, Mohamed Yassine Kabouri, Minahil Raza et al. (+2 more)
This paper studies how to adapt a computer
vision object detector to an unknown environment under both a
robot navigation time and annotation budget constraint. Our approach selects informative
robot trajectories and image samples to retrain the detector, explicitly targeting its failure cases. Formally, the approach is an
embodied variant of batch active learning, where at each round an
agent has a limited
navigation budget to collect candidate samples and a limited annotation budget for the most relevant images. We leverage spatial consistency to identify images with inconsistent labels, which are likely to provide the greatest improvement to the
vision model. We evaluate the approach using different active learning objectives on large scenes from the AI2-THOR simulator and on a real-world setup using a Boston Dynamics Spot
robot with the real-time object detector YOLOv5. Through comparison against several baselines, our experimental results show that spatial inconsistency helps guide the
agent and select relevant images without external supervision, achieving the highest detection accuracy at the end of the adaptation process under the same budget. The open-source project can be found at https://mkabouri.github.io/
embodied-active-learning-od
Read the full abstract ▾
All authors Hadrien Crassous, Mohamed Yassine Kabouri, Minahil Raza, Joni Pajarinen, Riad Akrour
Computer Vision · Cross-listed from cs.GR
Rendering 3D Gaussians on a Graph Processor
Nicholas Fry, Ignacio Alzugaray, Mark Pupilli et al. (+2 more)
We present the first implementation of a 3D Gaussian renderer on an Intelligence Processing Unit (IPU), comprising 1,472 independent tiles with only on-chip SRAM; constraints that approximate properties of efficient sensor-processor architectures. Our input scenes are 3D Gaussian maps from real-world sequences. Each tile 'owns' a screen-space region of the framebuffer; Gaussian primitives are routed to destination tiles via Manhattan-distance hops on a north-east-west-south (NEWS) grid, then distributed to overlapping neighbours in an expanding tree pattern. Computation follows the IPU's Bulk Synchronous Parallel (BSP) model, with inter-tile communication defined at compile time. We show this hardware allows us to exploit spatial and temporal locality by enabling local data transfer between cores. We evaluate the bottlenecks in this SRAM-only implementation: inter-tile bandwidth, per-tile SRAM capacity, and workload imbalance from non-uniform Gaussian density. We analyse how these constraints affect performance and render quality. This exploration raises broader questions for conventional GPUs and 3D representations, suggesting that direct inter-SM (streaming multiprocessor) communication might offer ways to reduce DRAM access in GPU kernels. We discuss these implications for the future of on-sensor and DRAM-free architectures. Project page: https://nmjfry.github.io/ipu-3dgs/
Read the full abstract ▾
All authors Nicholas Fry, Ignacio Alzugaray, Mark Pupilli, Paul H. J. Kelly, Andrew J. Davison