Computer Vision · Cross-listed from quant-ph
QiT: Quantum-Inspired Transformer for Visual Recognition Task
Badri N. Patro and Vijay Agneeswaran
Quantum machine learning offers a compelling representational perspective: angle-encoded states inhabit
Hilbert spaces in which periodic similarities and interactions can be expressed naturally. Realizing this perspective for visual recognition remains difficult, however, because present
quantum neural networks are constrained by limited
qubit counts, costly
circuit simulation and measurement, noise, and unstable optimization on noisy intermediate-scale
quantum devices. We investigate whether useful structural ideas from
quantum models can instead be realized as scalable classical
Transformer operations. We introduce QiT, a
Quantum-inspired
Transformer for
vision tasks with three components: (i) angle-inspired encoding that maps image
tokens to learned trigonometric
Hilbert-space features analogous to
quantum rotation-based state encoding; (ii) self-
attention over these periodic features, inducing a classical cosine kernel approximated to
quantum fidelity kernels; and (iii) gated multiplicative emulation, a trainable classical surrogate for interaction terms found in variational
circuits. All components are differentiable tensor operations, so QiT claims neither
quantum computation nor
quantum speedup and retains the $\mathcal{O}(N^2D)$
attention complexity of a standard
Vision Transformer. Across image-classification
benchmarks, QiT is competitive with a matched classical
Transformer while avoiding the severe runtime cost observed for a small simulated
quantum Transformer. QiT-B reaches 78.3% ImageNet-1K top-1 accuracy with 45.7M parameters and 11.5 GFLOPs. These results position QiT as a scalable baseline for isolating and evaluating
quantum-motivated inductive biases in visual recognition.
Read the full abstract ▾
Computer Vision
Beyond Performance Metrics: Uncertainty Mapping of Label Ambiguity in Fazekas Score Prediction
Susanne Schmid, Johanna Ospel, Richard Frayne et al. (+1 more)
Reference labels used to train medical image classification models are not always as certain as they may appear, and this uncertainty has implications on performance metrics. In this study, we propose a framework to analyze model performance for periventricular Fazekas score prediction that goes beyond conventional metrics. The Fazekas score is an ordinal visual rating scale used to assess the severity of white matter hyperintensities and is known to be affected by inter-rater variability. While the best Fazekas score prediction model achieved a Matthews correlation coefficient (MCC) of 0.70, performance varied across data splits and loss functions, making interpretation of model capabilities difficult. Rather than interpreting epistemic uncertainty of a model's prediction as an isolated scalar value, our approach of uncertainty mapping relates uncertainty to its position within the learned feature representation. This highlights regions of class-boundary transitions where cases appear more ambiguous and misclassifications are more likely. It also identifies potential label disagreement, including low-uncertainty misclassified cases that expert review found to be inconsistent with the original reference Fazekas score. Therefore, uncertainty mapping allows model behaviour to be examined in relation to class separation and potential model-label disagreement. Loss function choice also influenced the uncertainty profile, with some models showing clearer class separation and more localized uncertainty in ambiguous regions than others. These findings suggest that uncertainty mapping for Fazekas score predictions can support model interpretation and targeted
dataset review when reference labels are affected by ambiguity/ inter-rater variability.
Read the full abstract ▾
All authors Susanne Schmid, Johanna Ospel, Richard Frayne, Roberto Souza
Computer Vision
How to make effective use of domain experts for image classification?
Dieu-Donné Fangnon, Diane Lingrand, Aurélie Liard et al. (+3 more)
A lot of expectations have been put for years on integrating domain expert knowledge in image classification models. Several approaches have been explored, Concept Bottleneck Models (CBMs) opened up a new avenue of research leading to many variants, and more recently to Concept-based
Embedding Models (CEMs). CBM consider binary encoding of each concept, while CEM expands this idea by
embedding each concept through two vectors. However in real-life scenarii, domain experts' knowledge is usually organized in concepts determined by various attributes, each attribute encoded either with numerical values, or range of values, or binary values, or categorical values. In this work, we first finetune an image feature extractor for classifying attributes representing the downstream object classes, where the class attributes have been specified by experts under various encoding formats. A classification head is then learnt from these various attributes to categorize target objects. We experimentally show that it improves the classification for three
datasets: Kaggle fish
dataset, AWA2 and a more challenging new wood charcoal
dataset. We then propose an automatic selection of potential missclassified data. In this second step, experts are asked for those data to eventually modify the predicted attributes in order to improve the classification.
Read the full abstract ▾
All authors Dieu-Donné Fangnon, Diane Lingrand, Aurélie Liard, Marco Corneli, Antoine Pasqualini, Frédéric Precioso
Computer Vision
Geometry-Driven Shadow Harmonisation for Composited Faces: A Multiplicative, Albedo-Preserving Relighting Pipeline
Vijesh KP
Face swapping and face compositing pipelines routinely produce a face that is geometrically well aligned but photometrically implausible: the donor face carries flat, near-frontal studio illumination while the host body and background carry directional scene light. Most existing remedies re-synthesise the face through colour transfer, neural relighting, or inverse rendering, and therefore risk altering identity, skin tone, and texture. We present a conservative alternative: geometry-driven form-shadow injection. The pipeline never repaints the face. It estimates a per-pixel gain field $g\in[g_{\min},1]$ from a rasterised 3D face proxy and multiplies it channel-uniformly onto linear RGB, so the operator can only darken and cannot shift chromaticity. A dense landmark mesh is rasterised into a depth buffer, from which we derive surface normals, a cavity term, and screen-space cast shadows. Key-light direction is estimated from host-side cues (body, background, hair
halo); on-face cues are downweighted because they recover the donor's lighting. Shadow magnitude is not matched to the host: it is set by a three-parameter transfer $(\tau,\sigma,g_{\min})$. The shading field is divided by its 75th percentile over skin, then gated, scaled, clamped, smoothed, and re-clipped inside a feathered, skin-gated face mask. On an analytic face heightfield, the default $(\tau,\sigma,g_{\min})=(0.90,0.45,0.82)$ modifies 56.5% of face pixels with mean gain 0.938 (0.890 on modified pixels) and drives 3.4% of pixels to the floor. Hue invariance is a corollary of the operator. We analyse the transfer in closed form, ablate its parameters, and discuss failure modes of a monotone, darkening-only formulation, including double-shadowing of non-flat donors.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
Dingli Liang, Yiqiao Xie, Yukai Huang et al. (+6 more)
Wearable assistants require episodic
memory over
egocentric video, yet current
vision-
language models face bounded frame budgets, growing visual-
token costs, and long-context
retrieval failures. Under these practical constraints, we study whether textual captions can serve as reusable episodic
memory. We define the Episodic
Memory Video Caption QA task and introduce CapMem, a human-annotated
benchmark with 75
videos totaling 33.7 hours, and 1,000 multiple-choice questions across 16 scenarios. On long
videos (>20 min), full-coverage CaptionQA with 30s and 60s caption windows outperforms direct VideoQA for 10/12 and 8/12 models, respectively. On the same
video subset, a matched-frame control across six Qwen models retains mean accuracy gains of 3.22 and 2.55 points, respectively. Our caption-guided retrieve-and-verify harness further improves accuracy by up to 5.3 points. These results support the effectiveness of caption
memory for episodic
reasoning over long
egocentric video.
Read the full abstract ▾
All authors Dingli Liang, Yiqiao Xie, Yukai Huang, Zhaokai Wang, Weitong Cai, Guangwen Feng, Jifei Song, Zhensong Zhang, Hang Zhang
Computer Vision · Cross-listed from cs.RO
LEAP: Learning Emergent Active Perception for Quadruped Navigation
Ü. Bora Gökbakan, Stéphane Caron and Philippe Souères
Active perception allows autonomous
agents to select their viewpoints rather than passively process the viewpoints given to them, enabling them to target where to reduce uncertainty about their environment. Learned systems typically encourage this behavior with hand-designed proxy objectives, such as coverage or curiosity bonuses, that may conflict with the task. In this work, we propose a method to learn emergent active perception (LEAP) without augmentation of the task objective. We formulate the problem of goal-oriented
navigation over hazardous terrains with goals that must be discovered visually. We then propose an architecture for
navigation policies with active perception, and train them on a terrain curriculum where task pressure alone leads to the emergence of gaze control. Key to this emergence, LEAP works on a gaze-invariant representation that integrates depth images into
egocentric belief maps. We validate its performance in held-out evaluation scenarios, where it achieves a 92.7% success rate, compared to 74.2% for scripted or 34.5% for passive perception, and comes within 4.6 points of a privileged oracle. We validate that LEAP
navigation policies, unchanged, can be directly applied to steering quadrupedal locomotion policies in physics simulation.
Read the full abstract ▾
Computer Vision
PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control
Chuhao Chen, Peter Wonka, Chaoyang Wang et al. (+4 more)
Interactive control for
video generation is moving from coarse
prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Yet existing controllable methods either require the full control schedule before generation starts, or use pixel-space signals that dictate object positions rather than physical dynamics. To address these limitations, we propose PhysStream, an autoregressive model for physics-grounded image-to-
video synthesis that incorporates structured scene
memory—positional maps and object tracking maps derived online from previously generated frames—and supports fine-grained motion control via sparse velocity-increment signals that encode physical quantities, letting the model learn the underlying dynamics. We train our model in two stages: a bidirectional model is first finetuned with motion-control conditioning, then a causal autoregressive model is trained with additional structured scene
memory, further improving physical consistency. PhysStream enables interactive, mid-generation control over multi-object tabletop rigid-body scenes—a capability not supported by prior methods—reducing motion distribution distance (FVMD) by 33% and trajectory error by 12% over the strongest baselines on synthetic
benchmarks, and is preferred by human evaluators in over 85% of in-the-wild comparisons. Please check our website for more details: https://czzzzh.github.io/PhysStream
Read the full abstract ▾
All authors Chuhao Chen, Peter Wonka, Chaoyang Wang, Chen Wang, Qiao Feng, Sergey Tulyakov, Lingjie Liu
Computer Vision
Det-LIME: Detector-Aware, Multi-Instance Local Interpretable Model-Agnostic Explanations for Automated Marine Mammal Detection
Jiayi Zhou, David W. Johnston and Brinnae Bent
Despite the rapid uptake of black-box object detectors in marine mammal research and monitoring, explainability techniques are rarely integrated into conservation workflows. Furthermore, most classification-oriented explainability tools are ill-suited to detection tasks involving imagery of social organisms or those with colonial life histories, as they ignore multiple detections within a scene and produce single-instance outputs that blur evidence across individuals. These methods also generate low-resolution, often biologically irrelevant visuals, limiting their utility for debugging, targeted data augmentation, and refined data collection. We proposed Det-LIME, a detector-aware, multi-instance adaptation of Local Interpretable Model-Agnostic Explanations (LIME) that produced instance-specific, box-aligned explanations by combining per-detection weighting, a proximity kernel that emphasizes regions near each box, and Intersection-over-Union-based matching to track the same instance across perturbations. We evaluated Det-LIME on aerial drone imagery for harbor seal detection, with an additional seabird case study to assess generality, and compared it with vanilla LIME, Stabilized LIME, Deterministic LIME, and gradient-based attribution methods. Using the Attribution Ratio and Max Saliency Hit Rate metrics, we showed that Det-LIME consistently improved multi-instance attribution. In practice, these higher-resolution, instance-aware explanations provide insight into model outputs and support post-processing, debugging, and actionable improvements in modeling and data collection or augmentation.
Read the full abstract ▾
Computer Vision
Tables Decoded: DELTA for Structure, TARQA for Understanding
Jahanvi Rajput, Dhruv Kudale, Saikiran Kasturi et al. (+2 more)
Table understanding is a core task in document intelligence, encompassing two key subtasks: table reconstruction and table visual question answering (TabVQA). While recent approaches predominantly rely on
vision-
language models (VLMs) operating on table images, we propose a more scalable and effective alternative based on structured textual representations. These representations are easier to process, align more naturally with LLMs, and eliminate the need for
language-specific visual encoders, making them particularly suitable for
multilingual documents. We present DELTA, which separates physical structure recognition, logical structure recognition, and OCR to extract both layout and content accurately. DELTA outputs tables in Optimised Table Structure
Language (OTSL), a compact and unified format that encodes
cell arrangements and textual content. On table structure recognition (TSR), DELTA achieves TEDS- Structure scores comparable with state-of-the-art methods across FinTabNet, PubTabNet, and PubTables-1M. We further establish its robustness on non-English tables through our curated Hindi
benchmark, TORQUE. Building on this, we introduce TARQA, an LLM fine-tuned on OTSL sequences. Our approach yields gains of 9.3 p.p. on WTQ (TabQA) and 9.2 p.p. on FinTabNetQA (TabVQA), respectively. On TORQUE, our method ranks second among all VLMs and DELTA + LLM variants. We release our code, models, and
benchmark at: https://github.com/Tihiitborg/Tables-Decoded
Read the full abstract ▾
All authors Jahanvi Rajput, Dhruv Kudale, Saikiran Kasturi, Utkarsh Verma, Ganesh Ramakrishnan
Computer Vision
ORCA: Occlusion-Aware Refinement and Completion for Novel View Synthesis
Weronika Jakubowska, Maciej Zięba and Przemysław Spurek
Novel-view synthesis from a single image is a fundamentally ambiguous problem. As the camera moves away from the input viewpoint, previously hidden regions become visible, exposing missing geometry and holes in the reconstructed scene. Existing methods often rely on generative models to complete such regions. However, many of these artifacts are small gaps near depth boundaries and do not require generating new scene content. In order to eliminate expensive process of generating image we introduce ORCA, an occlusion-aware method for reconstructing and completing explorable 3D scenes from a single image. ORCA first introduces 3D structure into a Gaussian-anchor representation using
monocular depth while preserving the original camera-ray correspondence. During scene exploration, missing regions are handled based on their size and structure. Small disocclusions are repaired using RGB-D information already available in the reconstruction, while generative inpainting is reserved for larger regions that cannot be reliably recovered from the scene. New Gaussian anchors are added and optimized locally without modifying the existing representation. By reducing unnecessary reliance on generative inpainting, ORCA limits generation-induced
hallucinations and better preserves the content and structure of the original scene. On DIV2K, ORCA improves novel-view quality over VistaDream across all reported metrics, increasing MUSIQ from 61.60 to 68.71 and CLIP-IQA from 0.474 to 0.574. These results show that many novel-view artifacts can be repaired effectively by reusing information already present in the reconstructed scene.
Read the full abstract ▾
Computer Vision
BrainFocus: EEG-Guided ROI Selection for Efficient Vision-Language Models
Yihui Peng, Guorui Lu and Qinyu Chen
Vision-
language models (VLMs) achieve strong visual question answering (VQA) performance, but processing large cluttered images is computationally expensive when only a small region is relevant. Electroencephalography (
EEG) signals, which capture human neural responses to visual stimuli, can provide a human-derived semantic cue about the region of interest (ROI). However,
EEG-guided visual category decoding remains imperfect, making direct ROI routing unreliable. In this work, we propose BrainFocus, a reliable
EEG-guided efficient VLM framework for VQA. An
EEG classifier predicts a target category, and a YOLO detector localizes the matching ROI. The VLM receives the cropped ROI only when both predictions pass confidence thresholds; otherwise, it processes the full image. For evaluation, we build on
EEG-ImageNet to construct a 40-class
benchmark comprising generated cluttered images and real object-centric images, with target-ROI annotations and 600 English visual question-answer pairs. Across Qwen3.5-VL 2B, 4B, and 9B models, BrainFocus improves VQA accuracy by 4.14-9.87 percentage points (pp) on cluttered scenes while reducing input
tokens and total
tokens by 23.2%-39.4% and 23.2%-39.3%, and end-to-end floating-point operations (FLOPs) by 23.2%-39.5%. These results demonstrate that
EEG can guide efficient VLM inference even when its semantic decoding is imperfect.
Read the full abstract ▾
Computer Vision
Tracking the Unseen: An Occlusion-Robust Framework for Target Tracking Under Full and Long-Term Occlusion
Mais Mohammed, Sharifa Mohammed, Hanan Awadh et al. (+3 more)
Real-time multi-object tracking systems remain highly vulnerable to full and long-term occlusion, where targets temporarily or completely disappear from the camera's field of view. Conventional trackers may terminate trajectories prematurely, resulting in identity loss and reduced situational awareness in applications such as defense and surveillance. This work proposes an occlusion-robust target tracking framework that maintains target identity and trajectory continuity through the integration of YOLOv11n object detection, Kalman Filter motion prediction, and occlusion-aware appearance-based re-identification. The framework consists of three stages: object detection, position estimation during occlusion, and identity recovery after target reappearance. Six Re-Identification (Re-ID) architectures were evaluated within the same tracking framework under identical conditions, with the Occlusion-Aware Mask Network (OAMN) achieving the best overall performance and therefore selected for the final pipeline. The framework was benchmarked against OccluTrack on the public OVIS
dataset, achieving relative improvements of 18.1 percent in Multiple Object Tracking Accuracy (MOTA) and 25.1 percent in Identity F1 Score (IDF1), while reducing identity switches by 12.8 percent. On a custom military
dataset simulating surveillance and battlefield-like environments with long-term occlusion, the framework achieved a MOTA of 0.734 and an IDF1 of 0.729, corresponding to relative improvements of 14.2 percent and 5.8 percent over OccluTrack. The system demonstrated strong tracking continuity, robust identity preservation, and reliable trajectory estimation under challenging occlusion conditions, highlighting its effectiveness for defense-related surveillance applications requiring continuous target tracking during visibility loss.
Read the full abstract ▾
All authors Mais Mohammed, Sharifa Mohammed, Hanan Awadh, Haneen Bamaas, Raghad Bawazeer, Elham Alghamdi
Computer Vision
SlotDiT: Object-Centric Representations for Diffusion Transformers
Gjergj Plepi and Sven Behnke
Text-conditioned latent
diffusion models perform strongly in
video generation and are promising backbones for robotic applications. However, existing approaches rely on pixel-level or VAE-based latent representations that lack explicit semantic structure, leaving the impact of the representation space largely unexplored. Slot-based object-centric representations offer a structured alternative by decomposing scenes into object-level latents, or slots. While they have shown success in dynamics modeling and planning, they have not yet been explored for
diffusion-based generative modeling. We introduce SlotDiT, a text-guided
Diffusion Transformer (DiT) that operates in a slot-based latent space. Given a reference image and a
language instruction, SlotDiT decomposes the scene into object-centric slots representing individual entities. Conditioned on the instruction and observed scene context, the model autoregressively denoises future slot trajectories to predict scene dynamics. To systematically investigate latent-space design for
diffusion transformers, we compare slot-based representations against VAE-based and semantics-aligned alternatives within a unified DiT framework. Our experiments show that using slots as DiT latents yields competitive
video generation quality while consistently improving task-completion rates across four robotic
datasets. Furthermore, their compact representation provides a computationally efficient alternative to VAE-based and semantics-aligned latent spaces. Overall, our results demonstrate that object-centric structure is a powerful inductive bias for
diffusion-based generative modeling in robotic environments. The project page is available at https://slot-dit.github.io/.
Read the full abstract ▾
Computer Vision
SSC-Priors: Exploring Semantic and Visibility Priors to Boost Lidar Semantic Scene Completion
Tetiana Martyniuk, Jonathan Seele, Alexandre Boulch et al. (+3 more)
This paper investigates easy strategies to boost the performance of existing networks for
lidar semantic scene completion (SSC) without requiring complex architectural redesigns. The fact is that, over the last years, SSC methods have mostly pursued architectural innovations, making the models heavier and more complex, e.g., by jointly training a point cloud semantic
segmentation branch. In this work, we take a step back and explore two priors used as simple ingredients (possibly noisy) to improve existing approaches: semantic pseudo-labels and sensor visibility information. Concretely, we provide both kinds of information directly as additional inputs to a given SSC network, requiring only a minimal adaptation of the original architecture. We first demonstrate that endowing input point clouds with semantic pseudo-labels from off-the-shelf segmenters significantly improves the performance of existing SSC models. In fact, by evaluating these models against an oracle, we establish that high-quality semantic priors are a primary driver of semantic gains (mIoU), and that the SSC model can be trained just once with ground-truth semantics and then exploited without retraining using any segmenter. Furthermore, we equip the input
lidar point cloud with visibility information that distinguishes between empty spaces (between the
lidar and a scanned point) and unknown spaces (outside of lines of sight), providing a secondary performance boost across the tested architectures. We study the design space of data for representing visibility information and bound the remaining headroom with a ground-truth oracle on the free-space labels. On SemanticKITTI, these enhancements make older models competitive with state-of-the-art systems across four architectures, in one case even outperforming them. On the SSCBench-nuScenes
benchmark, both priors also transfer with the sparser 32-beam sensor.
Read the full abstract ▾
All authors Tetiana Martyniuk, Jonathan Seele, Alexandre Boulch, Gilles Puy, Renaud Marlet, Raoul de Charette
Computer Vision
PanoGS-SLAM: Panoramic 3D Gaussian Splatting SLAM
Yongqi Mao, Hao Shi, Yufan Zhang et al. (+3 more)
Real-time dense SLAM is a core capability for robotics applications that require robust localization and high- quality mapping in dynamic or fast-changing environments. Recent 3D Gaussian Splatting (3DGS)-based SLAM methods have shown promising performance, but most are designed for narrow-FoV pinhole cameras, where limited angular coverage weakens pose observability and often leads to unstable photo- metric optimization under rapid motion and large viewpoint changes. We present PanoGS-SLAM, the first panoramic dense SLAM system built on 3D Gaussian Splatting. Our method per- forms differentiable rendering and pose optimization directly in the spherical domain, enabling omnidirectional photometric constraints for more stable tracking. To improve
geometric consistency and robustness, we introduce (1) a sphere-consistent photometric loss that compensates for the area distortion of equirectangular projection, and (2) a depth-guided Gaussian initialization strategy that stabilizes incremental mapping in newly observed regions. Extensive experiments on both real and synthetic panoramic
benchmarks (PALVIO and SynPano) show that PanoGS-SLAM consistently outperforms
geometric and GS-based baselines in tracking accuracy and rendering quality, while achieving fast front-end convergence and real-time perfor- mance. In addition, controlled field-of-view experiments reveal a clear monotonic improvement in optimization conditioning and convergence stability as angular coverage increases, high- lighting the fundamental role of sensing geometry in shaping the optimization landscape of differentiable Gaussian-based SLAM. The source code will be made publicly available.
Read the full abstract ▾
All authors Yongqi Mao, Hao Shi, Yufan Zhang, Zhonghua Yi, Xiangfei Guo, Kaiwei Wang
Computer Vision
Optical-Flow Wingbeat Counting in MuJoCo: A Comparison of Convolutional, Spiking, and Attention-Based Temporal Models
Zhang Nengbo
Visual monitoring of flapping-wing vehicles requires distinguishing individual wingbeats from motion strength and average frequency. This paper presents a controlled MuJoCo evaluation of wingbeat counting from signed optical flow observed by virtual cameras mounted on Crazyflie vehicles. Three flapping-wing models were recorded at optical distances of 1.5 and 3.0 m, producing 1,440 clips from 240 paired scene configurations with a scene-level 3:1 training-test split. A common spatial convolutional encoder was combined with a causal temporal convolutional network, a recurrent leaky integrate-and-fire spiking network, or causal self-
attention. Each model predicted phase and activity, followed by the same directed-crossing event counter. The six existing convolutional models were retained, and all twelve new models were frozen before their test predictions were generated. Exact-count accuracies at 1.5 m were 96.67%, 95.00%, and 96.67%, respectively; at 3.0 m they were 94.44%, 92.22%, and 95.00%. All paired scene-bootstrap intervals for differences in exact-count accuracy included zero. Seven far-distance spiking-model clips had correct totals despite event-timing mismatches, demonstrating why total-count and event-level measurements must be reported together. The results support the feasibility of causal optical-flow counting in the tested setting and identify boundary-sensitive errors. They do not establish an architecture ranking across repeated training, real-flight robustness, or hardware efficiency.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Quantum-Inspired Trainable and Parameter-Efficient Tensor Networks for Image Inpainting
Shiwen An and Konstantinos Slavakis
This work introduces
quantum-inspired tensor-network
circuits as trainable transforms for image inpainting. Among the proposed architectures, the diagonal
quantum Fourier transform (QFT) relaxation is invertible with $O(N^2 \log N)$ computational cost for $N\times N$ images, inherently preserving minimum coherence throughout training via its
circuit structure and eliminating the need for explicit coherence penalties. Unconstrained gradient-based phase optimization (Riemannian-optimization free) enables efficient learning from randomly sampled training data, allowing the learned transform to generalize to test images observed through fixed sampling masks. Numerical tests show that the learned models outperform fixed transforms and per-image optimization while matching the performance of much larger
unitary architectures, yet with far fewer parameters.
Read the full abstract ▾
Computer Vision
Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Models
Ziheng Ren, Qian Gao, Jun Fan et al. (+3 more)
Multimodal large
language models generate natural-
language responses from visual inputs, yet may mention objects absent from an image. In medication assistance, accessible perception, and environmental decision-making, such
hallucinations can create real-world safety risks. We propose Semantic-Spatial Agreement Verification (SSAV), a training-free method for verifying object claims. A visually grounded claim should remain stable across semantically equivalent queries and repeatedly localize to the same image region. SSAV aggregates multiple
prompts to estimate semantic support and reduce sensitivity to query wording. Query-Induced Regional Verification (QIRV) combines cross-query region persistence, spatial overlap, and relative candidate dominance to identify isolated high responses and dispersed localizations. A
geometric mean fuses semantic and spatial evidence, lowering the verification score when either branch lacks support. Experiments on three base models and multiple evaluation protocols show that SSAV effectively mitigates object
hallucination. On LLaVA-1.5-7B, accuracy averaged across COCO, A-OKVQA, and GQA improves by 1.81 and 3.17 percentage points under POPE Popular and
Adversarial, respectively, while CHAIRs decreases from 49.40% to 32.80%. These results show that cross-query semantic stability and regional consistency provide interpretable external visual evidence for object claims.
Read the full abstract ▾
All authors Ziheng Ren, Qian Gao, Jun Fan, Guohui Ding, Zhenyu Yang, Yuteng Xiao
Computer Vision
Exploring 2D backbone effects for indoor semantic occupancy prediction
Shizhang Fanga, Wanling Yea and Qi Zheng
Semantic occupancy prediction gives an
embodied agent a voxel-level account of where space is free, occupied, and semantically meaningful. In RGB-D pipelines such as EmbodiedScan, the image encoder is often left as a default module, even though its features are the visual evidence later sampled into the 3D grid. We study this design choice directly. A central finding is that changing the 2D backbone improves occupancy accuracy more than several carefully designed occupancy architectures or modules. We keep the main RGB-D projection, depth branch, and occupancy head fixed, and replace only the image backbone. The compared encoders are CLIP-ResNet, CLIP-ViT, BLIP2, and DINOv2. Under the controlled setting, the measured mIoU changes substantially: DINOv2 obtains 30.55%, BLIP2 obtains 29.49%, CLIP-ViT obtains 24.33%, and CLIP-ResNet obtains 17.41%. The stronger encoders also exceed the original EmbodiedScan ResNet-50 baseline without modifying the downstream 3D fusion pipeline. Class-level results give a more detailed picture: DINOv2 is stronger on many layout and structural categories, whereas BLIP2 remains close on several object-centered classes. CLIP-ViT improves clearly over CLIP-ResNet, showing that the way CLIP features are exposed as dense
tokens matters for voxel lifting. These results indicate that the image backbone is not a secondary engineering detail in
embodied semantic occupancy, but a major source of variation in the final 3D prediction.
Read the full abstract ▾
Computer Vision
Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?
Zhaoyang Wei, Zipeng Wang, Yushe Cao et al. (+10 more)
Multimodal Large
Language Models have demonstrated impressive
video understanding, yet their ability to reason over long-form narratives is often masked by visual-centric evaluations and inefficient context processing. Existing
benchmarks over-rely on visual heuristics while marginalizing auditory cues, effectively reducing models to "silent observers" that bypass genuine cross-modal
reasoning. Moreover, standard dense sampling creates an evidence-context trade-off: increasing frames to capture evidence inevitably leads to
attention distraction and
token explosion. To bridge these gaps, we present
Video-HolmesV2, a novel
benchmark designed for Deep Audio-Visual Coupling. Unlike previous works, it enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence. To support this, we introduce: (1) a Multi-Model Cross-Verification pipeline to ensure task rigor; (2) a Spatio-temporal Evidence-Aware Metric for fine-grained calibration. Furthermore, we propose an Audio-Text Guided
Token Compression framework. By fusing task intent with auditory anchors, our method distills high-value
reasoning cues to mitigate long-context noise. In our evaluation, even strong proprietary models achieve below 60% accuracy, while our approach outperforms comparable open-source omni-models.
Read the full abstract ▾
All authors Zhaoyang Wei, Zipeng Wang, Yushe Cao, Chenhui Qiang, Shuaibing Cheng, Xuesong Yang, Sen Nie, Bowen Jiang, Wenchao Ding, Yanchao Hao, Zheng Wei, Xuehui Yu, Zhenjun Han
Computer Vision
DecoGS: Adaptive Static-Dynamic Decoupling of 3D Gaussians for Free-Viewpoint Video Streaming
Idil Sulo, Alexey Supikov, Ilke Demir et al. (+1 more)
Streaming 3D reconstruction demands both speed and temporal fidelity, goals that existing methods undermine by updating every Gaussian every frame, even in static regions. We present DecoGS, a method for efficient online training of 3D Gaussians from streaming
videos. Unlike prior methods that update the entire scene indiscriminately, DecoGS introduces an adaptive mechanism that selectively focuses optimization on spatiotemporal regions exhibiting motion or photometric changes. This targeted training strategy eliminates redundant updates that cause flickering and drift in nominally static regions, while enabling fast, high-fidelity scene updates. The pipeline further integrates region-aware Gaussian management through gradient gating and efficient visibility filtering to maintain temporal coherence and a compact
memory footprint. On N3DV and MeetRoom, DecoGS achieves 34.55 and 31.60 dB PSNR respectively, outperforming all streaming and offline baselines, while rendering at 261 FPS with $70\times$ lower temporal flicker than the best prior method, requiring no large-scale pretraining.
Read the full abstract ▾
All authors Idil Sulo, Alexey Supikov, Ilke Demir, Sainan Liu
Computer Vision
FROD: Feature Matching Residual Denoising Oracle Bone Decipher
Yanbin Hou, Biao Xiong, Guojun Xu et al. (+4 more)
Oracle bone script (OBS), one of the earliest Chinese writing systems, plays an important role in the study of Chinese etymology. Traditional decipherment relies heavily on domain experts who analyze characters through semantic context and structural
evolution. To assist this labor-intensive process, we formulate OBS decipherment assistance as a cross-era image translation task and propose FROD (Feature Matching Residual Denoising Oracle Bone Decipher). Although many OBS characters differ substantially from their modern counterparts, they often preserve local topological invariants at the radical level. During training, FROD leverages fast feature matching to provide gated
segmentation supervision: paired samples with sufficient matches are processed patch-wise to align fine-grained radicals, whereas low-similarity pairs are trained holistically to avoid mismatched artifacts. In addition, a Residual Denoising
Diffusion Model (RDDM) jointly estimates noise and residual signals, thereby reducing the positional drift and stroke disorder commonly observed in standard
diffusion models. Finally, a multi-stage font stylization refinement network refines the generated images by eliminating edge noise and stabilizing stroke structures. On our augmented character-disjoint
dataset, FROD achieves higher Top-1 recognition accuracy than the evaluated baselines, with a 3.8% absolute gain over OBSD.
Read the full abstract ▾
All authors Yanbin Hou, Biao Xiong, Guojun Xu, Jianwen Xiang, Cheng Tan, Yanchao Yang, Junwei Zhou
Computer Vision
InfoTaxa: Information-Calibrated Label-Free Clustering for Fine-Grained Visual Taxonomy
David Ahmedt-Aristizabal, Mohammad Ali Armin and Lars Petersson
Label-free clustering of frozen pretrained visual
embeddings offers a scalable route to biodiversity monitoring, but image-only fine-grained taxonomy exhibits a consistent coarse-to-fine failure mode: clusters recover broad taxonomic structure yet plateau at species level. We study this behaviour on BIOSCAN-5M through an information-calibrated clustering analysis. BioCLIP~2 features with UMAP and HDBSCAN reach $0.79$ AMI at family and $0.67$ at genus, substantially improving over the prior image baseline and remaining competitive with oracle-$K$, graph-based, and learned clustering heads on the same frozen features. To diagnose whether the remaining plateau is method-limited or information-limited, we introduce InfoTaxa, which combines clustering efficiency—the fraction of probe-estimated image information recovered by an unsupervised partition—with paired DNA as an audit signal only, not an inference input. The density pipeline recovers approximately $0.90$ and $0.81$ of the image-available information at order and family, respectively. Held-out late-fusion probes show that adding DNA to the image
embedding reduces species-level prediction error by approximately two bits. Robustness analyses cover multiple image encoders, described-species and rare-class subsets, probe diagnostics, and held-out-species coarse-rank generalisation and same-species
retrieval. Thus, in the tested setting, species-level label-free clustering is both clustering-limited and representation-limited: improved clustering may recover additional image-exposed structure, but cannot close the DNA-audited information gap alone.
Read the full abstract ▾
Computer Vision
Probe-VAD: Ordinal Likelihood Probing for Training-Free Video Anomaly Detection
Jiawei Gu, Qilin Zhao, Tengkuo Guo et al. (+6 more)
Video anomaly detection (VAD) aims to localize anomalous events in untrimmed
videos.
Vision-
language models (VLMs) provide rich visual understanding for training-free VAD, but existing approaches impose restrictive interfaces between visual understanding and anomaly scoring. Caption-based pipelines compress visual evidence into text, potentially discarding subtle cues, while direct numerical generation forces the model to express its judgment through a small set of predefined scores. Such interfaces can obscure subtle differences in anomaly severity, causing visually distinct clips to receive similar representations or scores and thereby limiting the resolution of anomaly ranking. We propose Probe-VAD, an ordinal binary-probing framework that directly probes severity preferences from a frozen VLM. Given raw
video clips, Probe-VAD queries ten ordered severity thresholds and extracts constrained YES/NO continuation likelihoods. Their normalized preferences form a cumulative severity profile, from which tail evidence is aggregated into a continuous anomaly score, with isotonic projection enforcing ordinal consistency. Experiments on public VAD
benchmarks demonstrate superior performance with low computational cost. Probe-VAD provides a simple interface for translating frozen VLM visual understanding into continuous, rank-sensitive anomaly scores without task-specific training or caption-based compression. Code is available at: https://github.com/yvestine/COVAS-VAD.
Read the full abstract ▾
All authors Jiawei Gu, Qilin Zhao, Tengkuo Guo, Zhiming Zhong, Shuangqing Zhang, Fan Lyu, Fang Zhao, Guo-Sen Xie, Caifeng Shan
Computer Vision
EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset
Ryosei Hara, Wataru Ikeda, Masashi Hatano et al. (+1 more)
3D hand mesh reconstruction is a challenging yet essential task for downstream applications, including human-
robot interaction and AR/VR. Although conventional cameras have been widely adopted for this task, methods that rely on them struggle in low-light environments and under severe motion blur. To address these limitations, event-based cameras have recently attracted
attention for their high dynamic range and high temporal resolution. However, applying event cameras to
egocentric hand reconstruction remains challenging because camera wearer's motion produces dense background events that obscure hand-specific signals. Although the first
egocentric event-based approach mitigates this issue using hand
segmentation, its binary hand mask does not distinguish between left and right hands. As a result, the model lacks instance-level hand information and predicts both hands even when only one or neither hand is present. This limitation leads to incorrect inter-hand relationships and degraded reconstruction accuracy. In this paper, we propose EventEgoHands++, a framework for event-based 3D hand mesh reconstruction from an
egocentric viewpoint. The proposed method incorporates a Hand Detector that estimates instance-level bounding boxes and masks for both the left and right hands. Moreover, we introduce Adaptive
Attention, which dynamically gates the
attention based on these detection results to accurately learn the spatial relationship and mutual interactions between the hands. To train and evaluate our framework, we extend the synthetic N-HOT3D
dataset and newly construct EEH-R, the largest real-world event-based
egocentric hand
dataset to date, comprising approximately 1M annotated frames captured in environments including low-light conditions. Extensive experiments on both synthetic and real
datasets demonstrate that our method consistently outperforms the baselines.
Read the full abstract ▾
All authors Ryosei Hara, Wataru Ikeda, Masashi Hatano, Mariko Isogawa
Computer Vision
Multimodal Cultural Heritage Architectural Style Classification for Residential Buildings in the UAE Based on CLIP Embeddings and SVM
Ahmed Ammar Kubba, Manar Abu Talib, Iman Ibrahim et al. (+1 more)
The analysis and classification of cultural heritage architectural styles remain challenging due to the complexity of visual images of buildings, which are highly relied on in traditional CNN-based classification approaches in comparison to textual descriptions, and the relative lack of non-western region-specific
datasets. This paper addresses this gap by proposing a
multimodal machine learning framework to analyze and classify Emirati residential architecture using OpenAI's CLIP model. We integrate visual features from images and textual features from expert descriptions into a unified 512-dimensional
embedding, followed by dimensionality reduction with UMAP for visualization and unsupervised clustering using K-Means. Cluster labels, which are derived from manual analysis of the K-Means clusters, are used to train an SVM classifier for automated architectural style classification. Our approach achieves a classification accuracy of 98% across eight identified style clusters, higher than every other study in the literature, demonstrating the effectiveness of combining visual and textual modalities. Overall, this paper highlights the potential of using
multimodal AI to support architectural heritage analysis, offering scalable and interpretable tools for exploring regional architectural identities.
Read the full abstract ▾
All authors Ahmed Ammar Kubba, Manar Abu Talib, Iman Ibrahim, Qassim Nasir
Computer Vision
MUMINS: Metadata-conditioned Uncertainty-aware Medical Image Next-state Synthesis
Anna Oliveras, Roger Marí, Rafael Redondo et al. (+7 more)
Forecasting anatomical changes such as tumor growth and neurodegeneration is a challenging generative
vision task. Morphological
evolution is subtle relative to static anatomy, highly patient-specific, and inherently
stochastic. Existing methods struggle with several issues: deterministic networks ignore biological stochasticity, while standard
diffusion models require computationally prohibitive multi-pass sampling to quantify uncertainty. We propose MUMINS (Metadata-conditioned Uncertainty-aware Medical Image Next-state Synthesis), an efficient
diffusion framework that jointly diffuses a baseline scan and its follow-up residual, summed to synthesize the follow-up scan, while concurrently predicting a spatial uncertainty map, in a single reverse
diffusion process. Conditioned on the time interval and relevant metadata, it preserves fine-grained anatomy by dynamically re-injecting the baseline as a soft anchor at every denoising step, and a negative-log-likelihood head learns the uncertainty map to explicitly flag error-prone regions. Designed without organ-specific heuristics, the same architecture is reused across anatomies via separate,
dataset-specific retraining. Extensive evaluations demonstrate that
dataset-specific retraining of MUMINS matches or outperforms dedicated, domain-specific state-of-the-art methods on lung CT (PNG) and
brain MRI (OASIS-3). Project page: https://github.com/aolivtous/MUMINS.
Read the full abstract ▾
All authors Anna Oliveras, Roger Marí, Rafael Redondo, Oriol Guardià, Cynthia Ifeyinwa Ugwu, Ana Tost, Bhalaji Nagarajan, Carolina Migliorelli, Vicent Ribas, Petia Radeva
Computer Vision · Cross-listed from cs.RO
HuMemSLAM: Efficient Human-Inspired Semantic Place Recognition for Robust Visual SLAM
Mayowa Adebambo, Sebastian Donnelly, Armand Amaritei et al. (+2 more)
Autonomous systems require reliable place recognition for efficient and effective simultaneous localisation and mapping (SLAM). Traditional
geometric visual SLAM approaches rely on low-level features and
geometric consistency, but remain vulnerable to perceptual aliasing, where different places appear similar, and perceptual variation, where the same place appears different. Although semantic SLAM and modern learned visual place recognition (VPR) methods improve robustness under challenging perceptual conditions, real-time deployment requires both high
retrieval accuracy and low latency. Inspired by human
memory and perception, we propose HuMem-VPR, which exploits the bidirectional relationship between bottom-up perceptual evidence and top-down contextual
reasoning to achieve high-level place understanding. We further introduce HuMemSLAM, the integration of HuMem-VPR with ORB-SLAM3. HuMem VPR achieved the highest aggregate
retrieval accuracy on the real-image
benchmark, competitive accuracy on the CARLA
benchmark, and approximately two to three times lower latency than the evaluated state-of-the-art VPR methods. Across the evaluated
dataset families and online experiments, HuMemSLAM substantially improved integrated Recall @1 over ORB-SLAM3's native
retrieval while reducing the proposals submitted to its
geometric backend.
Read the full abstract ▾
All authors Mayowa Adebambo, Sebastian Donnelly, Armand Amaritei, Andrew Bradley, Alexander Rast
Computer Vision
ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers
Jim Berend, Reduan Achtibat, Daniel Schäffer et al. (+4 more)
Vision Transformers (ViTs) are central to most modern
vision models, yet obtaining input attributions that are fine-grained, faithful, and stable remains challenging. Layer-wise Relevance Propagation (LRP) has been adapted to
transformer attention, but in ViTs it often produces noisy, unfaithful explanations. We show that the missing ingredient is the treatment of residual connections: cancellation effects in residual pathways lead to attribution explosion. Moreover, we find that these cancellations are substantially stronger in ViTs than in
language transformers. To address this issue, we introduce Residual-aware Layer-wise Relevance Propagation (ResLRP), a simple extension of LRP whose propagation rules explicitly account for cancellations in residual branches, are exactly conservative, and provably bound relevance explosion. Causal channel-wise interventions confirm that residual cancellation, not a generic regularization effect, drives the instability. ResLRP substantially improves attribution quality across faithfulness and localization, evaluated on ViT architectures spanning supervised, self-supervised, contrastive, hierarchical, and
multimodal families, as well as on the ground-truth-controlled FunnyBirds
benchmark. The largest gains arise in modern
Vision Language Models (VLMs), with +27-29% localization and up to 3.4x faithfulness scores. Beyond
benchmarks, ResLRP localizes Sparse Autoencoder (SAE) features in input space, and our residual amplification measure serves as an architecture-level diagnostic predicting where attribution degrades.
Read the full abstract ▾
All authors Jim Berend, Reduan Achtibat, Daniel Schäffer, Alexander Binder, Wojciech Samek, Sebastian Lapuschkin, Maximilian Dreyer
Computer Vision
From Foundation Embeddings to Cropland Maps: Label Efficiency, Temporal Transferability and Independent Human Validation
Mohammad Ammar Mughees, Giovanni Montefoschi, Zhongxin Chen et al. (+1 more)
Geospatial foundation models provide reusable representations of satellite imagery that support downstream mapping with limited task-specific modelling. We evaluate whether annual AlphaEarth
embeddings support binary cultivated-versus-non-cultivated mapping in Maine, USA, using 192 spatially separated patches and labels derived from the USDA Cropland Data Layer (CDL). Without fine-tuning the foundation model, a lightweight classifier reaches 93.7% overall accuracy and 90.8% balanced accuracy on held-out patches. Logistic regression is within 0.3 percentage points of a gradient-boosted ensemble, while a nearest-class-centroid rule, which uses class centroids but fits no parameters, reaches 90.2%. A balanced sample of 60,000 labelled pixels is within 1.3 percentage points of the full pool of 8.6 million pixels; because pixels are spatially autocorrelated, this result concerns pixel-sample efficiency rather than 60,000 independent annotation sites. In a same-region transfer experiment, classifiers trained in one year remain accurate across 2018 to 2023. Against a blind, two-interpreter consensus at 385 randomly sampled points in one contiguous 2023 block, the AlphaEarth-plus-random-forest map agrees at 95.3% ($\kappa=0.82$), compared with 91.7% for the CDL ($\kappa=0.72$; exact two-sided McNemar $p=0.0161$). This local result is consistent with partial smoothing of CDL label noise, but it does not establish statewide correction of the reference product. On the same points, the difference from a fine-tuned TerraMind
segmentation model is not statistically significant (95.3% versus 93.5%; $p=0.14$), and the experiment is not a controlled comparison of computational cost. These results support frozen geospatial
embeddings as a low-compute candidate for regional cropland mapping, subject to the limits of a single-state study, a 30 m-derived training reference, and a one-block human validation.
Read the full abstract ▾
All authors Mohammad Ammar Mughees, Giovanni Montefoschi, Zhongxin Chen, Maria Antonia Brovelli