Computer Vision
Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering
Shuliang Liu, Songbo Yang, Dong Fang et al. (+7 more)
Object
hallucination critically undermines the reliability of
Multimodal Large
Language Models, often stemming from a fundamental failure in cognitive introspection, where models blindly trust linguistic priors over specific visual evidence. Existing mitigations remain limited: contrastive decoding approaches operate superficially without rectifying internal semantic misalignments, while current latent steering methods rely on static vectors that lack instance-specific precision. We introduce
Vision-
Language Introspection (VLI), a training-free inference framework that simulates a metacognitive self-correction process. VLI first performs Attributive Introspection to diagnose
hallucination risks via probabilistic conflict detection and localize the causal visual anchors. It then employs Interpretable Bi-Causal Steering to actively modulate the inference process, dynamically isolating visual evidence from background noise while neutralizing blind confidence through adaptive calibration. VLI achieves state-of-the-art performance on advanced models, reducing object
hallucination rates by 12.67% on MMHal-Bench and improving accuracy by 5.8% on POPE.
Read the full abstract ▾
All authors Shuliang Liu, Songbo Yang, Dong Fang, Sihang Jia, Yuqi Tang, Lingfeng Su, Ruoshui Peng, Yibo Yan, Xin Zou, Xuming Hu
Computer Vision
Multi-Scale Local Speculative Decoding for Image Generation
Elia Peruzzo, Guillaume Sautière and Amirhossein Habibian
Autoregressive (AR) models have achieved remarkable success in image synthesis, yet their sequential nature imposes significant latency constraints. Speculative Decoding offers a promising avenue for acceleration, but existing approaches are limited by
token-level ambiguity and lack of spatial awareness. In this work, we introduce Multi-Scale Local Speculative Decoding (MuLo-SD), a novel framework that combines multi-resolution drafting with spatially informed verification to accelerate AR image generation. Our method leverages a low-resolution drafter paired with an up-sampling step to propose candidate image
tokens, which are then verified in parallel by a high-resolution target model. Crucially, we incorporate a local rejection and resampling mechanism, enabling efficient correction of draft errors by focusing on spatial neighborhoods rather than raster-scan resampling after the first rejection. When integrated with parallel decoding resampling, MuLo-SD achieves substantial speedups – up to $\mathbf{5\times}$ – outperforming both speculative decoding and parallel decoding baselines in terms of acceleration, while maintaining comparable semantic
alignment and perceptual quality. These results are validated using GenEval, DPG-Bench, and FID/HPSv2 on the MS-COCO 5k validation split. Extensive ablations highlight the impact of up-sampling design, probability pooling, and local rejection and resampling with neighborhood expansion. Our approach sets a new state-of-the-art in speculative decoding for image synthesis, bridging the gap between efficiency and fidelity. Project page is available at https://qualcomm-ai-research.github.io/mulo-sd-webpage/ .
Read the full abstract ▾
Computer Vision
Atlas 2 – Foundation models for clinical deployment
Maximilian Alber, Timo Milbich, Alexandra Carpen-Amarie et al. (+24 more)
Pathology foundation models substantially advanced the possibilities in computational pathology — yet tradeoffs in terms of performance, robustness, and computational requirements remained, which limited their clinical deployment. In this report, we present Atlas 2, Atlas 2-B, and Atlas 2-S, three pathology
vision foundation models which bridge these shortcomings by showing state-of-the-art prediction performance, robustness, and resource efficiency in a comprehensive evaluation across eighty public
benchmarks. Our models were trained on the largest pathology foundation model
dataset to date comprising 5.5 million histopathology whole slide images, collected from three medical institutions Charité - Universitätsmedizin Berlin, LMU Munich, and Mayo Clinic.
Read the full abstract ▾
All authors Maximilian Alber, Timo Milbich, Alexandra Carpen-Amarie, Stephan Tietz, Jonas Dippel, Lukas Muttenthaler, Beatriz Perez Cancer, Alessandro Benetti, Panos Korfiatis, Elias Eulig, Jérôme Lüscher, Jiasen Wu, Sayed Abid Hashimi, Gabriel Dernbach, Simon Schallenberg, Neelay Shah, Moritz Krügener, Aniruddh Jammoria, Jake Matras, Patrick Duffy, Matt Redlon, Philipp Jurmeister, David Horst, Lukas Ruff, Klaus-Robert Müller, Frederick Klauschen, Andrew Norgan
Computer Vision
A Two-Stage Multitask Vision-Language Framework for Explainable Crop Disease Visual Question Answering
Md. Zahid Hossain, Most. Sharmin Sultana Samu, Md. Rakibul Islam et al. (+1 more)
Visual question answering (VQA) for crop disease analysis requires accurate visual understanding and reliable
language generation. In this work, we present a lightweight and explainable
vision-
language framework for crop and disease identification from leaf images. The proposed approach integrates a Swin
Transformer vision encoder with sequence-to-sequence
language decoders. The
vision encoder is first trained in a multitask setup for both plant and disease classification, and then frozen while the text decoders are trained, forming a two-stage training strategy that enhances visual representation learning and cross-modal
alignment. We evaluate the model on the large-scale Crop Disease Domain
Multimodal (CDDM)
dataset using both classification and natural
language generation metrics. Experimental results demonstrate near-perfect recognition performance, achieving 99.94% plant classification accuracy and 99.06% disease classification accuracy, along with strong BLEU, ROUGE and BERTScore results. Without fine-tuning, the model further generalizes well to the external PlantVillageVQA
benchmark, achieving 83.18% micro accuracy in the VQA task. Our lightweight design outperforms larger
vision-
language baselines while using significantly fewer parameters. Explainability is assessed through Grad-CAM and
token-level attribution, providing interpretable visual and textual evidence for predictions. Qualitative results demonstrate robust performance under diverse user-driven queries, highlighting the effectiveness of task-specific visual pretraining and the two-stage training methodology for crop disease visual question answering. An interactive demo of the proposed Swin-T5 model is publicly available as a Gradio-based application at https://huggingface.co/spaces/Zahid16/PlantDiseaseVQAwithSwinT5 for community use.
Read the full abstract ▾
All authors Md. Zahid Hossain, Most. Sharmin Sultana Samu, Md. Rakibul Islam, Md. Siam Ansary
Computer Vision
VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control
Sixiao Zheng, Minghao Yin, Wenbo Hu et al. (+3 more)
Video world models aim to simulate dynamic, real-world environments, yet existing methods struggle to provide unified and precise control over camera and multi-object motion, as
videos inherently capture dynamics in the projected 2D image plane. To bridge this gap, we introduce VerseCrafter, a geometry-driven
video world model that generates dynamic, realistic
videos from a unified 4D
geometric world state. Our approach is centered on a novel 4D
Geometric Control representation, which encodes the world state as a static background point cloud and per-object 3D Gaussian trajectories. This representation captures each object's motion path and probabilistic 3D occupancy over time, providing a flexible, category-agnostic alternative to rigid bounding boxes and parametric models. We render 4D
Geometric Control into 4D control maps for a pretrained
video diffusion model, enabling high-fidelity, view-consistent
video generation that faithfully follows the specified dynamics. To enable training at scale, we develop an automatic data engine and construct VerseControl4D, a real-world
dataset of 35K training samples with automatically derived
prompts and rendered 4D control maps. Extensive experiments show that VerseCrafter achieves superior visual quality and more accurate control over camera and multi-object motion than prior methods.
Read the full abstract ▾
All authors Sixiao Zheng, Minghao Yin, Wenbo Hu, Xiaoyu Li, Ying Shan, Yanwei Fu
Computer Vision
VERSE: Visual Embedding Reduction and Space Exploration. Clustering-Guided Insights for Training Data Enhancement in Visually-Rich Document Understanding
Ignacio de Rodrigo, Alvaro J. Lopez-Lopez and Jaime Boal
This work introduces VERSE, a methodology for analyzing and improving
Vision-
Language Models applied to Visually-rich Document Understanding by exploring their visual
embedding space. VERSE enables the visualization of latent representations, supporting the assessment of model feasibility. It also facilitates the identification of problematic regions and guides the generation of synthetic data to enhance performance in those clusters. We validate the methodology by training on the synthetic MERIT
Dataset and evaluating on its real-world counterpart, MERIT Secret. Results show that VERSE helps uncover the visual features associated with error-prone clusters, and that retraining with samples containing these features substantially boosts F1 performance without degrading generalization. Furthermore, we demonstrate that on-premise models such as Donut and Idefics2, when optimized with VERSE, match or even surpass the performance of SaaS solutions like GPT-4 and Pixtral.
Read the full abstract ▾
Computer Vision
Re-Align: Structured Reasoning-guided Alignment for In-Context Image Generation and Editing
Runze He, Yiji Cheng, Tiankai Hang et al. (+11 more)
In-context image generation and editing (ICGE) enables users to specify visual concepts through interleaved image-text
prompts, demanding precise understanding and faithful execution of user intent. Although recent unified
multimodal models exhibit promising understanding capabilities, these strengths often fail to transfer effectively to image generation. We introduce Re-Align, a unified framework that bridges the gap between understanding and generation through structured
reasoning-guided
alignment. At its core lies the In-Context Chain-of-Thought (IC-CoT), a structured
reasoning paradigm that decouples semantic guidance and reference association, providing clear textual target and mitigating confusion among reference images. Furthermore, Re-Align introduces an effective RL training scheme that leverages a surrogate reward to measure the
alignment between structured
reasoning text and the generated image, thereby improving the model's overall performance on ICGE tasks. Extensive experiments verify that Re-Align outperforms competitive methods of comparable model scale and resources on both in-context image generation and editing tasks.
Read the full abstract ▾
All authors Runze He, Yiji Cheng, Tiankai Hang, Zhimin Li, Yu Xu, Zijin Yin, Shiyi Zhang, Wenxun Dai, Penghui Du, Ao Ma, Chunyu Wang, Qinglin Lu, Jizhong Han, Jiao Dai
Computer Vision
From Rays to Projections: Better Inputs for Feed-Forward View Synthesis
Zirui Wu, Zeren Jiang, Martin R. Oswald et al. (+1 more)
Feed-forward view synthesis models predict a novel view in a single pass with minimal 3D inductive bias. Existing works encode cameras as Plücker ray maps, which tie predictions to the arbitrary world coordinate gauge and make them sensitive to small camera transformations, thereby undermining
geometric consistency. In this paper, we ask what inputs best condition a model for robust and consistent view synthesis. We propose projective conditioning, which replaces raw camera parameters with a target-view projective cue that provides a stable 2D input. This reframes the task from a brittle
geometric regression problem in ray space to a well-conditioned target-view image-to-image translation problem. Additionally, we introduce a masked autoencoding pretraining strategy tailored to this cue, enabling the use of large-scale uncalibrated data for pretraining. Our method shows improved fidelity and stronger cross-view consistency compared to ray-conditioned baselines on our view-consistency
benchmark. It also achieves state-of-the-art quality on standard novel view synthesis
benchmarks.
Read the full abstract ▾
All authors Zirui Wu, Zeren Jiang, Martin R. Oswald, Jie Song
Computer Vision
UniLiPs: Unified LiDAR Pseudo-Labeling with Geometry-Grounded Dynamic Scene Decomposition
Filippo Ghilotti, Samuel Brucker, Nahku Saidy et al. (+3 more)
Unlabeled
LiDAR logs, in autonomous driving applications, are inherently a gold mine of dense 3D geometry hiding in plain sight - yet they are almost useless without human labels, highlighting a dominant cost barrier for autonomous-perception research. In this work we tackle this bottleneck by leveraging temporal-
geometric consistency across
LiDAR sweeps to lift and fuse cues from text and 2D
vision foundation models directly into 3D, without any manual input. We introduce an unsupervised multi-modal pseudo-labeling method relying on strong
geometric priors learned from temporally accumulated
LiDAR maps, alongside with a novel iterative update rule that enforces joint
geometric-semantic consistency, and vice-versa detecting moving objects from inconsistencies. Our method simultaneously produces 3D semantic labels, 3D bounding boxes, and dense
LiDAR scans, demonstrating robust generalization across three
datasets. We experimentally validate that our method compares favorably to existing semantic
segmentation and object detection pseudo-labeling methods, which often require additional manual supervision. We confirm that even a small fraction of our geometrically consistent, densified
LiDAR improves depth prediction by 51.5% and 22.0% MAE in the 80-150 and 150-250 meters range, respectively.
Read the full abstract ▾
All authors Filippo Ghilotti, Samuel Brucker, Nahku Saidy, Matteo Matteucci, Mario Bijelic, Felix Heide
Computer Vision
Driving on Registers
Ellington Kirby, Alexandre Boulch, Yihong Xu et al. (+11 more)
We present DrivoR, a simple and efficient
transformer-based architecture for end-to-end autonomous driving. Our approach builds on pretrained
Vision Transformers (ViTs) and introduces camera-aware register
tokens that compress multi-camera features into a compact scene representation, significantly reducing downstream computation without sacrificing accuracy. These
tokens drive two lightweight
transformer decoders that generate and then score candidate trajectories. The scoring decoder learns to mimic an oracle and predicts interpretable sub-scores representing aspects such as safety, comfort, and efficiency, enabling behavior-conditioned driving at inference. Despite its minimal design, DrivoR outperforms or matches strong contemporary baselines across NAVSIM-v1, NAVSIM-v2, and the photorealistic closed-loop HUGSIM
benchmark. Our results show that a pure-
transformer architecture, combined with targeted
token compression, is sufficient for accurate, efficient, and adaptive end-to-end driving. Code and checkpoints will be made available via the project page.
Read the full abstract ▾
All authors Ellington Kirby, Alexandre Boulch, Yihong Xu, Yuan Yin, Gilles Puy, Éloi Zablocki, Andrei Bursuc, Spyros Gidaris, Renaud Marlet, Florent Bartoccioni, Anh-Quan Cao, Nermin Samet, Tuan-Hung VU, Matthieu Cord
Computer Vision · Cross-listed from physics.med-ph
Quantitative mapping from conventional MRI using self-supervised physics-guided deep learning: applications to a large-scale, clinically heterogeneous dataset
Jelmer van Lune, Stefano Mandija, Oscar van der Heide et al. (+5 more)
Magnetic resonance imaging (MRI) is a cornerstone of clinical neuroimaging, yet conventional MRIs provide qualitative information heavily dependent on scanner hardware and acquisition settings. While quantitative MRI (qMRI) offers intrinsic tissue parameters, the requirement for specialized acquisition protocols and reconstruction
algorithms restricts its availability and impedes large-scale biomarker research. This study presents a self-supervised physics-guided deep learning framework to infer quantitative T1, T2, and
proton-density (PD) maps directly from widely available clinical conventional T1-weighted, T2-weighted, and FLAIR MRIs. The framework was trained and evaluated on a large-scale, clinically heterogeneous
dataset comprising 4,121 scan sessions acquired at our institution over six years on four different 3 T MRI scanner systems, capturing real-world clinical variability. The framework integrates Bloch-based signal models directly into the training objective. Across more than 600 test sessions, the generated maps exhibited white matter and gray matter values consistent with literature ranges. Additionally, the generated maps showed invariance to scanner hardware and acquisition protocol groups, with inter-group coefficients of variation $\leq$ 1.1%. Subject-specific analyses demonstrated excellent voxel-wise reproducibility across scanner systems and sequence parameters, with Pearson $r$ and concordance correlation coefficients exceeding 0.82 for T1 and T2. Mean relative voxel-wise differences were low across all quantitative parameters, especially for T2 ($<$ 6%). These results indicate that the proposed framework can robustly transform diverse clinical conventional MRI data into quantitative maps, potentially paving the way for large-scale quantitative biomarker research.
Read the full abstract ▾
All authors Jelmer van Lune, Stefano Mandija, Oscar van der Heide, Matteo Maspero, Martin B. Schilder, Jan Willem Dankbaar, Cornelis A.T. van den Berg, Alessandro Sbrizzi
Computer Vision
From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs)
Suyash Mishra, Qiang Li, Srikanth Patil et al. (+1 more)
Vision Language Models (VLMs) are poised to revolutionize the digital transformation of pharmacyceutical industry by enabling intelligent, scalable, and automated multi-modality content processing. Traditional manual annotation of heterogeneous data modalities (text, images,
video, audio, and web links), is prone to inconsistencies, quality degradation, and inefficiencies in content utilization. The sheer volume of long
video and audio data further exacerbates these challenges, (e.g. long clinical trial interviews and educational seminars). Here, we introduce a domain adapted
Video to
Video Clip Generation framework that integrates Audio
Language Models (ALMs) and
Vision Language Models (VLMs) to produce highlight clips. Our contributions are threefold: (i) a reproducible Cut & Merge
algorithm with fade in/out and timestamp normalization, ensuring smooth transitions and audio/visual
alignment; (ii) a personalization mechanism based on role definition and
prompt injection for tailored outputs (marketing, training, regulatory); (iii) a cost efficient e2e pipeline strategy balancing ALM/VLM enhanced processing. Evaluations on
Video MME
benchmark (900) and our proprietary
dataset of 16,159 pharmacy
videos across 14 disease areas demonstrate 3 to 4 times speedup, 4 times cost reduction, and competitive clip quality. Beyond efficiency gains, we also report our methods improved clip coherence scores (0.348) and informativeness scores (0.721) over state of the art VLM baselines (e.g., Gemini 2.5 Pro), highlighting the potential of transparent, custom extractive, and compliance supporting
video summarization for life sciences.
Read the full abstract ▾
All authors Suyash Mishra, Qiang Li, Srikanth Patil, Anubhav Girdhar
Computer Vision
Patch-based Representation and Learning for Efficient Deformation Modeling
Ruochen Chen, Thuy Tran and Shaifali Parashar
In this paper, we present a patch-based representation of surfaces, PolyFit, which is obtained by fitting jet functions locally on surface patches. Such a representation can be learned efficiently in a supervised fashion from both analytic functions and real data. Once learned, it can be generalized to various types of surfaces. Using PolyFit, the surfaces can be efficiently deformed by updating a compact set of jet coefficients rather than optimizing per-vertex degrees of freedom for many downstream tasks in computer
vision and graphics. We demonstrate the capabilities of our proposed methodologies with two applications: 1) Shape-from-template (SfT): where the goal is to deform the input 3D template of an object as seen in image/
video. Using PolyFit, we adopt test-time optimization that delivers competitive accuracy while being markedly faster than offline physics-based solvers, and outperforms recent physics-guided neural simulators in accuracy at modest additional runtime. 2) Garment draping. We train a self-supervised, mesh- and garment-agnostic model that generalizes across resolutions and garment types, delivering up to an order-of-magnitude faster inference than strong baselines.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Scalable neural pushbroom architectures for real-time denoising of hyperspectral images onboard satellites
Ziyao Yi, Davide Piccinini, Diego Valsesia et al. (+2 more)
The next generation of Earth observation satellites will seek to deploy intelligent models directly onboard the payload in order to minimize the latency incurred by the transmission and processing chain of the ground segment, for time-critical applications. Designing neural architectures for onboard execution, particularly for satellite-based hyperspectral imagers, poses novel challenges due to the unique constraints of this environment and imaging system that are largely unexplored by the traditional computer
vision literature. In this paper, we show that this setting requires addressing three competing objectives, namely high-quality inference with low complexity, dynamic power scalability and fault tolerance. We focus on the problem of hyperspectral image denoising, which is a critical task to enable effective downstream inference, and highlights the constraints of the onboard processing scenario. We propose a neural network design that addresses the three aforementioned objectives with several novel contributions. In particular, we propose a mixture of denoisers that can be resilient to radiation-induced faults as well as allowing for time-varying power scaling. Moreover, each denoiser employs an innovative architecture where an image is processed line-by-line in a causal way, with a
memory of past lines, in order to match the acquisition process of pushbroom hyperspectral sensors and greatly limit
memory requirements. We show that the proposed architecture can run in real-time, i.e., process one line in the time it takes to acquire the next one, on low-power hardware and provide competitive denoising quality with respect to significantly more complex state-of-the-art models. We also show that the power scalability and fault tolerance objectives provide a design space with multiple tradeoffs between those properties and denoising quality.
Read the full abstract ▾
All authors Ziyao Yi, Davide Piccinini, Diego Valsesia, Tiziano Bianchi, Enrico Magli
Computer Vision
Higher-Order Adversarial Patches for Real-Time Object Detectors
Jens Bayer, Stefan Becker, David Münch et al. (+2 more)
Higher-order
adversarial attacks can directly be considered the result of a cat-and-mouse game – an elaborate action involving constant pursuit, near captures, and repeated escapes. This idiom describes the enduring circular training of
adversarial attack patterns and
adversarial training the best. The following work investigates the impact of higher-order
adversarial attacks on object detectors by successively training attack patterns and hardening object detectors with
adversarial training. The YOLOv10 object detector is chosen as a representative, and
adversarial patches are used in an evasion attack manner. Our results indicate that higher-order
adversarial patches are not only affecting the object detector directly trained on but rather provide a stronger generalization capacity compared to lower-order
adversarial patches. Moreover, the results highlight that solely
adversarial training is not sufficient to harden an object detector efficiently against this kind of
adversarial attack. Code: https://github.com/JensBayer/HigherOrder
Read the full abstract ▾
All authors Jens Bayer, Stefan Becker, David Münch, Michael Arens, Jürgen Beyerer
Computer Vision
OceanSplat: Object-aware Gaussian Splatting with Trinocular View Consistency for Underwater Scene Reconstruction
Minseong Kweon and Jinsun Park
We introduce OceanSplat, a novel 3D Gaussian Splatting-based approach for high-fidelity
underwater scene reconstruction. To overcome multi-view inconsistencies caused by scattering media, we design a trinocular setup for each camera pose by rendering from horizontally and vertically translated virtual viewpoints, enforcing view consistency to facilitate spatial optimization of 3D Gaussians. Furthermore, we derive synthetic epipolar depth priors from the virtual viewpoints, which serve as self-supervised depth regularizers to compensate for the limited
geometric cues in degraded
underwater scenes. We also propose a depth-aware alpha adjustment that modulates the opacity of 3D Gaussians during early training based on their depth along the viewing direction, deterring the formation of medium-induced primitives. Our approach promotes the disentanglement of 3D Gaussians from the scattering medium through effective
geometric constraints, enabling accurate representation of scene structure and significantly reducing floating artifacts. Experiments on real-world
underwater and simulated scenes demonstrate that OceanSplat substantially outperforms existing methods for both scene reconstruction and restoration in scattering media.
Read the full abstract ▾
Computer Vision
SparseLaneSTP: Leveraging Spatio-Temporal Priors with Sparse Transformers for 3D Lane Detection
Maximilian Pittner, Joel Janai, Mario Faigle et al. (+1 more)
3D lane detection has emerged as a critical challenge in autonomous driving, encompassing identification and localization of lane markings and the 3D road surface. Conventional 3D methods detect lanes from dense birds-eye-viewed (BEV) features, though erroneous transformations often result in a poor feature representation misaligned with the true 3D road surface. While recent sparse lane detectors have surpassed dense BEV approaches, they completely disregard valuable lane-specific priors. Furthermore, existing methods fail to utilize historic lane observations, which yield the potential to resolve ambiguities in situations of poor visibility. To address these challenges, we present SparseLaneSTP, a novel method that integrates both
geometric properties of the lane structure and temporal information into a sparse lane
transformer. It introduces a new lane-specific spatio-temporal
attention mechanism, a continuous lane representation tailored for sparse architectures as well as temporal regularization. Identifying weaknesses of existing 3D lane
datasets, we also introduce a precise and consistent 3D lane
dataset using a simple yet effective auto-labeling strategy. Our experimental section proves the benefits of our contributions and demonstrates state-of-the-art performance across all detection and error metrics on existing 3D lane detection
benchmarks as well as on our novel
dataset.
Read the full abstract ▾
All authors Maximilian Pittner, Joel Janai, Mario Faigle, Alexandru Paul Condurache
Computer Vision
TEA: Temporal Adaptive Satellite Image Semantic Segmentation
Juyuan Kang, Hao Zhu, Yan Zhu et al. (+6 more)
Crop mapping based on satellite images time-series (SITS) holds substantial economic value in agricultural production settings, in which parcel
segmentation is an essential step. Existing approaches have achieved notable advancements in SITS
segmentation with predetermined sequence lengths. However, we found that these approaches overlooked the generalization capability of models across scenarios with varying temporal length, leading to markedly poor
segmentation results in such cases. To address this issue, we propose TEA, a TEmporal Adaptive SITS semantic
segmentation method to enhance the model's resilience under varying sequence lengths. We introduce a teacher model that encapsulates the global sequence knowledge to guide a student model with adaptive temporal input lengths. Specifically, teacher shapes the student's feature space via intermediate
embedding, prototypes and soft label perspectives to realize knowledge transfer, while dynamically aggregating student model to mitigate knowledge forgetting. Finally, we introduce full-sequence reconstruction as an auxiliary task to further enhance the quality of representations across inputs of varying temporal lengths. Through extensive experiments, we demonstrate that our method brings remarkable improvements across inputs of different temporal lengths on common
benchmarks. Our code will be publicly available.
Read the full abstract ▾
All authors Juyuan Kang, Hao Zhu, Yan Zhu, Wei Zhang, Jianing Chen, Tianxiang Xiao, Yike Ma, Hao Jiang, Feng Dai
Computer Vision
Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics
Subhadeep Roy, Gagan Bhatia and Steffen Eger
Automatic metrics are widely used to evaluate text-to-image models, often replacing human judgment in benchmarking, model selection, and large-scale data filtering. Yet they may reward images that look plausible or prototypical rather than images that faithfully satisfy the
prompt. We identify prototypicality bias as a systematic blindspot in
multimodal evaluation: metrics can prefer a semantically incorrect but visually or socially prototypical image over a correct but less prototypical one. We introduce PROTOBIAS, a controlled diagnostic
benchmark across Animals, Objects, and Demography, where semantically correct images are contrasted with plausible prototypical adversaries containing a single controlled semantic violation. Grounded in prototype theory and social-category prototypicality, PROTOBIAS is constructed with multiple
prompt generators, image generators, and independent VLM filters, and validated through
prompt-quality, human-annotation, and image-quality controls. Using PROTOBIAS, we show that widely used
embedding, reward, VQA-based, and VLM-as-judge metrics frequently fail these contrasts, while human judgments remain more faithful to semantic correctness. We further introduce PROTOSCORE, a lightweight contrastively trained evaluator, as an initial mitigation baseline. PROTOBIAS provides a focused
benchmark for measuring prototypicality-driven metric failures and developing more semantically faithful T2I evaluators.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CR
Decentralized Privacy-Preserving Federal Learning of Computer Vision Models on Edge Devices
Damian Harenčák, Lukáš Gajdošech and Martin Madaras
Collaborative training of a machine learning model comes with a risk of sharing sensitive or private data.
Federated learning offers a way of collectively training a single global model without the need to share client data, by sharing only the updated parameters from each client's local model. A central server is then used to aggregate parameters from all clients and redistribute the aggregated model back to the clients. Recent findings have shown that even in this scenario, private data can be reconstructed only using information about model parameters. Current efforts to mitigate this are mainly focused on reducing
privacy risks on the server side, assuming that other clients will not act maliciously. In this work, we analyzed various methods for improving the
privacy of client data concerning both the server and other clients for neural networks. Some of these methods include homomorphic encryption, gradient compression, gradient noising, and discussion on possible usage of modified
federated learning systems such as split learning, swarm learning or fully encrypted models. We have analyzed the negative effects of gradient compression and gradient noising on the accuracy of convolutional neural networks used for classification. We have shown the difficulty of data reconstruction in the case of
segmentation networks. We have also implemented a proof of concept on the NVIDIA Jetson TX2 module used in edge devices and simulated a
federated learning process.
Read the full abstract ▾
Computer Vision
Rotation-Robust Regression with Convolutional Model Trees
Hongyi Li, William Ward Armstrong and Jun Xu
We study rotation-robust learning for image inputs using Convolutional Model Trees (CMTs) [1], whose split and leaf coefficients can be structured on the image grid and transformed geometrically at deployment time. In a controlled MNIST setting with a rotation-invariant regression target, we introduce three geometry-aware inductive biases for split directions – convolutional smoothing, a tilt dominance constraint, and importance-based
pruning – and quantify their impact on robustness under in-plane rotations. We further evaluate a deployment-time orientation search that selects a discrete rotation maximizing a forest-level confidence proxy without updating model parameters. Orientation search improves robustness under severe rotations but can be harmful near the canonical orientation when confidence is misaligned with correctness. Finally, we observe consistent trends on MNIST digit recognition implemented as one-vs-rest regression, highlighting both the promise and limitations of confidence-based orientation selection for model-tree ensembles.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
V-FAT: Benchmarking Visual Fidelity Against Text-bias
Ziteng Wang, Yujie He, Guanliang Li et al. (+3 more)
Recent advancements in
Multimodal Large
Language Models (MLLMs) have demonstrated impressive performance on standard visual
reasoning benchmarks. However, there is growing concern that these models rely excessively on linguistic shortcuts rather than genuine visual grounding, a phenomenon we term Text Bias. In this paper, we investigate the fundamental tension between visual perception and linguistic priors. We decouple the sources of this bias into two dimensions: Internal Corpus Bias, stemming from statistical correlations in pretraining, and External Instruction Bias, arising from the
alignment-induced tendency toward sycophancy. To quantify this effect, we introduce V-FAT (Visual Fidelity Against Text-bias), a diagnostic
benchmark comprising 4,026 VQA instances across six semantic domains. V-FAT employs a Three-Level Evaluation Framework that systematically increases the conflict between visual evidence and textual information: (L1) internal bias from atypical images, (L2) external bias from misleading instructions, and (L3) synergistic bias where both coincide. We introduce the Visual Robustness Score (VRS), a metric designed to penalize "lucky" linguistic guesses and reward true visual fidelity. Our evaluation of 12 frontier MLLMs reveals that while models excel in existing
benchmarks, they experience significant visual collapse under high linguistic dominance.
Read the full abstract ▾
All authors Ziteng Wang, Yujie He, Guanliang Li, Siqi Yang, Jiaqi Xiong, Songxiang Liu
Computer Vision
Scaling Vision Language Models for Pharmaceutical Long Form Video Reasoning on Industrial GenAI Platform
Suyash Mishra, Qiang Li, Srikanth Patil et al. (+2 more)
Vision Language Models (VLMs) have shown strong performance on
multimodal reasoning tasks, yet most evaluations focus on short
videos and assume unconstrained computational resources. In industrial settings such as pharmaceutical content understanding, practitioners must process long-form
videos under strict GPU, latency, and cost constraints, where many existing approaches fail to scale. In this work, we present an industrial GenAI framework that processes over 200,000 PDFs, 25,326
videos across eight formats (e.g., MP4, M4V, etc.), and 888
multilingual audio files in more than 20
languages. Our study makes three contributions: (i) an industrial large-scale architecture for
multimodal reasoning in pharmaceutical domains; (ii) empirical analysis of over 40 VLMs on two leading
benchmarks (
Video-MME and MMBench) and proprietary
dataset of 25,326
videos across 14 disease areas; and (iii) four findings relevant to long-form
video reasoning: the role of multimodality,
attention mechanism trade-offs, temporal
reasoning limits, and challenges of
video splitting under GPU constraints. Results show 3-8 times efficiency gains with SDPA
attention on commodity GPUs, multimodality improving up to 8/12 task domains (especially length-dependent tasks), and clear bottlenecks in temporal
alignment and keyframe detection across open- and closed-source VLMs. Rather than proposing a new "A+B" model, this paper characterizes practical limits, trade-offs, and failure patterns of current VLMs under realistic deployment constraints, and provide actionable guidance for both researchers and practitioners designing scalable
multimodal systems for long-form
video understanding in industrial domains.
Read the full abstract ▾
All authors Suyash Mishra, Qiang Li, Srikanth Patil, Satyanarayan Pati, Baddu Narendra
Computer Vision
DivAS: Interactive 3D Segmentation by Depth-Weighted Voxel Aggregation
Ayush Pande and Mayank Vatsa
Interactive 3D
segmentation of a reconstructed scene should not require a representation-specific optimization loop. We observe that the recipe for lifting 2D foundation-model masks into 3D, namely prompting a few views, refining the resulting masks with rendered depth, and fusing the multi-view evidence into a voxel grid, is shared across scene representations. What remains representation-specific is only the depth signal returned by the renderer and the occupancy prior that gates fusion. We present **DivAS** (Depth-interactive Voxel Aggregation
Segmentation), an optimization-free, training-free framework that realizes this recipe as a single interaction-and-fusion skeleton with lightweight, representation-specific adapters, instantiated on both Gaussian Splatting (GS) and NeRF backbones. On standard forward-facing and unbounded
benchmarks, the GS instantiation attains
segmentation quality competitive with state-of-the-art optimization-based methods, and the best on LLFF, while being the only one to reach this quality within the consumer-hardware
memory envelope at standard resolution. Both instantiations run end-to-end around $2$x faster than feature-field baselines, with a per-update fusion-kernel cost below $70$ ms. Because
segmentation evidence is gathered from a small, bounded set of anchor views, user effort and computation remain independent of the training-set size. The same skeleton applied to a NeRF backbone matches or exceeds the performance of optimization-based NeRF baselines, confirming that the recipe transfers across fundamentally different 3D representations.
Read the full abstract ▾
Computer Vision
Character Detection using YOLO for Writer Identification in multiple Medieval books
Alessandra Scotto di Freca, Tiziana D Alessandro, Francesco Fontanella et al. (+2 more)
Paleography is the study of ancient and historical handwriting, its key objectives include the dating of manuscripts and understanding the
evolution of writing. Estimating when a document was written and tracing the development of scripts and writing styles can be aided by identifying the individual scribes who contributed to a medieval manuscript. Although digital technologies have made significant progress in this field, the general problem remains unsolved and continues to pose open challenges. ... We previously proposed an approach focused on identifying specific letters or abbreviations that characterize each writer. In that study, we considered the letter "a", as it was widely present on all pages of text and highly distinctive, according to the suggestions of expert paleographers. We used template matching techniques to detect the occurrences of the character "a" on each page and the convolutional neural network (CNN) to attribute each instance to the correct scribe. Moving from the interesting results achieved from this previous system and being aware of the limitations of the template matching technique, which requires an appropriate threshold to work, we decided to experiment in the same framework with the use of the YOLO object detection model to identify the scribe who contributed to the writing of different medieval books. We considered the fifth version of YOLO to implement the YOLO object detection model, which completely substituted the template matching and CNN used in the previous work. The experimental results demonstrate that YOLO effectively extracts a greater number of letters considered, leading to a more accurate second-stage classification. Furthermore, the YOLO confidence score provides a foundation for developing a system that applies a rejection threshold, enabling reliable writer identification even in unseen manuscripts.
Read the full abstract ▾
All authors Alessandra Scotto di Freca, Tiziana D Alessandro, Francesco Fontanella, Filippo Sarria, Claudio De Stefano
Computer Vision · Cross-listed from physics.optics
Illumination Angular Spectrum Encoding for Controlling the Functionality of Diffractive Networks
Matan Kleiner, Lior Michaeli and Tomer Michaeli
Diffractive neural networks have recently emerged as a promising framework for all-optical computing. However, these networks are typically trained for a single task, limiting their potential adoption in systems requiring multiple functionalities. Existing approaches to achieving multi-task functionality either modify the mechanical configuration of the network per task or use a different illumination wavelength or polarization state for each task. In this work, we propose a new control mechanism, which is based on the illumination's angular spectrum. Specifically, we shape the illumination using an amplitude mask that selectively controls its angular spectrum. We employ different illumination masks for achieving different network functionalities, so that the mask serves as a unique task encoder. Interestingly, we show that effective control can be achieved over a very narrow angular range, within the paraxial regime. We numerically illustrate the proposed approach by training a single diffractive network to perform multiple image-to-image translation tasks. In particular, we demonstrate translating handwritten digits into typeset digits of different values, and translating handwritten English letters into typeset numbers and typeset Greek letters, where the type of the output is determined by the illumination's angular components. As we show, the proposed framework can work under different coherence conditions, and can be combined with existing control strategies, such as different wavelengths. Our results establish the illumination angular spectrum as a powerful degree of freedom for controlling diffractive networks, enabling a scalable and versatile framework for multi-task all-optical computing.
Read the full abstract ▾
Computer Vision
Oriol Rabasseda, Zenjie Li, Kamal Nasrollahi et al. (+1 more)
Automatic identification of events and recurrent behavior analysis are critical for
video surveillance. However, most existing content-based
video retrieval benchmarks focus on scene-level similarity and do not evaluate the action discrimination required in surveillance. To address this gap, we introduce SOVABench (Surveillance Opposite Vehicle Actions
Benchmark), a real-world
retrieval benchmark built from surveillance footage and centered on vehicle-related actions. SOVABench defines two evaluation protocols (inter-pair and intra-pair) to assess cross-action discrimination and temporal direction understanding. Although action distinctions are generally intuitive for human observers, our experiments show that they remain challenging for state-of-the-art
vision and
multimodal models. Leveraging the visual
reasoning and instruction-following capabilities of
Multimodal Large
Language Models (MLLMs), we present a training-free framework for producing interpretable
embeddings from MLLM-generated descriptions for both images and
videos. The framework achieves strong performance on SOVABench as well as on several spatial and counting
benchmarks where contrastive
Vision-
Language Models often fail. The code, annotations, and instructions to construct the
benchmark are publicly available.
Read the full abstract ▾
All authors Oriol Rabasseda, Zenjie Li, Kamal Nasrollahi, Sergio Escalera
Computer Vision
Integrated Framework for Selecting and Enhancing Ancient Marathi Inscription Images from Stone, Metal Plate, and Paper Documents
Bapu D. Chendage and Rajivkumar S. Mente
Ancient script images often suffer from severe background noise, low contrast, and degradation caused by aging and environmental effects. In many cases, the foreground text and background exhibit similar visual characteristics, making the inscriptions difficult to read. The primary objective of image enhancement is to improve the readability of such degraded ancient images. This paper presents an image enhancement approach based on binarization and complementary preprocessing techniques for removing stains and enhancing unclear ancient text. The proposed methods are evaluated on different types of ancient scripts, including inscriptions on stone, metal plates, and historical documents. Experimental results show that the proposed approach achieves classification accuracies of 55.7%, 62%, and 65.6% for stone, metal plate, and document scripts, respectively, using the K-Nearest Neighbor (K-NN) classifier. Using the Support Vector Machine (SVM) classifier, accuracies of 53.2%, 59.5%, and 67.8% are obtained. The results demonstrate the effectiveness of the proposed enhancement method in improving the readability of ancient Marathi inscription images.
Read the full abstract ▾
Computer Vision
Detector-Augmented SAMURAI for Long-Duration Drone Tracking
Tamara R. Lenhard, Andreas Weinmann, Hichem Snoussi et al. (+1 more)
Robust long-term tracking of drone is a critical requirement for modern surveillance systems, given their increasing threat potential. While detector-based approaches typically achieve strong frame-level accuracy, they often suffer from temporal inconsistencies caused by frequent detection dropouts. Despite its practical relevance, research on RGB-based drone tracking is still limited and largely reliant on conventional motion models. Meanwhile, foundation models like SAMURAI have established their effectiveness across other domains, exhibiting strong category-agnostic tracking performance. However, their applicability in drone-specific scenarios has not been investigated yet. Motivated by this gap, we present the first systematic evaluation of SAMURAI's potential for robust drone tracking in urban surveillance settings. Furthermore, we introduce a detector-augmented extension of SAMURAI to mitigate sensitivity to bounding-box initialization and sequence length. Our findings demonstrate that the proposed extension significantly improves robustness in complex urban environments, with pronounced benefits in long-duration sequences - especially under drone exit-re-entry events. The incorporation of detector cues yields consistent gains over SAMURAI's zero-shot performance across
datasets and metrics, with success rate improvements of up to +0.393 and FNR reductions of up to -0.475.
Read the full abstract ▾
All authors Tamara R. Lenhard, Andreas Weinmann, Hichem Snoussi, Tobias Koch
Computer Vision
PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inference
Denis Korzhenkov, Adil Karjauv, Animesh Karnewar et al. (+2 more)
Recently proposed pyramidal models decompose the conventional forward and backward
diffusion processes into multiple stages operating at varying resolutions. These models handle inputs with higher noise levels at lower resolutions, while less noisy inputs are processed at higher resolutions. This hierarchical approach significantly reduces the computational cost of inference in multi-step denoising models. However, existing open-source pyramidal
video models have been trained from scratch and tend to underperform compared to state-of-the-art systems in terms of visual plausibility. In this work, we present a pipeline that converts a pretrained
diffusion model into a pyramidal one through low-cost finetuning, achieving this transformation without degradation in quality of output
videos. Furthermore, we investigate and compare various strategies for step
distillation within pyramidal models, aiming to further enhance the inference efficiency. Our results are available at https://qualcomm-ai-research.github.io/PyramidalWan.
Read the full abstract ▾
All authors Denis Korzhenkov, Adil Karjauv, Animesh Karnewar, Mohsen Ghafoorian, Amirhossein Habibian