Computer Vision
Visual Product Search Benchmark
Karthik Sulthanpete Govindappa
Reliable product identification from images is a critical requirement in industrial and commercial applications, particularly in maintenance, procurement, and operational workflows where incorrect matches can lead to costly downstream failures. At the core of such systems lies the visual search component, which must retrieve and rank the exact object instance from large and continuously evolving catalogs under diverse imaging conditions. This report presents a structured
benchmark of modern visual
embedding models for instance-level image
retrieval, with a focus on industrial applications. A curated set of open-source foundation
embedding models, proprietary multi-modal
embedding systems, and domain-specific
vision-only models are evaluated under a unified image-to-image
retrieval protocol. The
benchmark includes curated
datasets, which includes industrial
datasets derived from production deployments in Manufacturing, Automotive, DIY, and Retail, as well as established public
benchmarks. Evaluation is conducted without post-processing, isolating the
retrieval capability of each model. The results provide insight into how well contemporary foundation and unified
embedding models transfer to fine-grained instance
retrieval tasks, and how they compare to models explicitly trained for industrial applications. By emphasizing realistic constraints, heterogeneous image conditions, and exact instance matching requirements, this
benchmark aims to inform both practitioners and researchers about the strengths and limitations of current visual
embedding approaches in production-level product identification systems. An interactive companion website presenting the
benchmark results, evaluation details, and additional visualizations is available at https://
benchmark.nyris.io.
Read the full abstract ▾
Computer Vision
Patient4D: Temporally Consistent Patient Body Mesh Recovery from Monocular Operating Room Video
Mingxiao Tu, Hoijoon Jung, Alireza Moghadam et al. (+2 more)
Recovering a dense 3D body mesh from
monocular video remains challenging under occlusion from draping and continuously moving camera viewpoints. This configuration arises in
surgical augmented reality (AR), where an anesthetized patient lies under
surgical draping while a surgeon's head-mounted camera continuously changes viewpoint. Existing human mesh recovery (HMR) methods are typically trained on upright, moving subjects captured from relatively stable cameras, leading to performance degradation under such conditions. To address this, we present Patient4D, a stationarity-constrained reconstruction pipeline that explicitly exploits the stationarity prior. The pipeline combines image-level foundation models for perception with lightweight
geometric mechanisms that enforce temporal consistency across frames. Two key components enable robust reconstruction: Pose Locking, which anchors pose parameters using stable keyframes, and Rigid Fallback, which recovers meshes under severe occlusion through silhouette-guided rigid
alignment. Together, these mechanisms stabilize predictions while remaining compatible with off-the-shelf HMR models. We evaluate Patient4D on 4,680 synthetic
surgical sequences and three public HMR
video benchmarks. Under
surgical drape occlusion, Patient4D achieves a 0.75 mean IoU, reducing failure frames from 30.5% to 1.3% compared to the best baseline. Our findings demonstrate that exploiting stationarity priors can substantially improve
monocular reconstruction in clinical AR scenarios.
Read the full abstract ▾
All authors Mingxiao Tu, Hoijoon Jung, Alireza Moghadam, Andre Kyme, Jinman Kim
Computer Vision
Generalist Multimodal LLMs Gain Biometric Expertise via Human Salience
Jacob Piland, Byron Dowling, Christopher Sweet et al. (+1 more)
Iris presentation attack detection (PAD) is critical for secure biometric deployments, yet developing specialized models faces significant practical barriers: collecting data representing future unknown attacks is impossible, and collecting diverse-enough data, yet still limited in terms of its predictive power, is expensive. Additionally, sharing biometric data raises
privacy concerns. Due to rapid emergence of new attack vectors demanding adaptable solutions, we thus investigate in this paper whether general-purpose
multimodal large
language models (MLLMs) can perform iris PAD when augmented with human expert knowledge, operating under strict
privacy constraints that prohibit sending biometric data to public cloud MLLM services. Through analysis of
vision encoder
embeddings applied to our
dataset, we demonstrate that pre-trained
vision transformers in MLLMs inherently cluster many iris attack types despite never being explicitly trained for this task. However, where clustering shows overlap between attack classes, we find that structured
prompts incorporating human salience (verbal descriptions from subjects identifying attack indicators) enable these models to resolve ambiguities. Testing on an IRB-restricted
dataset of 224 iris images spanning seven attack types, using only university-approved services (Gemini 2.5 Pro) or locally-hosted models (e.g., Llama 3.2-
Vision), we show that Gemini with expert-informed
prompts outperforms both a specialized convolutional neural networks (CNN)-based baseline and human examiners, while the locally-deployable Llama achieves near-human performance. Our results establish that MLLMs deployable within institutional
privacy constraints offer a viable path for iris PAD.
Read the full abstract ▾
All authors Jacob Piland, Byron Dowling, Christopher Sweet, Adam Czajka
Computer Vision · Cross-listed from cs.RO
SLAM Adversarial Lab: An Extensible Framework for Visual SLAM Robustness Evaluation under Adverse Conditions
Mohamed Hefny, Karthik Dantu and Steven Y. Ko
We present SAL (SLAM
Adversarial Lab), a modular framework for evaluating visual SLAM systems under
adversarial conditions such as fog and rain. SAL represents each
adversarial condition as a perturbation that transforms an existing
dataset into an
adversarial dataset. When transforming a
dataset, SAL supports severity levels using easily-interpretable real-world units such as meters for fog visibility. SAL's extensible architecture decouples
datasets, perturbations, and SLAM
algorithms through common interfaces, so users can add new components without rewriting integration code. Moreover, SAL includes a search procedure that finds the severity level of a perturbation at which a SLAM system fails. To showcase the capabilities of SAL, our evaluation integrates seven SLAM
algorithms and evaluates them across three
datasets under weather, camera, and
video transport perturbations.
Read the full abstract ▾
Computer Vision
GazeOnce360: Fisheye-Based 360{\deg} Multi-Person Gaze Estimation with Global-Local Feature Fusion
Zhuojiang Cai, Zhenghui Sun and Feng Lu
We present GazeOnce360, a novel end-to-end model for multi-person gaze estimation from a single tabletop-mounted upward-facing fisheye camera. Unlike conventional approaches that rely on forward-facing cameras in constrained viewpoints, we address the underexplored setting of estimating the 3D gaze direction of multiple people distributed across a 360{\deg} scene from an upward fisheye perspective. To support research in this setting, we introduce MPSGaze360, a large-scale synthetic
dataset rendered using Unreal Engine, featuring diverse multi-person configurations with accurate 3D gaze and eye landmark annotations. Our model tackles the severe distortion and perspective variation inherent in fisheye imagery by incorporating rotational convolutions and eye landmark supervision. To better capture fine-grained eye features crucial for gaze estimation, we propose a dual-resolution architecture that fuses global low-resolution context with high-resolution local eye regions. Experimental results demonstrate the effectiveness of each component in our model. This work highlights the feasibility and potential of fisheye-based 360{\deg} gaze estimation in practical multi-person scenarios. Project page: https://caizhuojiang.github.io/GazeOnce360/.
Read the full abstract ▾
Computer Vision
BEV-SLD: Self-Supervised Scene Landmark Detection for Global Localization with LiDAR Bird's-Eye View Images
David Skuddis, Vincent Ress, Wei Zhang et al. (+2 more)
We present BEV-SLD, a
LiDAR global localization method building on the Scene Landmark Detection (SLD) concept. Unlike scene-agnostic pipelines, our self-supervised approach leverages bird's-eye-view (BEV) images to discover scene-specific patterns at a prescribed spatial density and treat them as landmarks. A consistency loss aligns learnable global landmark coordinates with per-frame heatmaps, yielding consistent landmark detections across the scene. Across campus, industrial, and forest environments, BEV-SLD delivers robust localization and achieves strong performance compared to state-of-the-art methods.
Read the full abstract ▾
All authors David Skuddis, Vincent Ress, Wei Zhang, Vincent Ofosu Nyako, Norbert Haala
Computer Vision · Cross-listed from eess.IV
A Lensless Polarization Camera
Noa Kraicer, Shay Elmalem, Erez Yosef et al. (+2 more)
Polarization imaging is a technique that creates a pixel map of the polarization state in a scene. Although invisible to the human eye, polarization can assist various sensing and computer
vision tasks. Existing polarization cameras use spatial or temporal multiplexing, which increases the camera volume, weight, cost, or all of the above. Recent lensless imaging approaches, such as DiffuserCam, have demonstrated that compact imaging systems can be realized by replacing the lens with a coding element and performing computational reconstruction. In this work, we propose a compact lensless polarization camera composed of a diffuser and a simple striped polarization mask. By combining this optical design with a reconstruction
algorithm that explicitly models the polarization-encoded lensless measurements, four linear polarization images are recovered from a single snapshot. Our results demonstrate the potential of lensless approaches for polarization imaging and reveal the physical factors that govern reconstruction quality, guiding the development of high-quality practical systems.
Read the full abstract ▾
All authors Noa Kraicer, Shay Elmalem, Erez Yosef, Hani Barhum, Raja Giryes
Computer Vision
SMAL-pets: SMAL Based Avatars of Pets from Single Image
Piotr Borycki, Joanna Waczyńska, Yizhe Zhu et al. (+2 more)
Creating high-fidelity, animatable 3D dog avatars remains a formidable challenge in computer
vision. Unlike human digital doubles, animal reconstruction faces a critical shortage of large-scale, annotated
datasets for specialized applications. Furthermore, the immense morphological diversity across species, breeds, and crosses, which varies significantly in size, proportions, and features, complicates the generalization of existing models. Current reconstruction methods often struggle to capture realistic fur textures. Additionally, ensuring these avatars are fully editable and capable of performing complex, naturalistic movements typically necessitates labor-intensive manual mesh manipulation and expert rigging. This paper introduces SMAL-pets, a comprehensive framework that generates high-quality, editable animal avatars from a single input image. Our approach bridges the gap between reconstruction and generative modeling by leveraging a hybrid architecture. Our method integrates 3D Gaussian Splatting with the SMAL parametric model to provide a representation that is both visually high-fidelity and anatomically grounded. We introduce a
multimodal editing suite that enables users to refine the avatar's appearance and execute complex animations through direct textual
prompts. By allowing users to control both the aesthetic and behavioral aspects of the model via natural
language, SMAL-pets provides a flexible, robust tool for animation and virtual reality.
Read the full abstract ▾
All authors Piotr Borycki, Joanna Waczyńska, Yizhe Zhu, Yongqiang Gao, Przemysław Spurek
Computer Vision
MosaicMem: Hybrid Spatial Memory for Controllable Video World Models
Wei Yu, Runjia Qian, Yumeng Li et al. (+8 more)
Video diffusion models are moving beyond short, plausible clips toward world simulators that must remain consistent under camera motion, revisits, and intervention. Yet spatial
memory remains a key bottleneck: explicit 3D structures can improve reprojection-based consistency but struggle to depict moving objects, while implicit
memory often produces inaccurate camera motion even with correct poses. We propose Mosaic
Memory (MosaicMem), a hybrid spatial
memory that lifts patches into 3D for reliable localization and targeted
retrieval, while exploiting the model's native conditioning to preserve
prompt-following generation. MosaicMem composes spatially aligned patches in the queried view via a patch-and-compose interface, preserving what should persist while allowing the model to inpaint what should evolve. With PRoPE camera conditioning and two new
memory alignment methods, experiments show improved pose adherence compared to implicit
memory and stronger dynamic modeling than explicit baselines. MosaicMem further enables minute-level
navigation,
memory-based scene editing, and autoregressive rollout.
Read the full abstract ▾
All authors Wei Yu, Runjia Qian, Yumeng Li, Liquan Wang, Songheng Yin, Sri Siddarth Chakaravarthy P, Dennis Anthony, Yang Ye, Yidi Li, Weiwei Wan, Animesh Garg
Computer Vision
Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles
Zacharie Bugaud
Ensembling
Vision-
Language Models (VLMs) from different providers maximizes
benchmark accuracy, yet models from the same architectural family share correlated errors that standard voting ignores. We study this structure across 17 VLMs from 8 families on VQAv2, TextVQA, and GQA. Family-correlated errors reduce effective ensemble dimensionality to 2.5-3.6 independent voters and create a Misleading tier (1.5-6.5% of questions) where correlated majority errors destroy accuracy to 0% despite the best model being correct. We propose three family-aware methods. Hierarchical Family Voting (HFV) aggregates within families before voting across them, recovering +18-26 pp on the Misleading tier. QualRCCV, a training-free method weighting models by calibration, family quality, and inverse family size, is the first to beat calibrated voting on all three
benchmarks (p<0.05). Learned Candidate Scoring (LCS) trains a cross-validated classifier to re-rank candidate answers using support breadth, family diversity, and model quality, achieving the largest gains: +0.68% VQAv2, +0.61% TextVQA, +2.45% GQA – all significant – and is the only learned method that never degrades any
benchmark. On VQAv2 test-standard (EvalAI), LCS reaches 87.83% with 12 models, confirming generalization.
Read the full abstract ▾
Computer Vision
Pixel-level Counterfactual Contrastive Learning for Medical Image Segmentation
Marceau Lafargue-Hauret, Raghav Mehta, Fabio De Sousa Ribeiro et al. (+2 more)
Image
segmentation relies on large annotated
datasets, which are expensive and slow to produce. Silver-standard (AI-generated) labels are easier to obtain, but they risk introducing bias. Self-supervised learning, needing only images, has become key for pre-training. Recent work combining contrastive learning with counterfactual generation improves representation learning for classification but does not readily extend to pixel-level tasks. We propose a pipeline combining counterfactual generation with dense contrastive learning via Dual-View (DVD-CL) and Multi-View (MVD-CL) methods, along with supervised variants that utilize available silver-standard annotations. A new visualisation
algorithm, the Color-coded High Resolution Overlay map (CHRO-map) is also introduced. Experiments show annotation-free DVD-CL outperforms other dense contrastive learning methods, while supervised variants using silver-standard labels outperform training on the silver-standard labeled data directly, achieving $\sim$94% DSC on challenging data. These results highlight that pixel-level contrastive learning, enhanced by counterfactuals and silver-standard annotations, improves robustness to acquisition and pathological variations.
Read the full abstract ▾
All authors Marceau Lafargue-Hauret, Raghav Mehta, Fabio De Sousa Ribeiro, Mélanie Roschewitz, Ben Glocker
Computer Vision
LLM-Powered Flood Depth Estimation from Social Media Imagery: A Vision-Language Model Framework with Mechanistic Interpretability for Transportation Resilience
Nafis Fuad and Xiaodong Qian
Urban flooding poses an escalating threat to transportation network continuity, yet no operational system currently provides real-time, street-level flood depth information at the centimeter resolution required for dynamic routing, electric vehicle (EV) safety, and autonomous vehicle (AV) operations. This study presents FloodLlama, a fine-tuned open-source
vision-
language model (VLM) for continuous flood depth estimation from single street-level images, supported by a
multimodal sensing pipeline using TikTok data. A synthetic
dataset of approximately 190000 images was generated, covering seven vehicle types, four weather conditions, and 41 depth levels (0-40 cm at 1 cm resolution). Progressive curriculum training enabled coarse-to-fine learning, while LLaMA 3.2-11B
Vision was fine-tuned using QLoRA. Evaluation across 34797 trials reveals a depth-dependent
prompt effect: simple
prompts perform better for shallow flooding, whereas chain-of-thought (CoT)
reasoning improves performance at greater depths. FloodLlama achieves a mean absolute error (MAE) below 0.97 cm and Acc@5cm above 93.7% for deep flooding, exceeding 96.8% for shallow depths. A five-phase mechanistic interpretability framework identifies layer L23 as the critical depth-encoding transition and enables selective fine-tuning that reduces trainable parameters by 76-80% while maintaining accuracy. The Tier 3 configuration achieves 98.62% accuracy on real-world data and shows strong robustness under visual occlusion. A TikTok-based data pipeline, validated on 676 annotated flood frames from Detroit, demonstrates the feasibility of real-time, crowd-sourced flood sensing. The proposed framework provides a scalable, infrastructure-free solution with direct implications for EV safety, AV deployment, and resilient transportation management.
Read the full abstract ▾
Computer Vision
Accurate Shift Invariant Convolutional Neural Networks Using Gaussian-Hermite Moments
Jaspreet Singh, Petra Bosilj and Grzegorz Cielniak
The convolutional neural networks (CNNs) are not inherently shift invariant or equivariant. The downsampling operation, used in CNNs, is one of the key reasons which breaks the shift invariant property of a CNN. Conversely, downsampling operation is important to improve computational efficiency and increase the area of the receptive field for more contextual information. In this work, we propose Gaussian-Hermite Sampling (GHS), a novel downsampling strategy designed to achieve accurate shift invariance. GHS leverages Gaussian-Hermite
polynomials to perform shift-consistent sampling, enabling CNN layers to maintain invariance to arbitrary spatial shifts prior to training. When integrated into standard CNN architectures, the proposed method embeds shift invariance directly at the layer level without requiring architectural modifications or additional training procedures. We evaluate the proposed approach on CIFAR-10, CIFAR-100, and MNIST-rot
datasets. Experimental results demonstrate that GHS significantly improves shift consistency, achieving 100% classification consistency under spatial shifts, while also improving classification accuracy compared to baseline CNN models.
Read the full abstract ▾
Computer Vision
ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models
M. Arda Ayd{\i}n, Melih B. Yilmaz, Aykut Koç et al. (+1 more)
The success of CLIP-like
vision-
language models (VLMs) on natural images has inspired medical counterparts, yet existing approaches largely fall into two extremes: specialist models trained on single-domain data, which capture domain-specific details but generalize poorly, and generalist medical VLMs trained on multi-domain data, which retain broad semantics but dilute fine-grained diagnostic cues. Bridging this specialization-generalization trade-off remains challenging. To address this problem, we propose ACE-LoRA, a parameter-efficient adaptation framework for generalist medical VLMs that maintains robust zero-shot generalization. ACE-LoRA integrates Low-Rank Adaptation (LoRA) modules into frozen image-text encoders and introduces an
Attention-based Context Enhancement Hypergraph Neural Network (ACE-HGNN) module that captures higher-order contextual interactions beyond pairwise similarity to enrich global representations with localized diagnostic cues, addressing a key limitation of prior Parameter-Efficient Fine-Tuning (PEFT) methods that overlook fine-grained details. To further enhance cross-modal
alignment, we formulate a label-guided InfoNCE loss to effectively suppress false negatives between semantically related image-text pairs. Despite adding only 0.95M trainable parameters, ACE-LoRA consistently outperforms state-of-the-art medical VLMs and PEFT baselines across zero-shot classification,
segmentation, and detection
benchmarks spanning multiple domains. Our code is available at https://github.com/icon-lab/ACE-LoRA.
Read the full abstract ▾
All authors M. Arda Ayd{\i}n, Melih B. Yilmaz, Aykut Koç, Tolga Çukur
Computer Vision
Edge-Efficient Two-Stream Multimodal Architecture for Non-Intrusive Bathroom Fall Detection
Haitian Wang, Yiren Wang, Xinyu Wang et al. (+2 more)
Falls in wet bathroom environments are a major safety risk for seniors living alone. Recent work has shown that mmWave-only, vibration-only, and existing
multimodal schemes, such as vibration-triggered radar activation, early feature concatenation, and decision-level score fusion, can support
privacy-preserving, non-intrusive fall detection. However, these designs still treat motion and impact as loosely coupled streams, depending on coarse temporal
alignment and amplitude thresholds, and do not explicitly encode the causal link between radar-observed collapse and floor impact or address timing drift, object drop confounders, and latency and energy constraints on low-power edge devices. To this end, we propose a two-stream architecture that encodes radar signals with a Motion–Mamba branch for long-range motion patterns and processes floor vibration with an Impact–Griffin branch that emphasizes impact transients and cross-axis coupling. Cross-conditioned fusion uses low-rank bilinear interaction and a Switch–MoE head to align motion and impact
tokens and suppress object-drop confounders. The model keeps inference cost suitable for real-time execution on a Raspberry Pi 4B gateway. We construct a bathroom fall detection
benchmark dataset with frame-level annotations, comprising more than 3~h of synchronized mmWave radar and triaxial vibration recordings across eight scenarios under running water, together with subject-independent training, validation, and test splits. On the test split, our model attains 96.1% accuracy, 94.8% precision, 88.0% recall, a 91.1% macro F1 score, and an AUC of 0.968. Compared with the strongest baseline, it improves accuracy by 2.0 percentage points and fall recall by 1.3 percentage points, while reducing latency from 35.9 ms to 15.8 ms and lowering energy per 2.56 s window from 14200 mJ to 10750 mJ on the Raspberry Pi 4B gateway.
Read the full abstract ▾
All authors Haitian Wang, Yiren Wang, Xinyu Wang, Sheldon Fung, Atif Mansoor
Computer Vision
TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects
Yeheng Zong, Yizhou Chen, Alexander Bowler et al. (+2 more)
Structured 3D representations such as keypoints and meshes offer compact, expressive descriptions of deformable objects, jointly capturing
geometric and topological information useful for downstream tasks such as dynamics modeling and motion planning. However, robustly extracting such representations remains challenging, as current perception methods struggle to handle complex deformations. Moreover, large-scale 3D data collection remains a bottleneck: existing approaches either require prohibitive data collection efforts, such as labor-intensive annotation or expensive motion capture setups, or rely on simplifying assumptions that break down in unstructured environments. As a result, large-scale 3D
datasets and
benchmarks for deformable objects remain scarce. To address these challenges, this paper presents an affordable and autonomous framework for collecting 3D
datasets of deformable objects using only RGB-D cameras. The proposed method identifies 3D keypoints and robustly tracks their trajectories, incorporating motion consistency constraints to produce temporally smooth and geometrically coherent data. TrackDeform3D is evaluated against several state-of-the-art tracking methods across diverse object categories and demonstrates consistent improvements in both
geometric and tracking accuracy. Using this framework, this paper presents a high-quality, large-scale
dataset consisting of 6 deformable objects, totaling 110 minutes of trajectory data. Project page: https://roahmlab.github.io/trackDeform3D-core-tracking/
Read the full abstract ▾
All authors Yeheng Zong, Yizhou Chen, Alexander Bowler, Chia-Tung Yang, Ram Vasudevan
Computer Vision
DesertFormer: Transformer-Based Semantic Segmentation for Off-Road Desert Terrain Classification in Autonomous Navigation Systems
Yasaswini Chebolu
Reliable terrain perception is a fundamental requirement for autonomous
navigation in unstructured, off-road environments. Desert landscapes present unique challenges due to low chromatic contrast between terrain categories, extreme lighting variability, and sparse vegetation that defy the assumptions of standard road-scene
segmentation models. We present DesertFormer, a semantic
segmentation pipeline for off-road desert terrain analysis based on SegFormer B2 with a hierarchical Mix
Transformer (MiT-B2) backbone. The system classifies terrain into ten ecologically meaningful categories – Trees, Lush Bushes, Dry Grass, Dry Bushes, Ground Clutter, Flowers, Logs, Rocks, Landscape, and Sky – enabling safety-aware path planning for ground
robots and autonomous vehicles. Trained on a purpose-built
dataset of 4,176 annotated off-road images at 512x512 resolution, DesertFormer achieves a mean Intersection-over-Union (mIoU) of 64.4% and pixel accuracy of 86.1%, representing a +24.2% absolute improvement over a DeepLabV3 MobileNetV2 baseline (41.0% mIoU). We further contribute a systematic failure analysis identifying the primary confusion patterns – Ground Clutter to Landscape and Dry Grass to Landscape – and propose class-weighted training and copy-paste augmentation for rare terrain categories. Code, checkpoints, and an interactive inference dashboard are released at https://github.com/Yasaswini-ch/
Vision-based-Desert-Terrain-
Segmentation-using-SegFormer.
Read the full abstract ▾
Computer Vision
PaAgent: Portrait-Aware Image Restoration Agent via Subjective-Objective Reinforcement Learning
Yijian Wang, Qingsen Yan, Jiantao Zhou et al. (+2 more)
Image Restoration (IR)
agents, leveraging
multimodal large
language models to perceive degradation and invoke restoration tools, have shown promise in automating IR tasks. However, existing IR
agents typically lack an insight summarization mechanism for past interactions, which results in an exhaustive search for the optimal IR tool. To address this limitation, we propose a portrait-aware IR
agent, dubbed PaAgent, which incorporates a self-evolving portrait bank for IR tools and
Retrieval-Augmented Generation (RAG) to select a suitable IR tool for input. Specifically, to construct and evolve the portrait bank, the PaAgent continuously enriches it by summarizing the characteristics of various IR tools with restored images, selected IR tools, and degraded images. In addition, the RAG is employed to select the optimal IR tool for the input image by retrieving relevant insights from the portrait bank. Furthermore, to enhance PaAgent's ability to perceive degradation in complex scenes, we propose a subjective-objective reinforcement learning strategy that considers both image quality scores and semantic insights in reward generation, which accurately provides the degradation information even under partial and non-uniform degradation. Extensive experiments across 8 IR
benchmarks, covering six single-degradation and eight mixed-degradation scenarios, validate PaAgent's superiority in addressing complex IR tasks. Our project page is \href{https://wyjgr.github.io/PaAgent.html}{PaAgent}.
Read the full abstract ▾
All authors Yijian Wang, Qingsen Yan, Jiantao Zhou, Duwei Dai, Wei Dong
Computer Vision
Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models
Songchun Zhang, Zeyue Xue, Siming Fu et al. (+6 more)
Distilled autoregressive (AR)
video models enable efficient streaming generation but frequently misalign with human visual preferences. Existing reinforcement learning (RL) frameworks are not naturally suited to these architectures, typically requiring either expensive re-
distillation or solver-coupled reverse-process optimization that introduces considerable
memory and computational overhead. We present Astrolabe, an efficient online RL framework tailored for distilled AR models. To overcome existing bottlenecks, we introduce a forward-process RL formulation based on negative-aware fine-tuning. By contrasting positive and negative samples directly at inference endpoints, this approach establishes an implicit policy improvement direction without requiring reverse-process unrolling. To scale this
alignment to long
videos, we propose a streaming training scheme that generates sequences progressively via a rolling KV-cache, applying RL updates exclusively to local clip windows while conditioning on prior context to ensure long-range coherence. Finally, to mitigate reward hacking, we integrate a multi-reward objective stabilized by uncertainty-aware selective regularization and dynamic reference updates. Extensive experiments demonstrate that our method consistently enhances generation quality across multiple distilled AR
video models, serving as a robust and scalable
alignment solution.
Read the full abstract ▾
All authors Songchun Zhang, Zeyue Xue, Siming Fu, Jie Huang, Xianghao Kong, Y Ma, Haoyang Huang, Nan Duan, Anyi Rao
Computer Vision · Cross-listed from cs.LG
SCE-LITE-HQ: Smooth visual counterfactual explanations with generative foundation models
Ahmed Zeid and Sidney Bender
Modern neural networks achieve strong performance but remain difficult to interpret in high-dimensional visual domains. Counterfactual explanations (CFEs) provide a principled approach to interpreting black-box predictions by identifying minimal input changes that alter model outputs. However, existing CFE methods often rely on
dataset-specific generative models and incur substantial computational cost, limiting their scalability to high-resolution data. We propose SCE-LITE-HQ, a scalable framework for counterfactual generation that leverages pretrained generative foundation models without task-specific retraining. The method operates in the latent space of the generator, incorporates smoothed gradients to improve optimization stability, and applies mask-based diversification to promote realistic and structurally diverse counterfactuals. We evaluate SCE-LITE-HQ on natural and medical
datasets using a desiderata-driven evaluation protocol. Results show that SCE-LITE-HQ produces valid, realistic, and diverse counterfactuals competitive with or outperforming existing baselines, while avoiding the overhead of training dedicated generative models.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Do Understanding and Generation Fight? A Diagnostic Study of DPO for Unified Multimodal Models
Abinav Rao and Sujan Rachuri
Unified
multimodal models share a
language model backbone for both understanding and generating images. Can DPO align both capabilities simultaneously? We present the first systematic study of this question, applying DPO to Janus-Pro at 1B and 7B parameters under seven training strategies and two post-hoc methods. The central finding is negative: generation quality resists DPO
alignment across all tested conditions on this architecture. No method improves generation CLIPScore at 7B (|Delta| < 0.2, p > 0.5 at n=200 per seed, 3 seeds); at 1B, all methods degrade generation, and the result holds across preference data types (real-vs-generated and model-vs-model) and the data volumes tested (150-288 pairs). Gradient analysis reveals why: understanding and generation gradients are near-orthogonal (cos ~ 0) with ~11-14x magnitude imbalance driven by VQ
token count asymmetry (576 generation
tokens vs. ~30-100 text
tokens). This imbalance is the dominant interference mechanism in multi-task DPO; magnitude-balancing yields directionally positive understanding deltas (+0.01-0.04 VQA, though individually not significant), but the generation gap persists regardless. We identify discrete VQ tokenization as a likely structural bottleneck – supported by the generation DPO loss converging to ln(2) – and provide practical guidance for practitioners working with VQ-based unified models.
Read the full abstract ▾
Computer Vision
OpenQlaw: An Agentic AI Assistant for Analysis of 2D Quantum Materials
Sankalp Pandey, Thanh-Dat Truong, Xuan-Bac Nguyen et al. (+5 more)
Physics-aware
multimodal large
language models (MLLMs) can localize and characterize two-dimensional (2D) material flakes from optical microscopy images, producing rich candidate observations and scientific
reasoning. However, downstream workflows also require reliable operations over these outputs, including counting, selection, measurement, and rendering. We present OpenQlaw, a tool-using agentic artificial intelligence (AI) system that connects a general
vision-
language orchestrator with Physics-Aware Instruction Tuning (QuPAINT), a physics-aware domain expert. Through evaluation on fully reviewed optical microscopy images, we show that summary counts derived from longer free-form candidate enumerations can become inconsistent with the underlying returned observations. Controlled replay experiments further show that structuring the same observations alone does not resolve this behavior, while deterministic delegation returns the exact requested record count in all cohort trials. To address this, we introduce State-Grounded OpenQlaw, which converts expert outputs into image-scoped observation records and performs deterministic scientific operations over explicitly identified records with refusal conditions when required information is unavailable. This preserves the expert's original analysis while making downstream operations exact over the selected record and independently evaluable from candidate-list discrepancies.
Read the full abstract ▾
All authors Sankalp Pandey, Thanh-Dat Truong, Xuan-Bac Nguyen, Hoang-Quan Nguyen, Tim Faltermeier, Nicholas Borys, Hugh Churchill, Khoa Luu
Computer Vision
HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning
Shenzhi Wang, Shixuan Liu, Jing Zhou et al. (+8 more)
Vision-
language models (VLMs) show strong
multimodal capabilities but still struggle with fine-grained
vision-
language reasoning. We find that long chain-of-thought (CoT)
reasoning exposes diverse failure modes, including perception,
reasoning, knowledge, and
hallucination errors, which can compound across intermediate steps. However, most existing
vision-
language data used for reinforcement learning with verifiable rewards (RLVR) does not involve complex
reasoning chains that rely on visual evidence throughout, leaving these weaknesses largely unexposed. We therefore propose HopChain, a scalable framework for synthesizing multi-hop
vision-
language reasoning data for RLVR training of VLMs. Each synthesized multi-hop query forms a logically dependent chain of instance-grounded hops, where earlier hops establish the instances, sets, or conditions needed for later hops, while the final answer remains a specific, unambiguous number suitable for verifiable rewards. We train Qwen3.5-35B-A3B and Qwen3.5-397B-A17B under two RLVR settings: the original data alone, and the original data plus HopChain's multi-hop data, and compare them across 24
benchmarks spanning STEM and Puzzle, General VQA, Text Recognition and Document Understanding, and
Video Understanding. Although this multi-hop data is not synthesized for any specific
benchmark, it improves 20 of 24
benchmarks on both models, indicating broad and generalizable gains. Consistently, replacing full chained queries with half-multi-hop or single-hop variants reduces the average score across five representative
benchmarks from 70.4 to 66.7 and 64.3, respectively. Notably, multi-hop gains peak in long-CoT
vision-
language reasoning, exceeding 50 points in the ultra-long-CoT regime. These experiments establish HopChain as an effective, scalable framework for synthesizing multi-hop data that improves generalizable
vision-
language reasoning.
Read the full abstract ▾
All authors Shenzhi Wang, Shixuan Liu, Jing Zhou, Chang Gao, Xiong-Hui Chen, Binghai Wang, An Yang, Shiji Song, Bowen Yu, Gao Huang, Junyang Lin
Computer Vision
Empirical Recipes for Efficient and Compact Vision-Language Models
Jiabo Huang, Zhizhong Li, Sina Sajadmanesh et al. (+2 more)
Deploying
vision-
language models (VLMs) in resource-constrained settings demands low latency and high throughput, yet existing compact VLMs often fall short of the inference speedups their smaller parameter counts suggest. To explain this discrepancy, we conduct an empirical end-to-end efficiency analysis and systematically profile inference to identify the dominant bottlenecks. Based on these findings, we develop optimization recipes tailored to compact VLMs that substantially reduce latency while preserving accuracy. These techniques cut time to first
token (TTFT) by 53% on InternVL3-2B and by 93% on SmolVLM-256M. Our recipes are broadly applicable across both VLM architectures and common serving frameworks, providing practical guidance for building efficient VLM systems. Beyond efficiency, we study how to extend compact VLMs with structured perception outputs and introduce the resulting model family, ArgusVLM. Across diverse
benchmarks, ArgusVLM achieves strong performance while maintaining a compact and efficient design.
Read the full abstract ▾
All authors Jiabo Huang, Zhizhong Li, Sina Sajadmanesh, Weiming Zhuang, Lingjuan Lyu
Computer Vision
Are a Thousand Words Better Than a Single Picture? Beyond Images – A Framework for Multi-Modal Knowledge Graph Dataset Enrichment
Pengyu Zhang, Klim Zaporojets, Jie Liu et al. (+2 more)
Multi-Modal Knowledge Graphs (MMKGs) benefit from visual information, yet large-scale image collection is hard to curate and often excludes ambiguous but relevant visuals (e.g., logos, symbols, abstract scenes). We present Beyond Images, an automatic data-centric enrichment pipeline with optional human
auditing. This pipeline operates in three stages: (1) large-scale
retrieval of additional entity-related images, (2) conversion of all visual inputs into textual descriptions to ensure that ambiguous images contribute usable semantics rather than noise, and (3) fusion of multi-source descriptions using a large
language model (LLM) to generate concise, entity-aligned summaries. These summaries replace or augment the text modality in standard MMKG models without changing their architectures or loss functions. Across three public MMKG
datasets and multiple baseline models, we observe consistent gains (up to 7% Hits@1 overall). Furthermore, on a challenging subset of entities with visually ambiguous logos and symbols, converting images into text yields large improvements (201.35% MRR and 333.33% Hits@1). Additionally, we release a lightweight Text-Image Consistency Check Interface for optional targeted audits, improving description quality and
dataset reliability. Our results show that scaling image coverage and converting ambiguous visuals into text is a practical path to stronger MMKG completion. Code,
datasets, and supplementary materials are available at https://github.com/pengyu-zhang/Beyond-Images.
Read the full abstract ▾
All authors Pengyu Zhang, Klim Zaporojets, Jie Liu, Jia-Hong Huang, Paul Groth
Computer Vision
MAND: Modality-Aware Novelty Detection for Open-World Egocentric Activity Recognition
Hyejeong Im, Wonseon Lim and Dae-Won Kim
Multimodal egocentric activity recognition integrates visual and inertial cues for robust first-person behavior understanding. However, deploying such systems in open-world environments requires detecting novel activities while continuously learning from non-stationary data streams. Existing methods rely on the main fused logits for novelty scoring, without fully exploiting the complementary evidence available from individual modalities. Because these logits are often dominated by RGB, cues from other modalities, particularly IMU, remain underutilized, and this imbalance worsens as catastrophic forgetting accumulates. To address this, we propose MAND, a modality-aware framework for
multimodal egocentric open-world continual learning. At inference, Modality-aware Adaptive Scoring (MoAS) adaptively adjusts modality contributions using sample-wise reliability and refines novelty scoring with deviation and disagreement penalties. During training, Modality-aware Representation Stabilization Training (MoRST) preserves the discriminative capacity of each modality across tasks through modality-specific heads and modality-wise logit
distillation. Experiments on a public
multimodal egocentric benchmark show that MAND consistently improves novel activity detection and known-class accuracy while substantially reducing FPR95, indicating more reliable open-world recognition. The source code is available at \href{https://github.com/HyeJeongIm/MAND}{github.com/HyeJeongIm/MAND}.
Read the full abstract ▾
Computer Vision
MSRAMIE: Multimodal Structured Reasoning Agent for Multi-instruction Image Editing
Zhaoyuan Qiu, Ken Chen, Xiangwei Wang et al. (+3 more)
Existing instruction-based image editing models perform well with simple, single-step instructions but degrade in realistic scenarios that involve multiple, lengthy, and interdependent directives. A main cause is the scarcity of training data with complex multi-instruction annotations. However, it is costly to collect such data and retrain these models. To address this challenge, we propose MSRAMIE, a training-free
agent framework built on
Multimodal Large
Language Model (MLLM). MSRAMIE takes existing editing models as plug-in components and handle multi-instruction tasks via structured
multimodal reasoning. It orchestrates iterative interactions between an MLLM-based Instructor and an image editing Actor, introducing a novel
reasoning topology that comprises the proposed Tree-of-States and Graph-of-References. During inference, complex instructions are decomposed into multiple editing steps which enable state transitions, cross-step information aggregation, and original input recall, which enables systematic exploration of the image editing space and flexible progressive output refinement. The visualizable inference
topology further provides interpretable and controllable decision pathways. Experiments show that as the instruction complexity increases, MSRAMIE can improve instruction following over 15% and increases the probability of finishing all modifications in a single run over 100%, while preserving perceptual quality and maintaining visual consistency.
Read the full abstract ▾
All authors Zhaoyuan Qiu, Ken Chen, Xiangwei Wang, Yu Xia, Sachith Seneviratne, Saman Halgamuge
Computer Vision
CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization
Liangbin Huang, Xiaohua Liao, Chaoqun Cui et al. (+4 more)
Traditional speaker diarization systems have primarily focused on constrained scenarios such as meetings and interviews, where the number of speakers is limited and acoustic conditions are relatively clean. To explore open-world speaker diarization, we extend this task to the visual media domain, encompassing complex audiovisual programs such as films and TV series. This new setting introduces several challenges, including long-form
video understanding, a large number of speakers, cross-modal asynchrony between audio and visual cues, and uncontrolled in-the-wild variability. To address these challenges, we propose Cinematic Speaker Registration & Diarization (CineSRD), a unified
multimodal framework that leverages visual, acoustic, and linguistic cues from
video,
speech, and subtitles for speaker annotation. CineSRD first performs visual anchor clustering to register initial speakers and then integrates an audio
language model for speaker turn detection, refining annotations and supplementing unregistered off-screen speakers. Furthermore, we construct and release a dedicated speaker diarization
benchmark for visual media that includes Chinese and English programs. Experimental results demonstrate that CineSRD achieves superior performance on the proposed
benchmark and competitive results on conventional
datasets, validating its robustness and generalizability in open-world visual media settings.
Read the full abstract ▾
All authors Liangbin Huang, Xiaohua Liao, Chaoqun Cui, Shijing Wang, Zhaolong Huang, Yanlong Du, Wenji Mao
Computer Vision
Behavior-Centric Extraction of Scenarios from Highway Traffic Data and their Domain-Knowledge-Guided Clustering using CVQ-VAE
Niklas Roßberg, Sinan Hasirlioglu, Mohamed Essayed Bouzouraa et al. (+2 more)
Approval of ADS depends on evaluating its behavior within representative real-world traffic scenarios. A common way to obtain such scenarios is to extract them from real-world data recordings. These can then be grouped and serve as basis on which the ADS is subsequently tested. This poses two central challenges: how scenarios are extracted and how they are grouped. Existing extraction methods rely on heterogeneous definitions, hindering scenario comparability. For the grouping of scenarios, rule-based or ML-based methods can be utilized. However, while modern ML-based approaches can handle the complexity of traffic scenarios, unlike rule-based approaches, they lack interpretability and may not align with domain-knowledge. This work contributes to a standardized scenario extraction based on the Scenario-as-Specification concept, as well as a domain-knowledge-guided scenario clustering process. Experiments on the highD
dataset demonstrate that scenarios can be extracted reliably and that domain-knowledge can be effectively integrated into the clustering process. As a result, the proposed methodology supports a more standardized process for deriving scenario categories from highway data recordings and thus enables a more efficient validation process of automated vehicles.
Read the full abstract ▾
All authors Niklas Roßberg, Sinan Hasirlioglu, Mohamed Essayed Bouzouraa, Wolfgang Utschick, Michael Botsch
Computer Vision · Cross-listed from q-bio.QM
Topology-Guided Biomechanical Profiling: A White-Box Framework for Opportunistic Screening of Spinal Instability on Routine CT
Zanting Ye, Xuanbin Wu, Guoqing Zhong et al. (+9 more)
Routine oncologic computed
tomography (CT) presents an ideal opportunity for screening spinal instability, yet prophylactic stabilization windows are frequently missed due to the complex
geometric reasoning required by the Spinal Instability Neoplastic Score (SINS). Automating SINS is fundamentally hindered by metastatic osteolysis, which induces topological ambiguity that confounds standard
segmentation and black-box AI. We propose
Topology-Guided Biomechanical Profiling (TGBP), an auditable white-box framework decoupling anatomical perception from structural
reasoning. TGBP anchors SINS assessment on two deterministic
geometric innovations: (i) canal-referenced partitioning to resolve posterolateral boundary ambiguity, and (ii) context-aware morphometric normalization via covariance-based oriented bounding boxes (OBB) to quantify vertebral collapse. Integrated with auxiliary radiomic and large
language model (LLM) modules, TGBP provides an end-to-end, interpretable SINS evaluation. Validated on a multi-center, multi-cancer cohort ($N=482$), TGBP achieved 90.2% accuracy in 3-tier stability triage. In a blinded reader study ($N=30$), TGBP significantly outperformed medical oncologists on complex structural features ($\kappa=0.857$ vs.\ $0.570$) and prevented compounding errors in Total Score estimation ($\kappa=0.625$ vs.\ $0.207$), democratizing expert-level opportunistic screening.
Read the full abstract ▾
All authors Zanting Ye, Xuanbin Wu, Guoqing Zhong, Shengyuan Liu, Jiashuai Liu, Ge Song, Zhisong Wang, Jing Hao, Xiaolong Niu, Yefeng Zheng, Yu Zhang, Lijun Lu