Computer Vision
Head Similarity: Modeling Structured Whole-Head Appearance Beyond Face Recognition
Yingfeng Wang, Yuxuan Xiao and Shengcai Liao
Many
vision applications require identity consistency beyond strict biometric recognition, especially under non-frontal views or when facial cues are missing. However, conventional face recognition models enforce intra-identity invariance, collapsing appearance variations such as hairstyle or styling changes into a single representation, limiting their use in appearance-sensitive scenarios. To address this limitation, we introduce Head Similarity, a new formulation that extends identity-centric recognition to structured whole-head similarity modeling. Our approach explicitly captures intra-identity appearance variation and enforces hierarchical similarity ordering across identity and appearance states, enabling meaningful comparison even under occlusion or rear-view conditions. We construct a large-scale
benchmark from long-form
videos with weakly-supervised appearance states, covering diverse poses, occlusions, and temporal changes. As a first step, we develop a simple yet effective framework that jointly models identity discrimination and appearance-sensitive similarity through hierarchical supervision and identity-aware
distillation. Experiments show that conventional face recognition models fail to capture appearance-dependent similarity, while our approach demonstrates the feasibility of structured whole-head similarity modeling.
Read the full abstract ▾
Computer Vision
Benchmarking Foundation Models for Renal Lesion Stratification in CT
Hartmut Häntze, Sarah de Boer, Myrthe Buser et al. (+7 more)
The rapid proliferation of open-source medical foundation models (FMs) raises a practical question: how well do their pre-trained representations transfer to clinically relevant but data-scarce classification tasks? Particularly in CT-based renal lesion classification, a push toward greater generalizability would be meaningful, as the field is constrained by inherently limited training data. We addressed this through a
benchmark of three medical FMs on this specific task. This six-class problem spans common entities like cysts and clear
cell renal
cell carcinoma, alongside rare subtypes. Using a frozen feature-probing protocol, we compared FM
embeddings against a handcrafted radiomics classifier and a 3D ResNet-50 trained from scratch. Models were trained on a composite
dataset of 2,854 lesions and evaluated on an external test set of 234 lesions from The Cancer Imaging Archive. Our results reveal two key findings. First, FM performance (AUC 0.70-0.77) matched the from-scratch ResNet (AUC 0.72) while drastically reducing hardware demand, requiring only seconds on a CPU after feature extraction. However, the conventional radiomics baseline significantly outperformed all deep learning approaches, achieving an AUC of 0.88 (all p $\leq$ 0.002). This suggests that current generalist FM
embeddings do not yet capture the fine-grained texture and shape heterogeneity driving histological subtype discrimination. Despite their potential in data-scarce settings, medical FMs did not surpass established models for renal lesion stratification, leaving radiomics as the current state-of-the-art.
Read the full abstract ▾
All authors Hartmut Häntze, Sarah de Boer, Myrthe Buser, Alessa Hering, Bram van Ginneken, Mathias Prokop, Jawed Nawabi, Sebastian Ziegelmayer, Lisa Adams, Keno Bressem
Computer Vision
LAMES: A Large-Scale and Artisanal Mining Environmental Segmentation Dataset
Matthias Kahl, Zhaiyu Chen, Sudipan Saha et al. (+3 more)
Mining operations are of utmost importance to the economy of some nations. However, such operations result in land-use change, very high energy consumption, and negative impacts on the environment, including soil erosion and deforestation. The mining process can impact an area much larger than the mining site itself. Adding to the negative externalities linked to mining is the fact that, in addition to government-sanctioned legal mining operations, illegal mining is widespread, including in various countries of Africa. The ability to monitor remote mining site activities can be useful, e.g., for the detection of illegal artisanal mining activities and their environmental impacts. An important outcome of such monitoring could include a better understanding of the interrelationship between mine facility attributes (e.g., mining types, processing methods, commodities, etc.) and their impact on the natural environment. In this work, we present a data set that contains 150 Large Scale Mining (LSM) sites and 870km^2 annotated area of Artisanal Small-scale Mining (ASM) sites. The metadata includes nine eminent LSM sections and 27 mining site attributes for each LSM site. We also discuss the data set's possible contribution to the research community, social and environmental consequences, and researchers' responsibilities from an ethics perspective.
Read the full abstract ▾
All authors Matthias Kahl, Zhaiyu Chen, Sudipan Saha, Mrinalini Kochupillai, Lukas Kondmann, Xiao Xiang Zhu
Computer Vision
OphEdit: Training-Free Text-Guided Editing of Ophthalmic Surgical Videos
Ritul Jangir, Arkya Jyoti Bagchi, Mangalton Okram et al. (+3 more)
High-fidelity
surgical video generation holds great promise for medical training and AI development, yet precise, controllable editing of such
videos remains a formidable challenge. Modifying
surgical attributes such as instrument, tissue interactions or procedural phases is particularly difficult due to the strict anatomical and temporal constraints of ophthalmic procedures. We introduce OphEdit, the first training-free framework for text-guided editing of ophthalmic
surgical videos. Our approach uses a deterministic second-order ODE inversion pipeline to capture
attention Value tensors from the source
video and selectively injects them into the conditional branch of classifier-free guidance during denoising, preserving the intricate anatomical geometry of the eye while mapping text-driven semantic modifications onto the
video. Clinical evaluation by an ophthalmologist shows that OphEdit handles complex transformations, such as instrument swaps and procedural variations, with superior structural fidelity and temporal consistency compared to adapted general-domain
video editors, offering a scalable route to diverse
surgical training data. The code and
prompts can be accessed at https://github.com/ophedit/OphEdit
Read the full abstract ▾
All authors Ritul Jangir, Arkya Jyoti Bagchi, Mangalton Okram, Saurabh Seetaram Korgaonkar, Aiman Farooq, Deepak Mishra
Computer Vision · Cross-listed from cs.LG
Stochastic Transition-Map Distillation for Fast Probabilistic Inference
George Rapakoulias, Peter Garud, Lingjiong Zhu et al. (+1 more)
Diffusion models achieve strong generation quality, diversity, and distribution coverage, but their performance often comes with expensive inference. In this work, we propose
Stochastic Transition-Map
Distillation (STMD), a teacher-free framework for accelerating
diffusion model inference while preserving probabilistic sample generation. In contrast to score-based
diffusion models, whose denoising parametrization models the mean of the posterior distribution, STMD distills the full transition map associated with the sampling
stochastic differential equation (SDE). We parameterize these SDE transitions with a conditional Mean Flow model, yielding a one- or few-step
stochastic sampler that retains the transition structure of the underlying
diffusion process. This perspective is especially useful for downstream tasks that require
stochastic inference, such as
diffusion posterior sampling, inverse problems, and energy-based fine-tuning. Compared to recent
distillation methods, STMD requires no pretrained teacher, bi-level optimization, or trajectory simulation and caching, enabling efficient and scalable training. We derive convergence bounds for our method in the Wasserstein distance, providing a strong theoretical foundation for our approach, and validate STMD on various image generation examples on the MNIST, CIFAR-10, and CelebA
datasets.
Read the full abstract ▾
All authors George Rapakoulias, Peter Garud, Lingjiong Zhu, Panagiotis Tsiotras
Computer Vision
Towards Billion-scale Multi-modal Biometric Search
Arka Koner, Chetan S. Naik, Lokesh Kurre et al. (+5 more)
Searching a multi-biometric database of a billion records for a country-level identity system requires pushing the limits of all aspects of a biometric system, including acquisition, preprocessing, feature extraction, accuracy, matching speed, presentation attack detection, and handling of special cases (e.g., missing finger digits). This is the first paper that gives insights into such a large-scale
multimodal biometric search system, called Bharat ABIS, based on open-source architectures. The end-to-end pipeline of Bharat ABIS processes fingerprint, face and iris modalities through modality-specific stages of preprocessing (
segmentation), quality assessment, presentation attack detection, and learning an
embedding (feature extraction), producing a concatenated template of 13.5KB per person. We present a detailed analysis of the modalities and how they are integrated to create an efficient and effective solution for 1:N search (de-duplication). Evaluations on a demographically stratified gallery of 220 million identities, randomly sampled from 1.55 billion records in India's Aadhaar database, yield an FNIR of 0.3% at an FPIR of 0.5%, for adult probes (over 18 years). We also compare the performance of Bharat ABIS against three state-of-the-art COTS systems on a 20M gallery. Our system achieves a throughput of 100 searches per second on a gallery of 40M on a single server (8xNvidia H100 GPUs, 2TB RAM).
Read the full abstract ▾
All authors Arka Koner, Chetan S. Naik, Lokesh Kurre, Vivek Raghavan, Barada P. Sabut, Tanusree Deb Barma, Anoop M. Namboodiri, Anil K. Jain
Computer Vision
Aquatic Neuromorphic Optical Flow
Pei Zhang, Yunkai Liang and Kaiqiang Wang
Underwater environments impose severe constraints on conventional imaging systems and demand solutions that balance high-quality sensing with strict resource efficiency. While emerging event cameras offer a promising alternative, their potential in aquatic scenarios remains largely unexplored. Through the lens of neuromorphic
vision, this work pioneers the investigation of motion fields that serve as key media for agile
underwater perception. Built upon spiking neural networks, we introduce a self-supervised framework to estimate per-pixel optical flow from asynchronous event streams, elegantly bypassing the long-standing bottleneck of
underwater data scarcity. Extensive evaluations demonstrate that our method achieves competitive visual and quantitative results against leading techniques while operating with superior computational efficiency. By bridging neuromorphic sensing and aquatic intelligence, this work opens new frontiers for lightweight, real-time, and low-cost perception on resource-constrained
underwater edge platforms.
Read the full abstract ▾
Computer Vision
Breaking Spatial Uniformity: Prior-Guided Mamba with Radial Serialization for Lens Flare Removal
Zijia Fu, Yuanfei Huang, Lizhi Wang et al. (+1 more)
Lens
flares, caused by complex optical aberrations, severely degrade image quality especially in nighttime photography. Although recent restoration methods have made remarkable progress, most still rely on spatially uniform processing. They are failing to handle the region-dependent restoration demands of
flare scenes, where saturated light sources should be preserved,
flare artifacts removed, and background details recovered. To address this challenge, we propose DeflareMambav2, a prior-guided Mamba framework for lens
flare removal. Specifically, we introduce a
Flare Prior Network (FPN) to estimate
flare priors and guide adaptive restoration. Besides, a novel radial serialization strategy breaks spatially homogeneous processing by performing
flare-aware targeted sampling, and better supports long-range modeling in State Space Models (SSMs). Based on these priors, the backbone adopts a dual-level adaptive scheme. It explicitly preserves light-source regions to avoid over-processing, and applies curriculum-based restoration to the remaining contaminated areas while calibrating restoration intensity at the pixel level. Extensive experiments demonstrate that DeflareMambav2 achieves state-of-the-art performance with reduced parameter burden. Code is available at https://github.com/BNU-ERC-ITEA/DeflareMambav2.
Read the full abstract ▾
All authors Zijia Fu, Yuanfei Huang, Lizhi Wang, Hua Huang
Computer Vision
Operating Within the Operational Design Domain: Zero-Shot Perception with Vision-Language Models
Berkehan Ünal, Hauke Dierend, Dren Fazlija et al. (+1 more)
Over the last few years, research on autonomous systems has matured to such a degree that the field is increasingly well-positioned to translate research into practical, stakeholder-driven use cases across well-defined domains. However, for a wide-scale practical adoption of autonomous systems, adherence to safety regulations is crucial. Many regulations are influenced by the Operational Design Domain (ODD), which defines the specific conditions in which an autonomous
agent can function. This is especially relevant for Automated Driving Systems (ADS), as a dependable perception of ODD elements is essential for safe implementation and
auditing.
Vision-
language models (VLMs) integrate visual recognition and
language reasoning, functioning without task-specific training data, which makes them suitable for adaptable ODD perception. To assess whether VLMs can function as zero-shot "ODD sensors" that adapt to evolving definitions, we contribute (i) an empirical study of zero-shot ODD classification and detection using four VLMs on a custom
dataset and Mapillary Vistas, along with failure analyses; (ii) an ablation of zero-shot optimization strategies with a cost-performance overview; and (iii) a suite of reusable prompting templates with guidance for adaptation. Our findings indicate that definition-anchored chain-of-thought prompting with persona decomposition performs best, while other methods may result in reduced recall. Overall, our results pave the way for transparent and effective ODD-based perception in safety-critical applications.
Read the full abstract ▾
All authors Berkehan Ünal, Hauke Dierend, Dren Fazlija, Christopher Plachetka
Computer Vision
Jaeyoung Choi, Hyeondong Kim, Yujin Kim et al. (+1 more)
Forecasting future 3D hand pose sequences from
egocentric video is essential for understanding human intention and enabling
embodied applications such as AR/VR assistance and human-
robot interaction. However, this task remains a highly challenging problem because
egocentric hand motion is driven by complex human intent, exhibits highly dexterous articulations, and is observed under drastic viewpoint shifts induced by ego-motion. In this work, we introduce EggHand, a foundation-model-based framework for
egocentric hand pose
forecasting that unifies
multimodal semantic
reasoning with dynamic motion modeling. Our approach couples an action decoder from a
Vision-
Language-Action (VLA) model, which captures the structured temporal dynamics of hand motion, with an
egocentric video-text encoder that provides viewpoint-aware contextual information learned from large-scale first-person
video. Together, these components overcome the brittleness of generic visual encoders under ego-motion and enable joint
reasoning over motion, context, and high-level intent-without relying on body pose or external tracking. Experiments on the EgoExo4D
dataset show that EggHand sets a new state of the art in
forecasting accuracy, remains robust under severe ego-motion, and further enables controllable prediction via
language-based task
prompts. Project page: https://jyoun9.github.io/EggHand
Read the full abstract ▾
All authors Jaeyoung Choi, Hyeondong Kim, Yujin Kim, Daehee Park
Computer Vision
LithoBench: Benchmarking Large Multimodal Models for Remote-Sensing Lithology Interpretation
Jun Wang, Fengpeng Li, Hang Dong et al. (+2 more)
Remote sensing lithology interpretation is fundamental to geological surveys, mineral exploration, and regional geological mapping. Unlike general land-cover recognition, lithology interpretation is a knowledge-intensive task that requires experts to infer rock types from various features, e.g., subtle visual, spectral, textural, geomorphological, and contextual cues, making reliable automated interpretation highly challenging. Geological knowledge-guided large
multimodal models offer new opportunities, yet their evaluation remains constrained by the lack of
benchmarks that capture lithological annotations, multi-level geological semantics, and expert-informed assessment. Here, we propose LithoBench, a multi-level
benchmark for evaluating geological semantic understanding in remote sensing lithology interpretation. LithoBench contains 10,000 expert-annotated interpretation instances across 12 representative lithological categories, including 4,000 multiple-choice and 6,000 open-ended tasks organized into five cognitive levels: Identification and Description, Comparative Analysis, Mechanism Explanation, Practical Application, and Comprehensive
Reasoning. We further develop an expert-in-the-loop, knowledge-grounded semi-automated construction pipeline, coupling multi sub-processes, e.g., structured geological image descriptions, to enhance geological validity and evaluation reliability. Experiments with multiple large
vision-
language models eveal substantial limitations in geological semantic understanding, particularly on higher-order explanation, application, and
reasoning tasks.
Read the full abstract ▾
All authors Jun Wang, Fengpeng Li, Hang Dong, Tianjin Huang, Wei Han
Computer Vision
FS-I2P:A Hierarchical Focus-Sweep Registration Network with Dynamically Allocated Depth
Zhixin Cheng, Yujia Chen, Xujing Tao et al. (+4 more)
Image-to-point cloud registration is often challenged by viewpoint changes, cross-modal discrepancies, and repetitive textures, which induce scale ambiguity and consequently lead to erroneous correspondences. Recent detection-free methods alleviate this issue by leveraging multi-scale features and
transformer-based interactions. However, they still suffer from
attention drift across layers and intra-scale inconsistencies, hindering precise registration. Inspired by human behavior, we propose a “Focus–Sweep” paradigm and develop a Hierarchical Focus–Sweep Interaction Module within an SSM-based framework to enhance multi-level cross-modal feature association. In addition, we introduce a Dynamic Layer Allocation Strategy that adaptively determines the iteration depth to better exploit
geometric constraints and improve matching robustness. Extensive experiments and ablations on two
benchmarks, RGB-D Scenes V2 and 7-Scenes, demonstrate that our approach achieves state-of-the-art performance.
Read the full abstract ▾
All authors Zhixin Cheng, Yujia Chen, Xujing Tao, Bohao Liao, Xiaotian Yin, Baoqun Yin, Tianzhu Zhang
Computer Vision
SAM 3D Animal: Promptable Animal 3D Reconstruction from Images in the Wild
Xuyi Hu, Jin Lyu, Jiuming Liu et al. (+4 more)
3D animal reconstruction in the wild remains challenging due to large species variation, frequent occlusions, and the prevalence of multi-animal scenes, while existing methods predominantly focus on single-animal settings. We present SAM 3D Animal, the first promptable framework for multi-animal 3D reconstruction from a single image. Built on the SMAL+ parametric animal model, our method jointly reconstructs multiple instances and supports flexible
prompts in the form of keypoints and masks which enable more reliable disambiguation in crowded and occluded scenes. To train such a model, we further introduce Herd3D, a multi-animal 3D
dataset containing over 5K images, designed to increase diversity in species, interactions, and occlusion patterns. Experiments on the Animal3D, APTv2, and Animal Kingdom
datasets show that our framework achieves state-of-the-art results over both existing model-based and model-free methods, demonstrating a scalable and effective solution for
prompt-driven animal 3D reconstruction in the wild.
Read the full abstract ▾
All authors Xuyi Hu, Jin Lyu, Jiuming Liu, Yebin Liu, Silvia Zuffi, Liang An, Stefan Goetz
Computer Vision
TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos
Hengyi Feng, Hao Liang, Mingrui Chen et al. (+6 more)
Real-world audio-visual understanding requires chaining evidence that is sparse, temporally dispersed, and split across the visual and auditory streams, whereas existing
benchmarks largely fail to evaluate this capability. They restrict
videos to short clips, isolate modalities, or reduce questions to one-hop perception. We introduce TraceAV-Bench, the first
benchmark to jointly evaluate multi-hop
reasoning over long audio-visual trajectories and
multimodal hallucination robustness. TraceAV-Bench comprises 2,200 rigorously validated multiple-choice questions over 578 long
videos, totaling 339.5 hours, spanning 4 evaluation dimensions and 15 sub-tasks. Each question is grounded in an explicit multi-hop evidence trajectory that averages 3.68 hops across a 15.1-minute temporal span. The
dataset is built by a three-step pipeline followed by a strict quality assurance process. Evaluation of multiple representative OmniLLMs on TraceAV-Bench reveals that the
benchmark poses a persistent challenge across all models, with the strongest closed-source model (Gemini 3.1 Pro) reaching only 68.29% on general tasks, and the best open-source model (Ming-Flash-Omni-2.0) reaching 51.70%, leaving substantial headroom. Further analysis shows that current models struggle to gather and combine evidence across long audio-visual
videos, with performance varying across modalities and
reasoning tasks. We hope TraceAV-Bench will facilitate progress toward reliable long-form audio-visual
reasoning in OmniLLMs.
Read the full abstract ▾
All authors Hengyi Feng, Hao Liang, Mingrui Chen, Bohan Zeng, Meiyi Qiang, Zhengyang Zhao, Zimo Meng, Zeang Sheng, Wentao Zhang
Computer Vision
Beyond Defenses: Manifold-Aligned Regularization for Intrinsic 3D Point Cloud Robustness
Pedro Alonso, Chongshou Li and Tianrui Li
Despite extensive progress in point cloud robustness, existing methods primarily rely on augmentation strategies or defense mechanisms while overlooking the
geometric nature of
adversarial fragility. We hypothesize that
adversarial vulnerability in 3D networks arises from a
manifold misalignment between the latent geometry learned by the model and the intrinsic geometry of the underlying surface. Small, geometry-preserving perturbations along the input
manifold often induce disproportionate distortions in feature space, potentially leading to misclassifications. We formalize this phenomenon by developing a
geometric interpretation of 3D robustness that links classical
adversarial theory to the intrinsic structure of point clouds. Motivated by this analysis, we introduce
Manifold-Aligned Point Recognition (MAPR), a framework that regularizes the latent geometry by aligning predictions across intrinsic perturbations. MAPR augments each point cloud with intrinsic features capturing local
curvature and
diffusion structure, and applies a consistency loss that preserves invariance to intrinsic, geometry-preserving perturbations. Without relying on
adversarial training or additional data, MAPR consistently improves robustness under multiple
adversarial attacks across several
datasets, achieving average robustness gains of +20.02 and +8.83 percentage points over vanilla models on ModelNet40 and ScanObjectNN, respectively.
Read the full abstract ▾
Computer Vision
Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding
Ke Ma, Jiaqi Tang, Bin Guo et al. (+8 more)
Proactive streaming
video understanding requires
Video-LLMs to decide when to respond as a
video unfolds, a task where existing methods often fall short due to their implicit, query-agnostic modeling of visual evidence. We introduce Response-G1, a novel framework that establishes explicit, structured
alignment between the accumulated
video evidence and the query's expected response conditions via scene graphs. The framework operates in three fine-tuning-free stages: (1) online query-guided scene graph generation from streaming clips; (2)
memory-based
retrieval of the most semantically relevant historical scene graphs; and (3)
retrieval-augmented trigger prompting for per-frame "silence/response" decisions. By grounding both evidence and conditions in a shared graph representation, Response-G1 achieves more interpretable and accurate response timing decisions. Experimental results on established
benchmarks demonstrate the superiority of our method in both proactive and reactive tasks, validating the advantage of explicit scene graph modeling and
retrieval in streaming
video understanding.
Read the full abstract ▾
All authors Ke Ma, Jiaqi Tang, Bin Guo, Xueting Han, Ruonan Xu, Qingfeng He, Ziheng Wang, Xu Wang, Qifeng Chen, Zhiwen Yu, Yunhao Liu
Computer Vision
PolarVLM: Bridging the Semantic-Physical Gap in Vision-Language Models
Yuliang Li, Chu Zhou, Heng Guo et al. (+3 more)
Mainstream
vision-
language models (VLMs) fundamentally struggle with severe optical ambiguities, such as reflections and transparent objects, due to the inherent limitations of standard RGB inputs. While polarization imaging captures polarimetric physical parameters that resolve these ambiguities, existing methods are constrained by fixed-format outputs and remain isolated from open-ended
reasoning. To bridge this semantic-physical gap, we introduce PolarVLM, the first
multimodal framework integrating polarimetric physical parameters into VLMs. By employing a dual-stream architecture and a progressive two-stage training strategy, PolarVLM effectively prevents physical misinterpretations while preserving general visual abilities. Complementing our architecture, we construct PolarVQA, the first
benchmark for polarization-aware VQA, featuring 75K physics-grounded instruction-tuning pairs targeting reflective and transparent scenes. Experiments show that PolarVLM surpasses the RGB baseline by 25.4% overall across five evaluation tasks, with remarkable gains of 26.6% in reflection recognition and 34.0% in glass counting, successfully unlocking physics-aware semantic understanding.
Read the full abstract ▾
All authors Yuliang Li, Chu Zhou, Heng Guo, Boxin Shi, Imari Sato, Zhanyu Ma
Computer Vision
Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMs
Peitao Han, Fei Cheng, Lis K. Pereira et al. (+2 more)
The Arrow-of-Time (AoT) task, determining whether a
video plays forward or backward by recognizing temporal irreversibility, is one humans solve with near-perfect accuracy, yet frontier
Video Large
Language Models (
Video-LLMs) perform only modestly above chance. This gap raises a key question: do visual backbones fail to encode temporal information, or does information bottleneck lie elsewhere in the
Video-LLM architecture? We address this question by isolating the
vision encoder from the
Video-LLM and tracing temporal information across the encoder, projector, and LLM. We find that
video-centric encoders with explicit temporal modeling encode strong temporal signals, whereas frame-centric encoders do not. However, when
video-centric representations are passed through a standard
Video-LLM architecture, performance often collapses, revealing a bottleneck of temporal information flow. We identify projector design as a key factor: Q-Former disrupts temporal information, while a time-preserved MLP projection substantially improves the LLM's access to such information. Our layer-wise analysis further shows temporal representation dynamics across encoder layers. Guided by these findings, we build a
Video-LLM with temporal-aware
video-centric encoder, time-preserved projector, and AoT supervision, surpassing human performance on AoT$_{PPB}$ with 98.1% accuracy, and improving broader temporal
reasoning tasks by up to 6.0 points on VITATECS-Direction and 1.3 points on TVBench. Our results show that temporal
reasoning in
Video-LLMs requires both effective temporal encoding and reliable transfer of this information to the LLM.
Read the full abstract ▾
All authors Peitao Han, Fei Cheng, Lis K. Pereira, Qianying Liu, Shigeru Kitazawa
Computer Vision
One Adapter, Every Resolution: Gated Low-Rank Adaptation for Remote Sensing VLMs
Song Zhang, Yanlong Chen, Yining Chen et al. (+2 more)
Remote sensing imagery spans ground sampling distances (GSDs) from centimeters to tens of meters, so both the visual evidence for a geographic concept and the questions it can support change with physical scale. Existing remote sensing
vision-
language models (RS-VLMs) either ignore GSD or encode it as a text
token, applying one set of scale-agnostic parameters across this spectrum. We show that GSD is decodable from frozen visual features, but exploiting it requires scale-dependent adaptation. We therefore introduce ScaleEarth, which conditions low-rank adaptation on the continuous scalar s = log10(GSD). Its core module, CS-HLoRA, gates the rank dimensions of a single shared LoRA with sigmoid functions of s whose thresholds are initialized at object, structure, and semantic scales; the gates route gradients by scale during training and select the active adaptation subspace at inference, while keeping the backbone frozen. When metadata is unavailable, a heteroscedastic head, SSE-U, estimates s with calibrated uncertainty and falls back to a default scale when uncertain. To align supervision with this mechanism, we build GeoScale-VQA, a
dataset of 1.5M QA pairs in which every sample carries a resolved GSD and newly generated questions are conditioned on the same s. With an 8B backbone, ScaleEarth achieves an average score of 59.4 on XLRS-Bench, improving by 5.2 points over the strongest RS specialist. On OmniEarth-Bench, it achieves an average score of 40.71, improving by 7.45 points over the strongest open-source baseline. The largest gains occur on scale-sensitive subtasks. A 4-bit variant can be deployed on a single 40 GB GPU and retains an average score of 57.4 on XLRS-Bench, substantially reducing adaptation and deployment costs.
Read the full abstract ▾
All authors Song Zhang, Yanlong Chen, Yining Chen, Xiaowei Zhang, Yawei Li
Computer Vision
Multimodal Stepwise Clinically-Guided Attention Learning for Pathological Complete Response Prediction in Breast Cancer
Alice Natalina Caragliano, Valerio Guarrasi, Michela Gravina et al. (+2 more)
Pathological complete response (pCR) is a key prognostic factor in breast cancer patients undergoing neoadjuvant therapy, strongly associated with long-term survival and treatment personalization. However, accurate pre-treatment pCR prediction remains challenging due to severe class imbalance and limited generalizability across diverse clinical settings. In this work, we propose a
multimodal stepwise clinically-guided
attention learning framework for pCR prediction from breast
magnetic resonance imaging (MRI), designed to address these limitations through medically grounded spatial guidance and
multimodal integration. The approach follows a stepwise training strategy inspired by physician
reasoning: the model first learns global discriminative imaging patterns, then
attention mechanisms are introduced to constrain the network toward tumor regions, and finally clinical variables are integrated to refine decision-making. This guidance strategy encourages prioritization of task-relevant features, improving identification of responders despite their limited representation in the
dataset. Moreover, grounding
attention in anatomically consistent tumor regions reduces reliance on
dataset-specific patterns, thereby enhancing cross-institutional generalization. The framework is evaluated through external validation across heterogeneous MRI cohorts. Compared to non-guided single-stage baselines, the proposed approach improves sensitivity while maintaining competitive specificity, and produces anatomically coherent
attention maps that support interpretation of the model's predictions. These findings highlight the potential of clinically-guided
multimodal attention learning for robust and generalizable pCR prediction in breast cancer.
Read the full abstract ▾
All authors Alice Natalina Caragliano, Valerio Guarrasi, Michela Gravina, Carlo Sansone, Paolo Soda
Computer Vision
Dynamic Mode Decomposition along Depth in Vision Transformers
Nishant Suresh Aswani and Saif Eddin Jabari
Recent work has shown that contiguous
vision transformer (ViT) blocks (a) can be replaced by a linear map and (b) organize into recurrent phases of computation. We ask whether these observations coincide: does ViT depth implement approximately autonomous linear dynamics, admitting a single operator $K$ applied recurrently across a contiguous span? We test this using Dynamic Mode Decomposition (DMD), which fits $K$ from selected, consecutive hidden-state pairs and predicts $p$ steps ahead via $K^p$. On four pretrained DINO ViTs, we study the regularization, rank, and calibration budget required for stable fitting. For short spans ($p \leq 4$), $K^p$ tracks an unconstrained endpoint map to within $0.02$ cosine similarity on DINOv3-H/16+, while also recovering intermediate activations at each skipped block. At early cut starts, the fitted operators compress to rank $\ll d$ with minimal calibration data, and across
tokens, cls is most amenable to linearization; both properties decay monotonically with depth. Yet this local fidelity does not transfer downstream. At the final hidden state, after propagating through the remaining blocks, an identity baseline becomes competitive.
Read the full abstract ▾
Computer Vision
VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network
Zepeng Yang, Junxuan Bai, Hao Li et al. (+4 more)
The rapid advances in deep learning have significantly enhanced the accuracy of
multimodal 3D human pose estimation (HPE). However, the state-of-the-art (SOTA) HPE pipelines still rely on
Transformers, whose quadratic complexity makes real-time processing for long sequences impractical. Mamba addresses this issue through selective state-space modeling, enabling efficient sequence processing without sacrificing representational power. Nevertheless, it struggles to capture complex spatial dependencies in
multimodal settings. To bridge this gap, we propose VIMCAN, a hybrid architecture that combines the efficient sequence modeling of Mamba with the spatial
reasoning of Cross-
Attention, and performs robust visual-inertial fusion and human pose estimation between RGB keypoints and wearable IMU data. By leveraging Mamba's dynamic parameterization for temporal modeling and
Attention for spatial dependency extraction, VIMCAN achieves superior accuracy, with mean per-joint position errors (MPJPE) of 17.2 mm on TotalCapture and 45.3 mm on 3DPW. VIMCAN outperforms prior
Transformer-based and other SOTA approaches while supporting real-time inference at over 60 frames per second on consumer-grade hardware. The source code is available at \href{https://github.com/Eddieyzp/VIMCAN}{this GitHub repository}.
Read the full abstract ▾
All authors Zepeng Yang, Junxuan Bai, Hao Li, Ju Dai, Junjun Pan, Yongfeng Yin, Bin Li
Computer Vision
Mind the Gap: Geometrically Accurate Generative Reconstruction from Disjoint Views
Grzegorz Wilczynski, Mikołaj Zielinski, Bartosz Świrta et al. (+2 more)
3D
vision systems are fundamentally constrained by their reliance on visual overlap: reconstruction methods require it for
geometric alignment, while generative models use it to enforce multi-view consistency. This limitation is particularly acute in real-world scenarios such as distributed swarm robotics or crowd-sourced data collection, where capturing overlapping perspectives, both in terms of spatial and appearance overlap, is often impossible. We introduce Generative Reconstruction from Disjoint Views as a new paradigm, establish a comprehensive
dataset, and propose specialized evaluation metrics for zero-overlap scenarios. Our benchmarking demonstrates that existing state-of-the-art methods fail catastrophically on this task, producing disconnected geometries or semantically incoherent reconstructions. To address these limitations, we propose GLADOS, a general, modular framework that operates through three stages: (1) Generative Bridging, where foundation models synthesize intermediate perspectives to connect disjoint inputs; (2) Robust Coarse 3D Reconstruction, that establish coarse
geometric scaffold via global
alignment which absorbs local contradictions from generative process; and (3) Iterative Context Expansion and Consistency Optimization to fill missing regions and unify the reconstruction. As an architectureagnostic framework, GLADOS enables seamless integration of future advances in generation, reconstruction, and inpainting. The source code is available at: https://github.com/gwilczynski95/GLADOS.
Read the full abstract ▾
All authors Grzegorz Wilczynski, Mikołaj Zielinski, Bartosz Świrta, Dominik Belter, Przemysław Spurek
Computer Vision
Probabilistic Object Detection with Conformal Prediction
Christopher Ries, Moussa Kassem Sbeyti, Nicolas Bianco et al. (+1 more)
Conformal Prediction (CP) is a distribution-free method for constructing prediction sets with marginal finite-sample coverage guarantees, making it a suitable framework for reliable uncertainty quantification in safety-critical object detection. However, object detection introduces structured multi-output predictions, complicating the application of classical CP theory developed for single outputs. In addition, standard, unscaled CP produces fixed-width prediction intervals across inputs, leading to unnecessary width for low-uncertainty predictions. While scaled CP addresses this by adapting the interval width to an input-dependent uncertainty estimate, prior work has neither systematically compared unscaled and scaled CP for multi-class object detection, nor integrated CP with a complementary uncertainty quantification method in this setting. We fill this gap by: (i) applying CP coordinate-wise to bounding box corners with a Bonferroni correction for box-level guarantees; (ii) scaling the resulting intervals using per-prediction aleatoric uncertainty estimates derived from a probabilistic object detector trained with loss attenuation, evaluated in uncalibrated and two calibrated variants; (iii) extending to a two-step pipeline that constructs prediction sets for the class using RAPS and conditions the conformalized bounding boxes on the predicted class set. Across three autonomous driving
datasets (KITTI, BDD, CODA), including a cross-domain setting under distribution shift, scaled CP consistently improves interval sharpness over unscaled CP, achieving up to 19% higher IoU and 39% lower interval scores, without sacrificing coverage. Class-wise calibration further improves coverage for both variants with a negligible effect on sharpness. Together, these improvements yield more actionable uncertainty estimates for real-time, real-world object detection.
Read the full abstract ▾
All authors Christopher Ries, Moussa Kassem Sbeyti, Nicolas Bianco, Nadja Klein
Computer Vision
Implicit Preference Alignment for Human Image Animation
Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng et al. (+5 more)
Human image animation has witnessed significant advancements, yet generating high-fidelity hand motions remains a persistent challenge due to their high degrees of freedom and motion complexity. While reinforcement learning from human feedback, particularly direct preference optimization, offers a potential solution, it necessitates the construction of strict preference pairs. However, curating such pairs for dynamic hand regions is prohibitively expensive and often impractical due to frame-wise inconsistencies. In this paper, we propose Implicit Preference
Alignment (IPA), a data-efficient post-training framework that eliminates the need for paired preference data. Theoretically grounded in implicit reward maximization, IPA aligns the model by maximizing the likelihood of self-generated high-quality samples while penalizing deviations from the pretrained prior. Furthermore, we introduce a Hand-Aware Local Optimization mechanism to explicitly steer the
alignment process toward hand regions. Experiments demonstrate that our method achieves effective preference optimization to enhance hand generation quality, while significantly lowering the barrier for constructing preference data. Codes are released at https://github.com/mdswyz/IPA
Read the full abstract ▾
All authors Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng, Bing Ma, Kai Yu, Tianxiang Zheng, Qinglin Lu, Zhen Cui
Computer Vision · Cross-listed from cs.RO
Is the Future Compatible? Diagnosing Dynamic Consistency in World Action Models
Bo-Kai Ruan, Teng-Fang Hsiao, Ling Lo et al. (+1 more)
World Action Models (WAMs) enable decision-making through imagined rollouts by predicting future observations and actions. However, the reliability of these imagined futures remains under-examined: is a generated future merely visually plausible, or is it dynamically compatible with the action sequence it claims to model? In this work, we identify action-state consistency, the
alignment between predicted actions and induced state transitions, as a missing reliability axis for WAMs. Through a systematic study across representative joint-prediction and inverse-dynamics models, we find that action-state consistency systematically separates successful and failed rollouts across many tasks and follows similar success-failure trends as learned value estimates. These results suggest that consistency captures decision-relevant structure beyond visual realism. We further identify background collapse as an important boundary condition, where low-dynamics failed trajectories can become deceptively consistent because static futures are easier to predict. Building on these findings, we introduce a value-free consensus strategy for test-time selection, which ranks candidate rollouts by agreement among predicted futures. This strategy improves success rates on RoboCasa and RoboTwin 2.0 without additional training or reward modeling. Taken together, our findings establish action-state consistency as both a diagnostic tool for evaluating WAM reliability and a practical signal for value-free planning.
Read the full abstract ▾
All authors Bo-Kai Ruan, Teng-Fang Hsiao, Ling Lo, Hong-Han Shuai
Computer Vision
Hierarchical Dual-Subspace Decoupling for Continual Learning in Vision-Language Models
Mengxin Qin, Xiang Zhang, Kun Wei et al. (+2 more)
Class-incremental learning aims to continuously acquire new knowledge while preserving previously learned information, thereby mitigating catastrophic forgetting. Existing methods primarily restrict parameter updates but often overlook their structural properties in high-dimensional spaces. From a subspace perspective, updates induced by different tasks tend to lie in multiple overlapping low-rank subspaces, leading to cross-task subspace interference and severe forgetting. To address this issue, we propose HDSD, a Hierarchical Dual-Subspace Decoupling framework for continual learning in
vision-
language models. Specifically, we introduce a lightweight Feature Modulation Module (FMM) that explicitly decomposes the parameter space into general and task-specific subspaces. Building on this design, we develop two complementary components. First, a General Fusion Module (GFM) evaluates relative parameter changes across tasks and uses an adaptive threshold to capture stable and transferable knowledge. Second, a Hierarchical Learning Module (HLM) performs structured parameter decomposition via Singular Value Decomposition (SVD) and uses a scaling mechanism to constrain updates within distinct subspace scales. Together, these designs reduce subspace interference and parameter drift. Extensive experiments on conventional
benchmarks show that HDSD achieves state-of-the-art results.
Read the full abstract ▾
All authors Mengxin Qin, Xiang Zhang, Kun Wei, Xu Yang, Cheng Deng
Computer Vision
InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search
Bohan Hou, Jiuning Gu, Jiayan Guo et al. (+5 more)
Existing
benchmarks for
multimodal agentic search evaluate
multimodal search and visual browsing, but visual evidence is either confined to the input or treated as an answer endpoint rather than part of an interleaved search trajectory. We introduce InterLV-Search, a
benchmark for Interleaved
Language-
Vision Agentic Search, in which textual and visual evidence is repeatedly used to condition later search. It contains 2,061 examples across three levels: active visual evidence seeking, controlled offline interleaved
multimodal search, and open-web interleaved
multimodal search. Beyond existing
benchmarks, it also includes
multimodal multi-branch samples that involve comparison between multiple entities during the evidence search. We construct Level 1 and Level 2 with automated pipelines and Level 3 with a machine-led, human-supervised open-web pipeline. We further provide InterLV-
Agent for standardized tool use, trajectory logging, and evaluation. Experiments on proprietary and open-source
multimodal agents show that current systems remain far from solving interleaved
multimodal search, with the best model below 50% overall accuracy, highlighting challenges in visual evidence seeking, search control, and
multimodal evidence integration. We release the
benchmark data and evaluation code at https://github.com/hbhalpha/InterLV-Search-Bench
Read the full abstract ▾
All authors Bohan Hou, Jiuning Gu, Jiayan Guo, Ronghao Dang, Sicong Leng, Xin Li, Xuemeng Song, Jianfei Yang
Computer Vision
Jingyuan Zhu, Biaolong Chen, Le Zhang et al. (+3 more)
Efficiently aligning large-scale
video diffusion models with human intent requires a scalable and trajectory-aware pathway that bridges the inherent discrepancy between training noise distributions and practical inference trajectories. While existing paradigms such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) attempt to address this, they are often hindered by either reliance on bias-prone, complex reward models or suboptimal timestep sampling. In this paper, we propose
Diffusion-APO (Aligned Preference Optimization), a trajectory-aware
algorithm that resolves this misalignment by synchronizing training noise with inference-time denoising paths to maximize gradient signal efficacy. To translate this algorithmic innovation into a practical solution, we introduce a unified and modular RLHF framework that integrates online ranking, half-online anchoring, offline refinement, and
distillation-aware drift correction. This framework enables flexible, multi-stage preference
alignment across diverse data and computational constraints without relying on scalar-reward-based policy gradients. Through extensive experiments, we demonstrate that
Diffusion-APO consistently outperforms standard baselines in visual quality and instruction following, while effectively preserving generative fidelity during model acceleration, providing a robust, end-to-end pathway for scalable
video diffusion alignment.
Read the full abstract ▾
All authors Jingyuan Zhu, Biaolong Chen, Le Zhang, Aixi Zhang, Hao Jiang, Pipei Huang
Computer Vision
Cloud-top infrared observations reveal the four-dimensional precipitation structure
Tianchi Xu, Ziqiang Ma, Andrea Marinoni et al. (+11 more)
Accurate four-dimensional (4D) precipitation information is essential for understanding the Earth's energy and water cycles, yet remains observationally unresolved at global scales. Conventional theory holds that geostationary infrared observations primarily sense cloud-top properties, with limited sensitivity to sub-cloud precipitation. Here we show that cloud-top infrared measurements nevertheless encode sufficient information to recover the four-dimensional structure of precipitation, revealing a previously unexploited observability of sub-cloud processes. We introduce a physically constrained deep learning framework, 4DPrecipNet, in which a moisture-first constraint requires the latent representation to recover precipitable water vapour, anchoring the model in thermodynamic consistency. By integrating multi-channel infrared radiances with these constraints and radar-derived precipitation profiles, we reconstruct the vertical and temporal
evolution of precipitation systems from geostationary orbit. The framework captures deep convective structures and their
evolution, with robust performance across large samples and independent radar comparisons. These results demonstrate that sub-cloud precipitation is physically encoded in cloud-top infrared observations, establishing a new pathway for continuous global monitoring of precipitation structure.
Read the full abstract ▾
All authors Tianchi Xu, Ziqiang Ma, Andrea Marinoni, Yuanpeng He, Xiaoqing Li, Chuanfeng Zhao, Kang He, Jintao Xu, Bohan Zhou, Wenbo Zhao, Haoshuang Chen, Tun Wang, Dongdong Wang, Yang Hong