Peer Revenue An exchange of ideas

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Computer Science Computer Vision

cs.CV Watch Computer Vision

Saturday 4 July

Computer Vision · Cross-listed from cs.RO

From Region Arrival to Instance-Level Grounding in Vision-and-Language Navigation

Xiangyu Shi, Ruoxi Yang, Wei Tao et al. (+3 more)

Vision-and-Language Navigation (VLN) agents may satisfy conventional success criteria while still failing to establish reliable object-level grounding, because current evaluation protocols mainly reward stopping within a 3-meter radius and largely ignore the agent's final orientation and target visibility. We formalize this limitation as the Last-3-Meter Grounding Gap and introduce three instance-centric metrics to quantify proximity precision, target visibility, and final-view grounding. To mitigate this gap, we propose REALM (Region-to-Entity Alignment for Last-3-Meter Navigation), a plug-and-play, architecture-agnostic refinement module that decouples fine-grained target approaching from long-horizon navigation. REALM uses a visibility-aware stopping strategy to reduce premature termination and improve final viewpoint alignment. We further construct REVERIE-AIM, which provides object-instance-level goals and 180K short-horizon training samples for final-stage target approaching. Extensive evaluations across four diverse VLN backbones show that REALM consistently improves proximity precision and visual grounding success, demonstrating its broad applicability.
Read the full abstract

All authors Xiangyu Shi, Ruoxi Yang, Wei Tao, Jiwen Zhang, Yanyuan Qiao, Qi Wu

Computer Vision

G$^2$TAM: Geometry Grounded Track Anything Model

Chenming Zhu, Peizhou Cao, Jingli Lin et al. (+5 more)

Human spatial understanding arises from jointly perceiving geometry and semantics, enabling consistent object identification and localization across viewpoints and time. Current video segmentation models depend on explicit object appearance memory banks for instance tracking, yet they remain vulnerable to large viewpoint changes and long-term occlusions. Leveraging the spatial consistency afforded by modern feed-forward 3D reconstruction models, we propose the Geometry Grounded Tracking Anything Model (G$^2$TAM), a unified framework for promptable instance tracking in 3D using only unordered RGB images or videos. G$^2$TAM employs spatially aligned geometric representations as implicit memory, ensuring stable instance identity and localization across frames and views. At its core is a cross-modal spatial encoder that integrates visual and textual prompts into a shared geometric space, enabling end-to-end spatial reconstruction and instance-consistent mask prediction. To support training and evaluation, we construct InsTrack, a large-scale dataset with a dedicated validation split for benchmarking. Extensive experiments show that G$^2$TAM delivers strong cross-view consistency, promptable instance spatial tracking, video object segmentation and spatial reconstruction, establishing a foundation for interactive, geometry-grounded spatial reasoning.
Read the full abstract

All authors Chenming Zhu, Peizhou Cao, Jingli Lin, Wenbo Hu, Yunlong Ran, Jiangmiao Pang, Tai Wang, Xihui Liu

Computer Vision

Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

Zhenfeng Su, Kang Zhao, Han Bao et al. (+4 more)

While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size. Depth pruning, which removes entire layers from a ViT, is notoriously difficult for accuracy recovery despite its potential to deliver higher speedups, limiting the acceleration achieved by existing joint width-and-depth pruning methods. In this work, we reveal that the failure of existing depth pruning methods lies in their neglect of heterogeneity between different layers, and we introduce HetDPT, a heterogeneity-aware depth pruning method that avoids dimension mismatch. Comprehensive experiments on ImageNet-1K, CIFAR-100, COCO, and ADE20K validate our method: HetDPT achieves a 1.58$\times$ speedup for DeiT-B while maintaining accuracy and a 1.39$\times$ speedup for DeiT-S with nearly no accuracy degradation. Furthermore, when combined with width pruning, HetDPT+ sets a new state-of-the-art record in extreme ViT pruning, enhancing the acceleration ratio from 4.24$\times$ to 5.19$\times$ for the Isomorphic-Pruning-2.6G configuration while maintaining near-lossless accuracy; our code is available at https://github.com/Efficient-AI-for-All/HetDPT.
Read the full abstract

All authors Zhenfeng Su, Kang Zhao, Han Bao, Tao Yuan, Zhongzhe Hu, Xianzhi Yu, Wenxuan Wang

Computer Vision

City-Level 3D Surface Reconstruction with Viewpoint Orientation Partitioning and Scene Completion

Liang Han, Wenyuan Zhang, Junsheng Zhou et al. (+2 more)

Multi-view 3D surface reconstruction is a longstanding challenge in computer vision. Although recent large-scale reconstruction methods based on 3D Gaussian Splatting (3DGS) achieve impressive novel-view synthesis, producing high-quality surfaces over large scenes remains difficult, due to complex geometry, long optimization, and limited memory. In this paper, we propose a novel yet simple partitioning method to efficiently and faithfully reconstruct large-scale scene surfaces. Our key insight lies in a scene partitioning method based on viewpoint orientation. This partitioning approach ensures that views with similar orientations are jointly involved for more accurate depth estimations, leading to precise surface reconstructions and balanced computation on multiple GPUs in parallel. In addition, we propose a strategy to detect and repair missing regions in the initial point cloud caused by sparse viewpoints or insufficient textures, thereby further improving the geometric quality. Extensive experiments on the GauU-Scene, MatrixCity, and UrbanScene3D datasets demonstrate that our method outperforms the state-of-the-art approaches in surface reconstruction for large-scale scenes. Project page: https://hanl2010.github.io/VOP-GS.
Read the full abstract

All authors Liang Han, Wenyuan Zhang, Junsheng Zhou, Yu-Shen Liu, Zhizhong Han

Computer Vision

Self-Improving Diffusion Classifiers with Minority Preference Optimization

Hyunsoo Kim, Jungmyung Wi, Soobin Um et al. (+2 more)

Prior studies have demonstrated that diffusion classifiers achieve robust zero-shot classification performance. However, their effectiveness is strongly tied to the pretraining data distribution: they perform well in majority, high-density regions of the data manifold, but are significantly less accurate in minority, low-density regions. Although prior works on minority sampling have focused on generating more minority-like images, what minority sampling fundamentally enables beyond generation remains underexplored. In this paper, we reveal a direct relationship between minority sampling in generation and the perception capability of diffusion classifiers. Specifically, we show that enhancing minority sampling broadens the coverage of underrepresented regions on the data manifold, thereby improving diffusion-based recognition. To exploit this connection, we propose Self-Improving Diffusion Classifiers with Minority Preference Optimization (MiPO), which fine-tunes a pretrained diffusion model using minority preference rewards. Using only arbitrary caption data, MiPO generates candidate samples, rewards those that better cover minority regions, and optimizes the model with LoRA and Group Relative Policy Optimization, without additional image data, external foundation models, or external reward models. This enables stable, prompt-adaptive minority sampling and translates low-density generative coverage into improved zero-shot diffusion classification. To sum up, we show that diffusion classifier perception is biased toward majority regions, demonstrate that this bias can be alleviated through minority preference optimization, and evaluate MiPO on five standard datasets.
Read the full abstract

All authors Hyunsoo Kim, Jungmyung Wi, Soobin Um, Donghyun Kim, Suhyun Kim

Computer Vision · Cross-listed from eess.IV

GALOSH: Blind, Training-Free Denoising of Raw Bayer and sRGB Images by Parallel-Friendly Local Shrinkage

Yoshiro Sato

Classical training-free denoisers such as BM3D and non-local means owe much of their strength to search: content-dependent block matching whose memory traffic and data-dependent control flow parallelize poorly and preclude fixed-latency implementations. Learned denoisers reach the highest quality, but they need training data, degrade outside their training domain (which we also observe), and carry per-pixel compute budgets that effectively require a GPU. We present GALOSH (Generalized Anscombe LOcal SHrinkage), a redesign of training-free denoising that removes the search entirely and aims at multi-domain coverage, speed, and quality at once: a blind per-image Poisson-Gaussian noise fit, a generalized Anscombe transform, a two-pass local Walsh-Hadamard shrinkage of luminance, and a luminance-guided local regression of chrominance – two deliberately different operators for the two perceptually different noise components, each with its own strength control. Every stage is local, data-independent, and regular – the same computation graph for every pixel of every image. One core serves two domains: raw Bayer mosaics and sRGB/YUV images. On four real-noise benchmarks (SIDD Medium and RawNIND, raw and sRGB) GALOSH is consistently the strongest among the tested blind, training-free methods – surpassing BM3D- and NLM-family baselines even when those are given an oracle noise level – and approaches trained networks on raw data while remaining below in-domain trained networks at high ISO in sRGB. Being search-free makes it fast: 7x-650x faster than the DL baselines on the same GPU at full benchmark size, and the only strong method in the comparison that also runs practically on plain CPUs. The fixed, data-independent structure is designed to map naturally onto fixed-point and streaming hardware, supported by an operation-count analysis and a working INT16 fixed-point realization.
Read the full abstract

Computer Vision

Sparse-View Surface Reconstruction using Gaussian Splatting through High-Confidence Depth Propagation with Normal Priors

Liang Han, Bangcai Wei, Junsheng Zhou et al. (+2 more)

3D reconstruction from sparse views is a challenging task in 3D computer vision. Recent studies on 3D Gaussian Splatting (3DGS) have achieved remarkable results with sparse views in novel view synthesis, yet reconstructing high-quality geometric surfaces from sparse views remains a challenge, due to the limited geometry clues and the discreteness of Gaussians. In this paper, we propose a novel 3DGS-based method for high-fidelity surface reconstruction from sparse views. Our key insight is to introduce a normal-guided depth propagation approach, which can extend depth information from high-confidence regions to constrain the depth in low-confidence areas. Additionally, we propose an abnormal depth edge-aware regularization to address depth discontinuities caused by the discreteness of Gaussians. Extensive experiments on DTU and Tanks-and-Temples datasets demonstrate that our method outperforms the state-of-the-art methods in sparse view surface reconstruction. Project page: https://hanl2010.github.io/DP-GS.
Read the full abstract

All authors Liang Han, Bangcai Wei, Junsheng Zhou, Yu-Shen Liu, Zhizhong Han

Computer Vision · Cross-listed from eess.IV

Deep Learning-Based Characterization of Detonation-Cell Size Distributions in Soot-Foil Records

Mingyang Bu, Robson A. Schneider, Karl P. Chatelain et al. (+5 more)

The geometric size and regularity of detonation cells are key physical parameters for characterizing detonation waves. Traditional manual measurement of soot foils is time-consuming and subjective, while existing computer vision techniques often exhibit poor generalization on real experimental images with high noise, blurred boundaries, and severe overlapping. To address this, we propose a novel method for automated recognition and high-order feature extraction of detonation cells based on deep learning instance segmentation (Mask R-CNN). By constructing a custom heterogeneous dataset (numerical simulations and physical experiments) and integrating transfer learning, the model achieves accurate pixel-level mask prediction within highly noisy flow fields. Results indicate high pixel-level agreement in benchmark validations and strong robustness against noise in complex real-world soot foils. Predicted average cell sizes agree well with manual measurements, yielding relative errors under 2% and 3.5% for regular and irregular conditions, respectively. Sensitivity ablation experiments confirm the model's scale adaptability and guided the establishment of a standardized preprocessing paradigm for appropriate image patching. Overcoming the limitation of extracting only global average sizes, this model achieves automated tracking of the transient spatial evolution of cell sizes along the propagation direction. Furthermore, it quantitatively extracts high-order regularity features, such as the irregularity index (RI) and standard deviation of cell deflection angles, demonstrating consistency with theoretical expectations. The proposed method enhances the efficiency and objectivity of statistical analysis, providing a powerful data extraction tool for experimental and numerical soot foils.
Read the full abstract

All authors Mingyang Bu, Robson A. Schneider, Karl P. Chatelain, Mhedine Alicherif, Yingchen Shi, Andrés Z. Mendiburu, Deanna A. Lacoste, Bing Wang

Computer Vision

GeoSAM-Lite: A Lightweight Foundation Model for Onboard Remote Sensing Segmentation

Yongcong Wang, Jie Zhang, Rui Jiang et al. (+3 more)

The deployment of large-scale foundation models like Segment Anything Model (SAM) on resource-constrained Earth observation platforms is hindered by prohibitive computational costs and the domain shift between natural and remote sensing imagery. To address these challenges, we propose Geospatial Segment Anything Model-Lite (GeoSAM-Lite), a lightweight, prompt-free segmentation framework designed for efficient onboard remote sensing segmentation. GeoSAM-Lite incorporates two core innovations: (1) Geospatial-Domain Initialization (Geo-Init), a domain-aware pre-training strategy that distills geospatial priors from a specialized teacher to bridge the domain gap; and (2) Feature Fusion Layers (FFL), which recalibrate spatial features and restore high-frequency boundary cues to overcome the capacity bottlenecks of lightweight backbones. Experiments across representative datasets, with a primary focus on cloud scenarios to evaluate performance under extreme scale variations and complex boundaries, demonstrate that GeoSAM-Lite achieves competitive accuracy while reducing parameters by 92.8% compared to the heavyweight RSAM-Seg. By establishing a superior Pareto frontier between efficiency and fidelity, GeoSAM-Lite offers a practical solution for real-time segmentation on edge devices.
Read the full abstract

All authors Yongcong Wang, Jie Zhang, Rui Jiang, Xubing Yang, Ting Yun, Li Zhang

Computer Vision

Exploring SAM Supervision for Fine-Grained UAV Target Segmentation under Data Scarcity

Le-Anh Tran

Unmanned aerial vehicle (UAV) target segmentation remains challenging due to the small size of objects, appearance variations, cluttered backgrounds, and the scarcity of densely annotated data. These factors hinder the performance and practical deployment of lightweight segmentation models in real-world UAV applications. To address this problem, this paper investigates the use of SAM3 (Segment Anything Model 3) as a pseudo-label generator for training compact segmentation networks. Specifically, two supervision paradigms are explored: (i) direct pseudo-supervision using unaltered SAM3-generated masks, and (ii) a refinement strategy that re-applies SAM3 to localized image patches for improved mask quality. Based on these paradigms, a two-stage SAM3-guided pseudo-label generation framework is proposed. In the first stage, SAM3 generates coarse masks for initial object localization. The localized regions are subsequently cropped into patches and processed by SAM3 again to generate fine masks with accurate object boundaries and discard false positives. The resulting coarse and fine masks are then used as pseudo-labels to optimize a lightweight network, termed IPS-Seg, which consists of three components: an IdentityFormer backbone for feature extraction, an Atrous Spatial Pyramid Pooling module for multi-scale context aggregation, and a PixelShuffle-based decoder for spatial resolution recovery. Extensive experiments under multiple supervision settings demonstrate the effectiveness of the proposed framework. The results show that IPS-Seg achieves a favorable trade-off between segmentation accuracy and computational efficiency while benefiting consistently from the proposed pseudo-label generation strategy. These findings highlight the potential of large-scale foundation models as annotation sources for training compact task-specific segmentation networks in low-label vision domains.
Read the full abstract

Computer Vision

EmCom-Diffusion: Probing Visual Reflection in Emergent Languages via Image Generation

Haruumi Omoto and Tadahiro Taniguchi

Measuring the extent to which emergent languages encode the visual content of their inputs is an open problem. We refer to this property as visual reflection: the extent to which emergent messages preserve information about their source images that can be recovered without appeal to the speaker-listener pair that produced them. Existing metrics measure it only indirectly, through proxies such as human-defined concept inventories, natural-language captions, structural distance correlations, or Referential Game accuracy, each of which can either miss visual content the message encodes or credit content it does not. We propose EmCom-Diffusion, an evaluation framework that measures visual reflection directly: it reconstructs each input image from its emergent message and compares the reconstruction with the original image itself, rather than with human-defined targets. Concretely, it finetunes a pretrained text-to-image diffusion model on (image, emergent-message) pairs and scores visual reflection as the perceptual similarity between the reconstructed and original images, operating generatively rather than discriminatively. Instantiating it on MS-COCO with a Referential Game, we validate the metric against random and fixed-token baselines under three pretrained visual encoders, and compare it against four existing metrics (CBM, supervised translation, TopSim, and R@1). EmCom-Diffusion captures visual content the other metrics miss.
Read the full abstract

Computer Vision · Cross-listed from q-bio.QM

Multimodal Knowledge Aggregation for Microbial Keratitis: Geographic Confounding and Shortcut Risk in Pooled Multicountry Data

Yiqing Wang, Maria A. Woodward, Ziyun Yang et al. (+11 more)

To develop an open-source multimodal model for bacterial-versus-fungal microbial keratitis classification and assess shortcuts from geographic pooling. Participants: 1,645 US and Indian patients contributed 17,158 slit-lamp images under blue-light, sclerotic-scatter, and white-light illumination. Methods: The model combined cross-modality contrastive learning, modality-specific fine-tuning, and fusion with 7 clinical variables. Diagnoses used smear, culture, and polymerase chain reaction results, with expert review of culture-negative cases. The model and 5 comparators used 5 patient-disjoint outer folds stratified by country and organism, with inner validation. Secondary analyses examined within-country performance, a country-only reference, ablations, and imbalance mitigation. Code: https://github.com/yqwang01/TPMKA. Main Outcome Measures: Patient-level macro-F1, class-specific F1, accuracy, and area under the receiver operating characteristic curve (AUROC), overall and by country. Results: The model had the highest overall macro-F1 (83.06%; 95% CI, 81.58-84.56) and accuracy (84.50%; 95% CI, 83.16-85.84) among matched comparators, with 89.9% AUROC (95% CI, 88.7-91.1). Paired differences were not consistently significant. AUROC was 83.9% in India and 67.0% in the US; bacterial F1 in India was 47.78%, and fungal F1 in the US was 40.97%. A country-only reference achieved 81.88% accuracy and 78.2% AUROC overall despite 50.0% AUROC within each country. Resampling and group distributionally robust optimization improved some underrepresented groups without removing the geographic pattern. Conclusions: The model compared favorably with matched methods, but geographic prevalence differences made pooled performance more reassuring than within-country discrimination. Independent geographic validation is needed.
Read the full abstract

All authors Yiqing Wang, Maria A. Woodward, Ziyun Yang, N. Venkatesh Prajna, Chunming He, Leslie M. Niziol, Mercy Pawar, Ming-Chen Lu, Guillermo Amescua, Rachel A. F. Wozniak, Sejal Amin, Abinaya Krishnan, Prabhleen Kochar, Sina Farsiu

Computer Vision

Attending to Multimodal Generation One Token at a Time

Varun Gupta, Vineet Gandhi and Makarand Tapaswi

Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context. Prior work on interpretability has focused on individual layers and circuits (where), leaving the token-level dynamics of multimodal computation during generation (when) underexplored. We address this gap and study attention shifts as per semantic role; tracking model attention to image, text, instruction, and previously generated tokens, One Token at a Time (OTaT). We introduce multimodal tasks that require explicit switching between visual and textual context within a single response. Across two mainstream model families and four open-weight MLLMs of varying sizes, we establish consistent patterns: attention to image peaks at tokens requiring image-derived information, instruction tokens are revisited during task transitions, and attention to previously generated tokens increases as the generation progresses. Causal attention blocking interventions validate the functional role of these trends. We profile model behavior under disrupted attention and observe responses falling back to language priors, or exhibiting cross-modal leakage, denial, or recovery. Finally, informed of the attention dynamics through our novel analysis, we propose a simple test-time intervention to boost attention to the relevant modality at the right time, significantly improving multimodal task performance.
Read the full abstract

Computer Vision

ProxyUp: Training-Free Proxy-Conditioned Video Generation for Controllable Dynamics

Zanwei Zhou, Jiazhong Cen, Jiemin Fang et al. (+7 more)

Precise control over complex dynamics remains challenging for modern video generative models, as text prompts alone often cannot specify physically plausible, fine-grained motion and interactions. We introduce $\textit{proxy-conditioned video generation}$, where a coarse proxy video from physics-based simulation or real-world recording serves as a dynamics carrier to control foreground object motion. Given a proxy video and a text prompt, the goal is to synthesize a new video that preserves the proxy dynamics while generating novel content and plausible interactions aligned with the prompt. Since paired proxy-target videos are difficult to obtain, we propose $\textbf{ProxyUp}$, a training-free framework built on pretrained video generative models. ProxyUp first inverts the proxy video into an intermediate latent representation and applies $\textbf{region-wise latent noising}$, preserving motion-critical proxy latents while injecting noise into regions intended for text-driven regeneration. To mitigate the distribution mismatch and weak foreground-background coupling introduced by this heuristic latent composition, we further propose $\textbf{Stochastic Flow Relaxation (SFR)}$, which progressively relaxes the composed latent toward the model's learned distribution before ODE sampling. Experiments on both simulation and real-world proxies show that ProxyUp outperforms strong video editing and motion transfer baselines in dynamic fidelity and text alignment.
Read the full abstract

All authors Zanwei Zhou, Jiazhong Cen, Jiemin Fang, Yumeng He, Chen Yang, Sikuang Li, Fanpeng Meng, Zhikuan Bao, Wei Shen, Qi Tian

Computer Vision

Leveraging Pathology Co-occurrence for Test-Time Adaptation in Chest X-Ray Diagnosis

Woojin Jeong, Yujin Choi, Dongbin Kim et al. (+2 more)

Medical imaging models often degrade when deployed at new clinical sites due to differences in imaging equipment, protocols, and patient populations. Test-time adaptation (TTA) addresses this by updating a pretrained model using only unlabeled target data, without access to source data. However, existing TTA methods were designed for single-label classification on natural image benchmarks, minimizing entropy uniformly across all samples without considering label dependencies. This overlooks a key property of multi-label medical imaging: pathologies do not occur independently but exhibit structured co-occurrence patterns. In this work, we propose Co-occurrence Weighted Adaptation (CoWA), which leverages disease co-occurrence patterns as a reliability signal for adaptation. CoWA estimates label co-occurrence structure from model predictions and downweights samples that deviate from expected patterns, enabling adaptation to rely more on consistent predictions while reducing the impact of noisy ones. We evaluate CoWA on chest X-ray benchmarks under domain shifts and demonstrate consistent improvements over established baselines.
Read the full abstract

All authors Woojin Jeong, Yujin Choi, Dongbin Kim, Soyeon Park, Jaewook Lee

Computer Vision

IPDiff: Diffusion-driven ORSI Salient Object Detection with Information Reconstruction and Multi-Prior Guidance

Gongyang Li, Zhen Bai, Runmin Cong et al. (+3 more)

Existing Salient Object Detection in Optical Remote Sensing Image (ORSI-SOD) methods mainly adopt the static inference strategy, which uses fixed trained model parameters for saliency inference in the testing phase. This means that even if the generated saliency map has errors, it cannot be further optimized. In this paper, we propose the novel IPDiff, a Diffusion-driven ORSI-SOD method with Information Reconstruction and Multi-Prior Guidance. We build IPDiff based on a unique dynamic optimization strategy, which endows IPDiff with the ability to iteratively optimize saliency maps with a dynamic parameter. Specifically, we formulate ORSI-SOD as a conditional diffusion problem in IPDiff. IPDiff first extracts informative conditional priors from ORSIs, including the saliency prior and the hierarchical priors, in the prior network with the assistance of the information reconstruction-driven attention module. The saliency prior can provide positional information of salient objects, while the hierarchical priors can provide specific detail and semantic information of salient objects. Under the guidance of these priors, IPDiff then iteratively denoises random noise as the timestep dynamically changes in the denoising network, generating saliency maps that are close to ground truths. Notably, we simultaneously supervise IPDiff in both spatial and spectral domains through a hybrid loss function to achieve efficient network training. Comprehensive experiments on public ORSSD, EORSSD, and ORSI-4199 datasets demonstrate that our proposed IPDiff achieves the best performance compared to 46 state-of-the-art methods. The code and results of our method are available at https://github.com/MathLee/IPDiff.
Read the full abstract

All authors Gongyang Li, Zhen Bai, Runmin Cong, Dan Zeng, Weisi Lin, Xiao-Ping Zhang

Computer Vision · Cross-listed from cs.LG

Phase-Preserving Trimodal Transformer for Tropical Forest Biomass Estimation Using Optical and PolInSAR Data

Luiz Felipe Parente Santiago, Felipe Ferrari, Daniel Rodrigues dos Santos et al. (+1 more)

The accurate estimation of Above-Ground Biomass (AGB) in mature tropical forests remains a critical challenge in remote sensing, primarily due to the saturation of Synthetic Aperture Radar (SAR) signals in high-density areas and persistent cloud cover affecting optical imagery. To overcome these physical limitations, we propose the Trimodal Coherent Co-attention Transformer (TCCT), a physics-informed deep learning architecture. The TCCT natively fuses optical surface reflectance (Landsat-5) with complex-valued Polarimetric SAR Interferometry (PolInSAR) data from both P and L bands. Unlike traditional fusion methods, our architecture employs complex-valued encoders to preserve spatial phase coherence, coupled with a dynamic co-attention mechanism that acts as an adaptive gating module, reducing the weight of cloud-corrupted optical pixels and shifting reliance to microwave phase data. We also derived a localized spatial allometric calibration model via Levenberg-Marquardt optimization, tailored to the specific wood density of the Paracou region in the Amazon basin. Evaluated using a two-stage protocol, the TCCT first underwent a rigorous 5-fold cross-validation to establish robust global weights (achieving a global RMSE of 4.19 m). Subsequently, following a localized spatial fine-tuning phase over 200 epochs, the model attained an absolute RMSE of 3.78 m and an $R^2$ of 0.33 for Canopy Height Models (CHM), outperforming standard Random Forest, CNN, and Vision Transformer baselines. Our ablation study confirms that preserving phase coherence mitigates deep-canopy signal saturation. When converted to AGB, the fine-tuned TCCT map yielded a Relative RMSE (rRMSE) of 4.51% in dense forest areas above 50 Mg/ha. By meeting the European Space Agency (ESA) BIOMASS mission requirement of less than 20% error, the TCCT provides a robust framework for continuous carbon stock mapping in tropical biomes.
Read the full abstract

All authors Luiz Felipe Parente Santiago, Felipe Ferrari, Daniel Rodrigues dos Santos, Rosiane de Freitas

Computer Vision

From Geometric Labels to Semantic Understanding of Indoor Building Components Using Multimodal Large Language Models

Shuju Jing and Chao Yin

Point cloud-based understanding has become an important enabler for facility operation and maintenance involving indoor building components. However, existing methods output only discrete labels without explaining component functions or natural language interactions. This paper proposes Building-MLLM, a point cloud-centered multimodal large language model (MLLM) for indoor components, which models point clouds and instructions to generate responses across Simple Recognition, Complex Captioning, and Multi-Engineering Question Answering tasks. Building-MLLM addresses semantic concentration through four domain-specific mechanisms: Point Information Enhancer for task-relevant semantics, Geometry-Preserving Regularization preventing geometric erosion, fixed textual prefix for domain stabilization, and multi-dimensional LoRA balancing recognition with reasoning. A multi-constraint progressive instruction-generation engine is developed to compile a synthetic point cloud-text dataset with 4198 objects, 37,782 instruction-following pairs, and 47 categories. Experiments show that Building-MLLM achieves 88.00%, 65.10%, and 68.14% on the three task types, respectively, demonstrating superior indoor component language understanding and providing initial generalizability in transfer inference on other real-world datasets.
Read the full abstract

Computer Vision

ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation

Ahmed Abul Hasanaath, Bicheng Xu, Mir Rayat Imtiaz Hossain et al. (+2 more)

Gloss-free Sign Language Translation (SLT) translates sign language videos into spoken-language sentences without gloss annotations, avoiding costly labeling but requiring fine-grained modeling of hands, body, and facial cues. Existing methods often use single-modality or weakly fused features, limiting performance. We propose ViPo-MLLM, a framework that integrates spatio-temporal RGB and human pose features. Dedicated encoders model intra-modal dynamics and cross-modal attention captures long-range dependencies. The fused representation is conditioned with a structured prompt and processed by an LLM trained with contrastive and language modeling objectives. The proposed model was evaluated on the PHOENIX14T and CSL-Daily datasets and achieved new state-of-the-art results on both datasets. Moreover, the ViPo-MLLM model attained competitive performance compared to gloss-based recognition approaches, confirming the effectiveness of the proposed pose cues and cross-modal attention mechanisms.
Read the full abstract

All authors Ahmed Abul Hasanaath, Bicheng Xu, Mir Rayat Imtiaz Hossain, Leonid Sigal, Hamzah Luqman

Computer Vision

ClinOCR-Bench: A Comprehensive Clinical Scanned Document Dataset for Optical Character Recognition Model Evaluation

Enshuo Hsu, Jin Zhou and Kirk Roberts

Extracting textual information from scanned medical documents, such as external laboratory reports and manually filled forms, has been a major challenge in modern electronic health records (EHRs). Recent advancements in vision language models (VLMs) have shown great promise over traditional OCR tools. However, at this point, most clinical OCR studies were conducted on private, institutional data. To our knowledge, there are few publicly available datasets for evaluating OCR models in the clinical domain. Furthermore, common scanning artifacts that undermine OCR performance are not reflected in those datasets, leaving a systematic evaluation unfeasible. Therefore, we release a publicly available, realistic-looking OCR benchmark dataset, ClinOCR-Bench, with 384 scanned images across 6 subsets: Normal, Handwriting, Poor Quality, Rotation, Tables, and Mix-artifacts. ClinOCR-Bench features: 1) diverse document types and layouts, 2) full coverage of common EHR scan artifacts, 3) protected health information-free, 4) template-aware train/test split, and 5) adequate sample size for OCR benchmarking. Baseline OCR performance was evaluated using state-of-the-art open-weight and proprietary VLMs. The dataset and documentation are available on GitHub (https://github.com/ClinOCR-Bench/ClinOCR-Bench).
Read the full abstract

Computer Vision

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj et al. (+2 more)

Large vision language models (VLMs) report strong accuracy on medical question-answering, yet it remains unclear whether they reason from visual evidence or exploit textual shortcuts. We introduce a counterfactual evaluation framework that decouples visual and textual contributions by substituting input images with controlled surrogates blank, pixel-shuffled, image-absent, and CLIP-retrieved hard negatives and derive a suite of grounding metrics including the Visual Reliance Score (VRS) and Visual Hallucination Rate (VHR). We further introduce CORAL (COntrastive Retrieval-Augmented Learning), a 7B-parameter LoRA fine-tune of Qwen2.5-VL-7B trained with a Contrastive Grounding Objective (CGO) that penalises answer invariance under hard-negative image swaps. On a paired controlled evaluation across four closed-form medical VQA benchmarks (PathVQA, PMC-VQA, SLAKE, VQA-RAD; n=400 total), CORAL improves macro accuracy by +6.7 pp (P(Delta>0)=0.988) and reduces VHR by 8.0 pp (P<0.001) over the matched Qwen2.5-VL-7B base; neither MedVLThinker RL variant achieves a significant gain on either metric. Cross-domain diagnostics further reveal that image substitution costs only <=6.5 pp on medical benchmarks versus 48-61 pp on general-domain tasks, situating the grounding gap that CGO targets. We discuss evaluation limitations openly including train/eval benchmark overlap and underpowered secondary metrics and release our framework, training code, and model weights to support reproducible grounding audits of medical VLMs.
Read the full abstract

All authors Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj, Ashish Vashist, Jia Wu

Friday 3 July

Computer Vision · Cross-listed from cs.NI

HMS-SCP: Task-Oriented Multi-Scale Semantic Communication for V2X Cooperative Perception

Chun-Yeow Yeoh, Chee Keong Tan, Joanne Mun-Yee Lim et al. (+1 more)

Cooperative perception enables vehicles and infrastructure to exchange sensor data via Vehicle-to-Everything (V2X) communication, extending sensing coverage beyond occlusions and mitigating blind spots. While critical for autonomous driving and safety, practical deployments often rely on bandwidth-efficient late fusion. Recently, intermediate fusion has emerged as a promising approach for an optimal bandwidth-accuracy trade-off. However, in dense urban environments, cumulative bandwidth demands can overwhelm network capacity, potentially compromising safety-critical Cooperative Intelligent Transport Systems (C-ITS) functions. To alleviate these problems, this paper proposes Hierarchical Multi-Scale Semantic-Aware Cooperative Perception (HMS-SCP), a robust noise-resilient and bandwidth-efficient framework for task-oriented semantic communication in cooperative perception. HMS-SCP employs a spatial importance predictor to identify task-relevant grid elements at each scale, which are then directly mapped into complex-valued symbols for Joint Source-Channel Coding (JSCC). Unlike prior methods that rely on high-dimensional symbol projections for robustness, HMS-SCP exploits structural semantic redundancy across multiple scales to enhance resilience against channel noise, while maintaining an ultra-low symbol rate. This design significantly reduces bandwidth consumption and mitigates network congestion in high-density vehicular environments. Extensive evaluations on the simulated OPV2V and real-world DAIR-V2X datasets demonstrate that HMS-SCP effectively prevents performance collapse under severe Rayleigh fading and extreme compression ratio, maintaining high-confidence far-field detection with a real-time latency of below 16~ms, well within the safety-critical thresholds for dynamic V2X environments.
Read the full abstract

All authors Chun-Yeow Yeoh, Chee Keong Tan, Joanne Mun-Yee Lim, Heng-Siong Lim

Computer Vision · Cross-listed from cs.HC

DREAMS: Diverse Reactions of Engagement and Attention Mind States Dataset

Monisha Singh, Gulshan Sharma, Ximi Hoque et al. (+1 more)

Active attention and engagement are important in improving users' learning experiences. Engagement refers to the level of involvement and interest individuals show towards a particular task. Attention, on the other hand, refers to a state where someone is entirely focused on a particular task with conscious awareness. Engagement and attention are different but closely linked concepts and can influence each other bidirectionally. To explore the relationship between user engagement and attention, we introduce the Diverse Reactions of Engagement and Attention Mind States (DREAMS) dataset. The dataset includes facial video recordings of 32 users in naturalistic settings watching various stimuli to evoke diverse emotions. We then analyze user engagement and attention states in these videos by framing it as a classification problem, exploring single-task, transfer learning task, and multi-task settings. In single and transfer learning task settings, separate networks are applied to predict engagement and attention states. Whereas in multi-task settings a shared network is applied, which jointly learns to predict both engagement and attention states. Moreover, we examine participants' performance on video-based questionnaires and evaluate their perceived cognitive workload. In our findings, we observe (a) better classification performance in predicting engagement states in both transfer and multi-task learning compared to single-task learning and (b) higher engagement and attention states correlate with lower cognitive load and improved task performance. The dataset and the code are publicly available and can be accessed through https://sites.google.com/view/dreams-dataset/dataset.
Read the full abstract

All authors Monisha Singh, Gulshan Sharma, Ximi Hoque, Abhinav Dhall

Computer Vision · Cross-listed from cs.NE

Local Synaptic Rules Can Implement a SIGReg Gradient Without Backpropagation

Martin Andrews

We prove that two canonical local synaptic learning rules, the potentiation arm of spike-timing-dependent plasticity (STDP$^+$) and homeostatic plasticity (instantiated here via flashlight granule-cell-like neurons), together can implement the exact gradient of a SIGReg-like self-supervised learning objective. The equivalence requires no gradient calculations, no global error signals, no weight transport, and no label information: the only inputs are pre- and post-synaptic firing rates, local firing statistics, and the temporal contiguity of natural sensory streams. On a synthetic clustering task designed to probe whether class structure can be recovered from temporal ordering of inputs alone, ordered presentation raised cluster separation (CSR) to 2.49 while random ordering left it near baseline (0.83), a roughly threefold ($\approx 3.5\sigma$) separation attributable solely to input ordering. On temporally ordered MNIST, a two-layer network trained entirely with these rules achieved 87.3% linear-probe accuracy, showing that the mechanism functions end-to-end.
Read the full abstract

Computer Vision · Cross-listed from cs.CR

Abductive Corroboration of Probabilistic AI Models for Forensic Synthetic Media Detection

Junade Ali

Artificial Intelligence (AI) models, at their core, apply general learnings from broad datasets to individual circumstances using probabilistic behaviour. This inductive approach stands in contrast to deductive reasoning approaches which seek to prove conclusions from their premises. However, research has shown that deductive reasoning with AI models is a challenging problem and in the real-world it may not always be feasible. An alternative way forward is to leverage abductive reasoning, seeking to corroborate the output of multiple approaches to identify the most likely conclusion from the factual matrix. We apply this to synthetic media detection in forensic settings, and find we are able to disproportionately lower the risk of false positives to true positive recall. We also provide the first empirical evaluation of OpenAI's rollout of SynthID on synthetic images and evaluate how complementary different synthetic media detection approaches are.
Read the full abstract

Computer Vision

Moonstone: A Multimodal Foundation Model and Benchmark for Lunar Remote Sensing

Ayush Prasad and Swarnalee Mazumder

Decades of orbital missions have produced multi-modal remote sensing data for the Moon, spanning optical imagery, spectroscopy, thermal emission, radar, gravity, and elemental composition. Yet these datasets remain fragmented across archives, and no benchmark exists for evaluating machine learning on lunar data. We introduce Moonstone, the first multi-modal foundation model benchmark for lunar remote sensing. Our contributions are: (1) a 28-channel, 128 pixels-per-degree (~237 m) global lunar pretraining dataset from seven instrument families across five missions, (2) MG-MAE, a modality-grouped masked autoencoder with per-group convolutional tokenizers, a shared Vision Transformer encoder, attention masking for missing modalities, coverage-adaptive masking for heterogeneous spatial coverage, and spectral continuity regularization for physically plausible reconstructions, and (3) a benchmark of six downstream tasks covering classification, regression, and segmentation. MG-MAE pretrained features outperform scratch baselines on all tasks and surpass both ImageNet-pretrained and vanilla MAE baselines by large margins. Data and code are available at https://huggingface.co/datasets/ayushprd/Moonstone and https://github.com/ayushprd/Moonstone .
Read the full abstract

Computer Vision · Cross-listed from eess.IV

Model Confidence-Guided Multi-Image Fusion of Fundus Images for Diabetic Retinopathy Diagnosis

Ananya Raghu, Anisha Raghu, Alice S. Tang et al. (+3 more)

Purpose: Early screening for eye diseases is critical in low- and middle-income countries where access to care is limited. We investigate whether a confidence-guided, multi-image diabetic retinopathy diagnosis framework can integrate image filtering with confidence-aware predictions for reliable screening at capture. Methods: We develop a multi-image fusion method that aggregates retinal views to improve confidence and balanced accuracy. Our method uses confidence to identify unreliable predictions, prompting retakes when needed. We compare: (1) a cascaded image-quality and disease diagnosis pipeline using a single image per patient, (2) confidence-based prediction, and (3) our confidence-based multi-image fusion pipeline. All methods are evaluated using a RETFoundGreen backbone on the mBRSET (n = 1,234) and BRSET (n = 7,599) datasets. Results: At 70% coverage, our method achieves 91% balanced accuracy on mBRSET and 97% on BRSET, improvements of ~12% and ~6%, respectively, over cascade filtering. The image-quality cascade reaches sensitivities of 61% on mBRSET and 86% on BRSET, whereas our framework reaches 94% and 96%, respectively, at 50% coverage. Conclusions: Human-annotated quality labels are weakly associated with diagnostic performance, and confidence-based filtering consistently outperforms image quality-based cascaded pipelines. Translational Relevance: Using confidence-based multi-image fusion, patients receive more reliable predictions, reducing incorrect diagnoses during screening. The lightweight backbone and single inference pass per image make the framework compatible with low-latency mobile screening systems in resource-limited settings.
Read the full abstract

All authors Ananya Raghu, Anisha Raghu, Alice S. Tang, Yannis M. Paulus, Tyson N. Kim, Tomiko T. Oskotsky

Computer Vision

Probing Identity-Specific Motion Signatures: A Controlled Diagnostic Study

Yingtie Lei, Fangxun Liu, Baicheng Wu et al. (+13 more)

Identity recognition (e.g., person, animal re-identification) has traditionally relied heavily on static appearance cues. Yet motion–consistent, individual-specific dynamics–can provide a complementary and potentially more robust signature, especially when appearance is weak or variable. This raises a fundamental question: when identity-specific motion cues are clearly present, to what extent do modern video models use them for recognition? To investigate this question, we conduct a systematic diagnostic study and introduce BALLER120, a controlled benchmark of 120 professional basketball players performing free-throws. By focusing on the same multi-phase action across individuals, BALLER120 reduces action-level variation and identity-correlated acquisition biases, enabling fine-grained analysis of identity-specific kinematic patterns. We find that modern video models can predict identity accurately from RGB videos, but often rely on static appearance cues such as faces and jersey regions, even when informative motion cues are available. Strikingly, when appearance is suppressed through silhouette-only or skeleton-only inputs, the same model architectures shift toward motion micro-patterns (e.g., foot placement and elbow bending). Despite containing less visual information, appearance-suppressed representations achieve competitive accuracy and stronger robustness to appearance shifts. Our qualitative analyses further show that appearance-suppressed models attend to distinctive motion patterns across individuals. Overall, our study demonstrates that identity-specific motion signatures are present, informative, and learnable, but modern video models may overlook them in favor of easier static shortcuts unless appearance cues are explicitly suppressed.
Read the full abstract

All authors Yingtie Lei, Fangxun Liu, Baicheng Wu, Colin Lee, Ziheng Zhang, Junke Yang, Zhiyuan Tao, Xuyan Huang, Shuheng Wang, William Koran, Kyle Park, Elijah H Buckwalter, Cheng-Hsuan Chiang, Tejas Naik, Daniel Yi, Wei-Lun Chao

Computer Vision

RADIO1D: Elastic Representations for Condensed Vision Modeling

Greg Heinrich, Mike Ranzinger, Collin McCarthy et al. (+6 more)

This paper challenges the assumption that vision-language models (VLMs) require fixed patch-based 2D vision features. Analyzing fine-tuned vision encoders, we find that representations become increasingly abstract and less spatially coherent during VLM training. Notably, models trained with image-text alignment (such as SigLIP2) develop a small number of specialized tokens that effectively summarize global image content. Building on this, we introduce RADIO1D, which compresses images into a compact, variable-length 1D token sequence using multi-teacher knowledge distillation and an autoencoder design. The resulting representations exhibit strong hierarchical summarization, enabling accurate scene understanding - even with a single token - and support improved composition-aware image retrieval. In VLMs, RADIO1D provides flexible accuracy-efficiency tradeoffs through adjustable token counts, delivering competitive performance on diverse multimodal benchmarks with lower computational overhead and better accuracy.
Read the full abstract

All authors Greg Heinrich, Mike Ranzinger, Collin McCarthy, Natan Bagrov, Eugene Khvedchenya, Bryan Catanzaro, Jan Kautz, Andrew Tao, Pavlo Molchanov

Computer Vision

A Vision Based System for Guided and Collaborative Reconstruction of Fragmented Documents

Oliver Krumpek and Diana Leo

This paper presents the development and evaluation of a collaborative system for real-time reconstruction of fragmented paper documents in the context of cultural heritage preservation. The developed system includes a collaborative robot, or cobot, that can fully manage the positioning of paper fragments using a specially designed vacuum-based suction attachment. This attachment enables gentle and precise positioning, ensuring the preservation of fragile materials. With this device, we are able to achieve a positioning repeatability of 0.57mm for fragments of 8cm^2. The system offers users the flexibility to choose between manual positioning, with visual guidance, or fully automated positioning performed by the cobot. To further improve the reconstruction process, AI methods for image interpretation, specifically for segmentation and positioning tasks, were applied and evaluated for their applicability to template-based reconstruction of damaged paper fragments. Our investigation provides critical insights into the performance of different local feature matching methods under different document types, taking into account rotation, scale robustness, and the degree of damage to the fragments. With a focus on the reconstruction of damaged and optically altered archival material, SE2-LoFTR, a detector-free local feature matching method, was chosen as the preferred method for the system due to its robust performance in our experiments.
Read the full abstract