Computer Vision
SAGE: An Expert-Annotated South Asian GI Endoscopy Dataset for Multimodal Learning and Hallucination Analysis
Niyoj Oli, Sachin Acharya, Sandesh Pokhrel et al. (+7 more)
Gastrointestinal cancers represent a growing health burden in the South Asian region, driven largely by rapid changes in socio-economic conditions and lifestyle habits. However, early diagnosis remains limited by inadequate equipment, financial resources, and scarce GI expertise. AI-assisted diagnosis and report generation show great promise in alleviating this problem by providing non-specialist healthcare workers the technical expertise to perform diagnosis. Yet, almost all open-source, publicly available
datasets are predominantly collected from the European region, with minimal representation from the South Asian region. The lack of open-source GI
datasets from diverse geographic regions has made it difficult to assess whether population bias is present in existing models, and to develop geographically inclusive AI tools for automated GI diagnosis. To address this gap, we introduce SAGE: An Expert-Annotated South Asian GI Endoscopy
dataset for
Multimodal Learning and
Hallucination Analysis, for image captioning, multi-label classification, and VQA tasks. It consists of 1,300 images, their captions along with
hallucination tag, 18 labels and 14,276 question-answer pairs making it well-suited for diverse range of tasks including classification, benchmarking, and fine-tuning LMMs. We further conducted benchmarking of task-specific models, like multi-class classifiers on the effect of population shift which reveals that such models suffers the most with an average F1 score drop of 54 points on South Asian
dataset. Further, benchmarking of contemporary LMMs reveals a substantial drop in the average GREEN score for anatomical landmark detection (0.308) and abnormality detection (0.410). We open-source our
dataset under the CC BY-SA 4.0 license, and we hope to encourage others to contribute toward more inclusive
dataset representation and help counteract population bias in medical AI.
Read the full abstract ▾
All authors Niyoj Oli, Sachin Acharya, Sandesh Pokhrel, Sanjay Bhandari, Ramesh Rana, Nikesh Mani Shrestha, Ram Bahadur Gurung, Yash Raj Shrestha, Prashnna K Gyawali, Binod Bhattarai
Computer Vision
Feed-forward Motion In-betweening for Any 4D
Hiroki Nishizawa, Hubert P. H. Shum, Yoshihiro Fukuhara et al. (+2 more)
4D dynamics (3D geometry evolving over time) is a fundamental representation of the physical world and plays a crucial role in world modeling (e.g., animation and games). Owing to the scarcity of large-scale, long-horizon 4D mesh data with arbitrary shapes, early text-to-4D methods rely on
distillation or test-time optimization from
video diffusion priors, making inference prohibitively slow. Recent feed-forward generators greatly reduce inference cost but offer limited spatiotemporal controllability, and short-horizon generation often leads to error accumulation in long-horizon sequences. We propose a novel feed-forward in-betweening framework for arbitrary 4D meshes with keyframe conditioning. Building on universal mesh-animation latents, we introduce a frame-wise mesh VAE that encodes each frame into
topology-agnostic latent
tokens anchored by a reference mesh for keyframe conditioning. We further introduce a keyframe-conditioned rectified flow model with an MMDiT backbone that synthesizes non-keyframe frames conditioned on sparse keyframes. Experiments show strong performance and improved controllability on both DyMesh16 and DyMesh32
benchmarks.
Read the full abstract ▾
All authors Hiroki Nishizawa, Hubert P. H. Shum, Yoshihiro Fukuhara, Hirokatsu Kataoka, Shigeo Morishima
Computer Vision
Surgical Anatomy Recognition with Context Learning using Foundation Representations
Ronald L. P. D. de Jong, Tim J. M. Jaspers, Raf A. H. Vervoort et al. (+9 more)
Accurate recognition of anatomical structures is essential for safe and effective minimally invasive surgery (MIS), yet it remains underexplored in
surgical computer
vision due to limited annotated data and methods tailored primarily to natural scenes. In this work, we present a combined
dataset and model framework to advance anatomy-aware perception in MIS. First, we introduce ATLAS-120k, a large-scale clip-level semantic
segmentation dataset comprising over 120,000 annotated frames from 100
surgical videos spanning 14 procedures and multiple modalities, including laparoscopic and
robot-assisted surgery. The
dataset captures substantial procedural variability and was created using a scalable annotation pipeline that integrates expert manual labeling, automated propagation, iterative refinement, and surgeon verification to ensure high-quality annotations. Second, we propose ATLAS (Anatomy Recognition with Context Learning using Foundation Representations), a
video semantic
segmentation model specifically designed for
surgical anatomy recognition. Unlike conventional approaches that emphasize object tracking, ATLAS leverages foundation-model
embeddings together with lightweight temporal
reasoning to incorporate contextual cues such as procedure type,
surgical phase, and short-term visual
memory. This design enables temporally consistent and accurate predictions while maintaining real-time feasibility. Together, the
dataset and model establish a practical foundation for robust
surgical scene understanding and support the development of clinically applicable guidance systems for minimally invasive surgery. The models,
dataset annotations and annotation platform are publicly available at: https://github.com/TimJaspers0801/ATLAS.
Read the full abstract ▾
All authors Ronald L. P. D. de Jong, Tim J. M. Jaspers, Raf A. H. Vervoort, Aron F. H. A. Bakker, Yiping Li, Jip L. Tolenaar, Jelle P. Ruurda, Willem M. Brinkman, Josien P. W. Pluim, Marcel Breeuwer, Daan de Geus, Fons van der Sommen
Computer Vision
Accurate identification and measurement of the precipitate area by two-stage deep neural networks in novel chromium-based alloys
Zeyu Xia, Kan Ma, Sibo Cheng et al. (+7 more)
The performance of advanced materials for extreme environments is underpinned by their microstructure, including the size and distribution of reinforcing phases. Chromium-based superalloys are a recently proposed alternative to conventional face-centred-cubic superalloys for high-temperature applications, such as Concentrated
Solar Power, and their development requires efficient measurement of precipitate volume fraction and size distribution from electron microscopy images. Traditional fixed-threshold image processing is sensitive to background noise, generalises poorly across materials, and requires substantial manual measurement effort. To address these bottlenecks, this study proposes DT-SegNet, an end-to-end two-stage deep learning scheme based on YOLOv5 and SegFormer for object detection and
segmentation in electron microscopy images. The approach combines the training efficiency of convolutional neural networks at the detection stage with the
segmentation accuracy of a
Vision Transformer. Numerical experiments show that DT-SegNet substantially outperforms state-of-the-art
segmentation tools offered by Weka and ilastik across metrics including accuracy, precision, recall, and F1-score. The model provides a useful tool for
alloy-development microstructure examinations and helps address the large
datasets associated with high-throughput
alloy development.
Read the full abstract ▾
All authors Zeyu Xia, Kan Ma, Sibo Cheng, Thomas Blackburn, Ziling Peng, Kewei Zhu, Weihang Zhang, Dunhui Xiao, Alexander J Knowles, Rossella Arcucci
Computer Vision · Cross-listed from cs.LG
OphthaDT: Generative Digital Twins for Forecasting Visual Acuity Trajectories in Ophthalmology
Pietro Belligoli, Nikita Makarov, Sayedali Shetab Boushehri et al. (+3 more)
Precision medicine in ophthalmology requires accurate longitudinal predictions, but the fragmented nature of
multimodal clinical data remains a barrier to
forecasting. We introduce OphthaDT, an LLM-based digital twin for ophthalmology that serializes longitudinal patient histories from 3,220 patients across four Phase III clinical trials into structured narratives to forecast best corrected visual acuity (BCVA). In
benchmarks spanning up to 100 weeks, OphthaDT demonstrated the lowest prediction error in neovascular age-related macular degeneration (nAMD), achieving an average mean absolute error (MAE) reduction of 6.0% compared to all baselines. In diabetic macular edema (DME), OphthaDT demonstrated competitive performance against all baselines while outperforming Random Forest and XGBoost by an average MAE reduction of 2.6% and 6.9%, respectively. Results reveal that OphthaDT's predictive advantage scales with trajectory complexity: whereas linear models remain effective for the more stable treatment responses of DME, OphthaDT's capacity is better suited for capturing the high longitudinal variability of nAMD. Finally, OphthaDT handles irregular sampling without imputation, positioning LLM-based clinical trajectory modeling as a methodology that could reduce patient burden and accelerate drug development.
Read the full abstract ▾
All authors Pietro Belligoli, Nikita Makarov, Sayedali Shetab Boushehri, Fabian Schmich, Raul Rodriguez-Esteban, Michael Menden
Computer Vision
Cross-View Yaw Estimation in Location Uncertainty with Line-Aligning Yaw Scoring
Taeho Kang, Nairan Zhang, Yelin Kim et al. (+2 more)
Accurate yaw estimation is a bottleneck in cross-view localization between ground view and Bird's Eye View (BEV). Existing methods couple yaw with translation and rely on height or projection assumptions that degrade under large yaw ambiguity. We disentangle yaw from location accuracy and introduce LAYS, a radially invariant line-consensus voting method. By exploiting the radial invariance of our formulation, we achieve sub-degree yaw precision via 3D voting over all candidate poses, while eliminating the need for accurate location. Our key observation is that a ground-image column matched to BEV pixels induces the same yaw across all camera positions along the radial direction of the pixels. LAYS matches BEV pixels to ground columns using feature similarity and accumulates the induced yaw votes into discrete 3D bins, where correct correspondences along the radial line concentrate into a sharp peak for the correct yaw. Experiments on Mapillary, Ford, KITTI, and VIGOR show significant gains under unknown yaw, particularly for normal FoV with unknown yaw (+28$\sim$45%p), and using LAYS as a yaw prior improves downstream 3-DoF localization.
Read the full abstract ▾
All authors Taeho Kang, Nairan Zhang, Yelin Kim, Yujiao Shi, Youngki Lee
Computer Vision
BAC-JEPA: Label-Efficient Breast Arterial Calcification Segmentation via Synthetic Mammography-Guided Supervision
Scott Chase Waggener and Lakshman Tamil
Breast arterial calcification (BAC) on screening mammograms is an emerging cardiovascular risk biomarker, but quantitative use requires reproducible
segmentation and expert pixel-level labels are costly. We present BAC-JEPA, a label-efficient
segmentation framework trained on procedurally generated arterial calcification inserted into real mammographic backgrounds with exact masks. Candidate backgrounds were selected from model-screened mammograms with low predicted BAC response; the generator samples arterial structure, disease burden, radiographic appearance, and hard-negative distractors including nonarterial calcifications and metallic objects. Synthetic masks are paired with mammography self-supervised
Vision Transformer encoders and a high-resolution convolutional decoder to produce full-resolution
segmentation maps. The study used 75,472 mammography studies from 34,956 patients for background selection and representation learning, trained on synthetic images from 10,000 backgrounds, selected checkpoints with 1,000 development backgrounds, and evaluated transfer on all 1,000 human-labeled BacSeg synthetic 2D mammograms. On held-out synthetic validation data, the larger backbone achieved IoU 0.5325 and Dice 0.6357. On BacSeg, image-level classification from
segmentation probability maps reached AUROC 0.8719, with 0.8547 for the smaller backbone. Four-view inference required 110.68–213.63 ms on an RTX 5090 GPU, and severe-preset synthetic image generation averaged 2.7071 s per image on a multicore workstation. These results indicate that BAC-specific synthetic supervision can produce useful image-level transfer without human pixel-level training masks, while expert-reviewed real-mammogram
segmentation remains necessary for clinical validation and calibration.
Read the full abstract ▾
Computer Vision
Morphology-Aware Multimodal Representation Learning for Insect Phylogenetic Reconstruction
Zixuan Liu, Kaijie Yu, Chun He et al. (+5 more)
Morphological traits provide important evidence for phylogenetic reconstruction and evolutionary relationship analysis. Recent image-based approaches have introduced deep learning, particularly convolutional models, to derive morphological features from specimen images, but these methods generally rely on single-modality visual representations and do not explicitly incorporate morphological semantics. This study proposes a morphology-aware
multimodal alignment framework for insect phylogenetic reconstruction. The framework combines specimen images with curated morphological descriptions by adapting a
vision transformer through parameter-efficient fine-tuning and supervised contrastive learning, followed by image-text
alignment in a shared latent space. The learned image
embeddings are then used as continuous traits for
Bayesian phylogenetic reconstruction. On the public Rove-Tree-11
dataset, comparative and ablation experiments across multiple visual backbones and feature adaptation strategies demonstrate that
multimodal alignment improves topological agreement with the reference phylogeny. The results indicate that the proposed framework can derive morphology-aware visual traits for computational phylogenetic reconstruction.
Read the full abstract ▾
All authors Zixuan Liu, Kaijie Yu, Chun He, Xiaoxu Cai, Xinhai Ye, Haishuai Wang, Gongyin Ye, Jiajun Bu
Computer Vision
Learning Cross-View Semantic Priors for Single-Reference Unseen Object Pose Estimation
Jiahong Chen, Jinghao Wang, Ziwen Wang et al. (+3 more)
Single-reference unseen object 6D pose estimation reduces object onboarding by estimating poses of arbitrary novel objects from only one reference view. Recent correspondence-based pipelines have achieved robust performance with
vision foundation model (VFM) features. However, they typically treat these features as intra-view descriptors, leaving dense visual-semantic cues, including appearance, structure, and context, insufficiently exchanged across views before
geometric decoding. Consequently, the decoded point features may lack joint semantic and
geometric discriminability, making correspondence estimation still difficult in challenging cases. Instead of processing features independently, we build the correspondence pipeline around an early cross-view semantic prior. Specifically, cross-view semantic interaction (CVSI) enables dense query and reference VFM
tokens to exchange semantic context and form a cross-view prior. Nevertheless, direct CVSI may disturb the VFM
token structure, while the resulting semantic prior still needs 3D representation consistency for rigid correspondence. To make this CVSI prior reliable for 3D correspondence learning, we introduce two complementary training-time constraints: the intra-view structure preservation (IVSP) loss preserves the original intra-view
token affinity structure during interaction, while the reference-anchored
geometric consistency (RAGC) loss enforces spatial representation consistency of decoded point features. The final pose is recovered from learned correspondences through weighted SVD. We further construct a challenging view-pair protocol from the BOP Challenge
datasets YCB-V and TUD-L to evaluate robustness in difficult matching scenarios. Extensive experiments on six
benchmarks under different view-pair settings show that our method achieves state-of-the-art performance while maintaining comparable inference speed.
Read the full abstract ▾
All authors Jiahong Chen, Jinghao Wang, Ziwen Wang, Zi Wang, Banglei Guan, Qifeng Yu
Computer Vision
A Controlled Study of CLIP-Based Body-Scene Fusion for Emotion Recognition in Context
Zubair Abbas, Muhammad Umair and Muqaddas Hameed
Apparent
emotion in natural images is often not visible from the face alone. The face may be small, hidden, or neutral, while posture and scene context carry much of the evidence. This work studies context-aware
emotion recognition on EMOTIC with an image-only two-stream model. A ResNet-18 body stream encodes the target-person crop, and a CLIP ViT-B/16 scene stream encodes the full image. The fused feature predicts 26 categorical
emotion labels and the continuous valence, arousal, and dominance values. This study examines whether small context-debiasing or rare-class training changes still help after adding a CLIP scene encoder. The clean two-stream model is compared with simplified CCIM-style intervention, CLEF-lite context-bias subtraction, ASL tuning, and class-balanced sampling under the same implementation pipeline. No tested variant improves over the clean two-stream model, which achieves 34.52% mAP on the EMOTIC test split. CLIP gives the model broad scene semantics, but the simplified causal, counterfactual, and rare-class changes do not automatically improve performance. Most remaining errors are in rare and subtle
emotion categories, so the next step should focus on label relationships and finer subject-context interaction.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
When Does a Video-Language Model Stop Watching? Reward Strength Controls the Formation and Reversal of Visual Shortcuts in Multimodal RLVR
Zekun Xu
Reinforcement learning with verifiable rewards (RLVR) is increasingly applied to large
vision-
language models (LVLMs), yet outcome-only optimization can drive a model to stop attending to the
video and instead exploit linguistic priors – a failure we call a visual shortcut. While the existence of such perception bypass is by now documented, how it forms, whether it can be undone, and when intervention still helps remain open. We treat the strength of a grounding penalty, lambda, as a control knob and characterize the formation-reversal dynamics of visual shortcuts along the training time axis. On a held-out, out-of-distribution diagnostic set, we find: (i) a sharp onset – shortcut reliance emerges abruptly over a narrow window of optimization steps and is robust across random seeds; (ii) a monotone dose-response – increasing lambda progressively suppresses the shortcut, and at an intermediate dose the trajectory first forms and then reverses the shortcut, exposing a hysteresis-like asymmetry between acquiring and removing it; and (iii) a critical intervention window – applying the penalty before onset arrests shortcut formation, whereas the same penalty applied after consolidation is markedly less effective. Together these results recast visual-shortcut collapse not as a binary defect but as a controllable, time-dependent, and asymmetric process, with direct implications for when and how strongly to regularize
multimodal RLVR.
Read the full abstract ▾
Computer Vision
IDAG-Edit: Multi-Object Video Editing via Instance-Decoupled Attention and Guidance
Yuan-Zhih Lin, Huu-Thang Nguyen, Huu-Phu Do et al. (+2 more)
Diffusion-based
video editing has made significant progress; however, achieving precise and temporally consistent object-level control, especially in multi-object scenarios, remains challenging due to
attention leakage, identity drift, and unstable temporal dynamics. In this work, we propose IDAGEdit, a training-free framework for fine-grained multi-object
video editing with strong temporal consistency. The framework adopts Layout-guided
Attention Modulation to facilitate coherent multi-object editing, while Instance-level Masks are introduced to preserve individual object identity and enforce localized
attention within each object region, thereby enabling fine-grained, object-level editing. Extensive qualitative and quantitative evaluations demonstrate that our method improves temporal stability and multi-object controllability over state-of-the-art
video editing approaches.
Read the full abstract ▾
All authors Yuan-Zhih Lin, Huu-Thang Nguyen, Huu-Phu Do, Hong-Han Shuai, Ching-Chun Huang
Computer Vision
Topological summaries of fingerprint ridge patterns carry identity information
Chad M. Topaz, Niny Arcila-Maya, Elizabeth Munch et al. (+2 more)
Fingerprints are the most widely deployed biometric. Verifying whether two impressions come from the same finger typically relies on minutiae, small landmarks such as skin ridge endings and bifurcations. These landmarks are extracted through a multi-stage pipeline of image enhancement, skeletonization, minutiae detection, and
alignment. We investigate an alternative: using topological data analysis to represent the full pattern of skin ridges and valleys directly, bypassing minutiae detection and the downstream matching pipeline. We apply persistent homology, a topological tool that tracks how loops in the ridge pattern form and fill in across spatial scales, producing multi-scale summaries of ridge geometry. We develop and compare a range of verification methods on a standard
benchmark dataset, FVC2000 DB1. Even the simplest topological summaries, with no trained parameters, substantially outperform geometry-only baselines. A trained method achieves an AUC of 0.91, while an optimal-transport method excels at the strictest false-accept thresholds, suggesting they capture different aspects of the ridge pattern. Fusing these two approaches yields the best performance at every low false-accept threshold we examine. Our results establish that these topological summaries capture substantial fingerprint identity information, far more effective for verification than raw pixel-level geometry. Because the entire pipeline is openly specified, it offers a transparent complement to minutiae-based systems, and we provide a modular framework for constructing, evaluating, and combining topological verification methods.
Read the full abstract ▾
All authors Chad M. Topaz, Niny Arcila-Maya, Elizabeth Munch, Zofia Stanley, Lori Ziegelmeier
Computer Vision
One-Shot Data Selection for Medical Image Classification via Graph Coverage
Zahiriddin Rustamov, Nadia Badawi, Rafat Damseh et al. (+1 more)
Training medical image classifiers on entire
datasets is wasteful when annotation budgets are limited: not all samples contribute equally, yet acquiring expert labels is expensive. Active learning reduces annotation cost through iterative querying, but assumes repeated access to an oracle and requires multiple rounds of model training. One-shot geometry-based methods such as facility location avoid retraining but operate on pairwise distances that ignore the local structure of the data
manifold. We propose a graph-based one-shot selection method that operates entirely on frozen foundation model
embeddings. Given
embeddings from a pretrained encoder, we construct a k-nearest neighbor graph over all training samples and derive a two-term coverage kernel from the heat
diffusion kernel, capturing both direct and two-hop neighborhood relationships. Greedy facility location on this kernel selects class-balanced subsets that maximize coverage of the data
manifold. The two-term kernel matches the full spectral heat kernel in selection behavior while reducing computation to sparse
matrix operations with a single hyperparameter. We evaluate on five MedMNIST
datasets spanning histopathology, radiology, and microscopy, comparing against both training-dynamics and geometry-based baselines. Our method achieves the highest balanced accuracy on nine of ten
dataset-ratio conditions, with the largest gains on class-imbalanced
datasets where global graph construction captures cross-class structure that per-class methods miss, all without any model training during selection. Code is available at https://github.com/zahiriddin-rustamov/graph-coverage-selection.
Read the full abstract ▾
All authors Zahiriddin Rustamov, Nadia Badawi, Rafat Damseh, Nazar Zaki
Computer Vision · Cross-listed from cs.SE
From Driving Videos to Simulatable Scenarios
Alexandre Levy, Ernest Valveny Llobet and Antonio Manuel López
Autonomous vehicles (AVs) face driving scenarios ranging from routine traffic to rare events. To assess safety it is crucial to reproduce these scenarios in a controllable, repeatable, and scalable manner, with simulation playing a key role. This paper introduces D-V2S, a novel framework that automatically generates simulatable driving scenarios from driving
videos. D-V2S operates in two stages: a Driving Record Analyzer (DRA) uses a
vision language model (VLM) with our designed
prompt to produce natural-
language descriptions from input
videos, capturing road layouts and dynamic traffic interactions; subsequently, a Scenario Generator (SG) uses a large
language model (LLM) and our conditioning context to translate these descriptions into executable scenarios. Using simulations, we show that D-V2S generates scenarios where 90% of the relevant semantic elements of the
videos are present. We also provide qualitative results demonstrating D-V2S's capability to transform real-world driving
videos into simulatable scenarios. Moreover, we provide both semantic and human driven ablative analyses of D-V2S's modules. In particular, we show how the VLM choice matters for DRA, and how our SG achieves a 75% preference rate over other state-of-the-art methods.
Read the full abstract ▾
Computer Vision
CoDMD: Copula-aware Distribution Matching Distillation for Fast Video Generation
Wenhu Zhang, Kun Cheng, Changyuan Wang et al. (+7 more)
Few-step
distillation for
video diffusion models has attracted significant
attention, driven by the urgent demand for efficient deployment in real-world scenarios. However, Distribution Matching
Distillation (DMD), a leading paradigm, tends to degrade under limited NFE budgets, manifesting in
video generation as layout instability, oversaturation, and broken motion dynamics. We trace this failure to a structural limitation: standard DMD is an intra-sample distribution-matching objective with coordinate-wise gradients, and thus imposes no explicit constraint on the relational geometry across batch elements or temporal frames, leaving the underlying copula largely unregulated. Combined with the mode-seeking tendency of its reverse-KL objective, this absence of relational guidance makes DMD prone to collapsing into local optima in the few-step regime. Motivated by this insight, we propose Copula-aware DMD (CoDMD), a lightweight relational regularizer that reuses score estimates already produced by the frozen teacher and the online fake model to construct pairwise relation
matrices across samples and frames. These are matched through a supplementary distributional objective that requires no additional networks,
datasets, or sampling trajectories. On the Wan-2.1-T2V model series at 1.3B & 14B scales, CoDMD distills 50-step teachers into 4-step students, achieving an approximate 25$\times$ speed-up while attaining VBench scores of 84.46 & 84.87, outperforming prior trajectory-based (rCM 82.81 & 84.05) and distribution-based (DMD 83.38 & 83.81) methods.
Read the full abstract ▾
All authors Wenhu Zhang, Kun Cheng, Changyuan Wang, Shiyao Li, Yuechen Zhang, Wenbo Li, Jiajun Zha, Jingyi Zhang, Kang Zhao, Jiaya Jia
Computer Vision · Cross-listed from cs.HC
Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems
Jingjing Jiang, Atsumoto Ohashi and Ryuichiro Higashinaka
Full-duplex spoken dialogue models, such as Moshi, enable natural, low-latency voice conversations. However, they remain limited to the audio modality, lacking the facial expressions that are integral to human communication. We present Moshi-Face, the first full-duplex dialogue model that jointly processes the user's audio and facial input while simultaneously generating
speech and facial motion. We first construct a vector-quantized variational autoencoder (VQ-VAE) as a face codec that encodes 3D head meshes extracted from facial
videos into compact discrete
tokens, referred to as face
tokens, and conversely reconstructs 3D meshes from these
tokens. We then extend Moshi with a Face
Transformer module that generates face
tokens non-autoregressively, enabling Moshi-Face to produce synchronized audio and face
tokens in real time. Experiments show that Moshi-Face achieves audiovisual
alignment at low latency while preserving the dialogue quality of the original audio-only model.
Read the full abstract ▾
Computer Vision
Look Before You Zoom: Adaptive Routing for the Resolution-Context Trade-off in Visual RAG
Oanh N. Tran, Thanh Quoc Hung Le, Oscar Chew et al. (+2 more)
Vision-
Language Models (VLMs) struggle as query-relevant objects become smaller. To address this, recent training-free approaches dynamically retrieve and zoom into local image regions. However, we show that indiscriminately applying
retrieval ignores a critical vulnerability: the resolution-context trade-off. Patch-based zooming recovers details for small targets, but can split large objects and destroy global spatial context;
attention-based
retrieval better preserves large objects, but remains less reliable on tiny details; and global perception is often fastest when
retrieval is unnecessary. Motivated by these failure modes, we introduce ViRGo (Visual
Retrieval or Global Perception), a lightweight framework that formulates visual
retrieval as an adaptive routing problem. ViRGo estimates object scale from the VLM's intrinsic localization heads during the initial forward pass and combines it with semantic
token confidence to select between global perception, patch-based
retrieval, and
attention-based
retrieval with minimal additional computation. Experiments across multiple VQA
benchmarks and object-size groups show that ViRGo improves the accuracy-efficiency trade-off: it matches patch
retrieval on small details, leverages
attention-based
retrieval for larger objects, and reduces inference time by routing to the global baseline when zooming is unnecessary.
Read the full abstract ▾
All authors Oanh N. Tran, Thanh Quoc Hung Le, Oscar Chew, Kuan-Hao Huang, Khoa D. Doan
Computer Vision
Denoising-Enhanced Coarse-to-Fine Infrared Small Target Detection with Attention Prior-Guided Knowledge Distillation
Houzhang Fang, Ruixuan Huang, Qiuhuan Chen et al. (+3 more)
Infrared small target detection (IRSTD) in high-resolution images is crucial for unmanned aerial vehicle (
UAV) surveillance and
UAV-based ground monitoring. However, small target size, weak features, and interference from complex dynamic backgrounds make IRSTD challenging. Existing methods incur redundant computation in non-target background regions and insufficiently exploit target context, limiting detection performance. To address these issues, we propose ECFNet, an efficient coarse-to-fine IRSTD framework with
attention prior-guided knowledge
distillation. In the coarse stage, we design a region binary classification network (RBCN) on grid-based multi-scale feature maps to efficiently identify target-containing context region proposals. A new denoising-assisted training strategy incorporates noisy ground-truth (GT) masks into RBCN feature maps and trains the network to reconstruct the original GT masks. This auxiliary task encourages explicit learning of target-background context to better distinguish target proposals from background regions. In the fine stage, we customize a lightweight target detector to the coarse-stage region proposals to balance accuracy and efficiency. Furthermore, we introduce a knowledge
distillation strategy guided by a teacher-student cross-
attention prior. This strategy directs the student to focus on critical target regions, enhancing discriminative feature representations for infrared small targets. Extensive experiments on three real infrared
datasets demonstrate that ECFNet outperforms existing single-stage and two-stage approaches while maintaining high real-time processing efficiency. Code: https://github.com/IVPLabs/ECFNet.
Read the full abstract ▾
All authors Houzhang Fang, Ruixuan Huang, Qiuhuan Chen, Xiaolin Wang, Yi Chang, Luxin Yan
Computer Vision
CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales
Xinlong Chen, Jiafu Tang, Yue Ding et al. (+12 more)
Accurate and comprehensive
video captions with consistent subject references are critical for downstream understanding and generation tasks. However, few existing
benchmarks can objectively and comprehensively evaluate these properties across diverse durations and scenarios, thereby hindering the advancement of
video captioning models. To bridge this gap, we propose CapRiCorn-1K, a comprehensive
benchmark designed to evaluate both
video captioning quality and subject referential consistency across long temporal horizons and diverse
video domains. To accommodate varied evaluation needs, our
benchmark supports both audiovisual and visual-only settings. Extensive experiments on CapRiCorn-1K reveal that current models generally struggle to generate accurate and comprehensive captions while maintaining consistent subject references. Moreover, as
video duration increases, both the overall caption quality and subject referential consistency decline. Notably, our evaluation metrics exhibit strong correlations with the performance of downstream understanding and generation tasks conditioned on the generated captions, further validating their effectiveness. The project is available at https://github.com/xlchen0205/CapRiCorn-1K .
Read the full abstract ▾
All authors Xinlong Chen, Jiafu Tang, Yue Ding, Yizhuo Jia, Bozhou Li, Bohan Zeng, Yang Shi, Shihao Li, Yiyan Ji, Qiang Liu, Weihong Lin, Yuanxing Zhang, Pengfei Wan, Liang Wang, Tieniu Tan
Computer Vision
ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers
Changjun Li, Runqing Jiang, Lian Xu et al. (+3 more)
Vision Transformers have achieved remarkable success in many fields, yet their deployment on edge devices remains challenging due to their substantial computational demands. Post-Training Quantization (PTQ) offers an attractive solution by compressing models using a small calibration set with minimal training overhead. However, most existing PTQ works adopt a static quantization paradigm that is uniformly applied to all instances. Given the substantial diversity of natural images, the activation distributions vary significantly across samples, making these methods inherently suboptimal. In this paper, we propose ScalePredictor, a dynamic quantization framework for accurate and efficient quantization scale learning of ViTs. We first reveal a hidden correlation between the distribution range of shallow-layer activations and the optimal scales of deeper layers. Based on this, we develop a scale learning mechanism that integrates an efficient range extraction approach to capture robust range statistics at the shallow stage, which are then fed into a Taylor-motivated
polynomial scale projection module to generate all quantization scales simultaneously. With the efficiency of
polynomial approximation, ScalePredictor introduces insignificant computational overhead while avoiding costly just-in-time calibration. Extensive experiments on ImageNet demonstrate that ScalePredictor consistently outperforms prior PTQ methods, achieving a more favorable accuracy-efficiency trade-off. Code and additional results are shown in the supplementary materials.
Read the full abstract ▾
All authors Changjun Li, Runqing Jiang, Lian Xu, Ye Zhang, Qingyong Hu, Yulan Guo
Computer Vision
Artic-O: End-to-End Articulated Object Reconstruction via Latent Geometry Learning
Xuyang Wang, Zhenyu Li, Jian Ding et al. (+4 more)
Reconstructing articulated objects from sparse images requires recovering complete geometry, movable parts, and motion parameters. Recent methods typically separate geometry reconstruction, part
reasoning, and articulation estimation into different stages. This separation can weaken consistency between shape, active parts, and motion, while also incurring substantial inference cost. We introduce Artic-O, an end-to-end, feed-forward framework for articulated object reconstruction via latent geometry learning. Instead of fitting geometry in image or view space, Artic-O maps sparse multi-state observations into a pretrained latent geometry space, where a frozen flow-matching decoder provides a complete-shape prior for recovering visible and occluded structures. To connect geometry with articulation, Artic-O fuses visual
tokens, geometry latents, and point-wise decoder features in an image-grounded part-
reasoning module for active-part
segmentation and articulation prediction. We further train the model with a geometry-to-articulation curriculum and a decoupled two-pass strategy to balance reconstruction and part-level supervision. On PartNet-Mobility, Artic-O achieves strong reconstruction quality while being substantially more efficient than LARM, a strong prior method. It reduces Chamfer Distance, improves F-score, and achieves comparable or better articulation accuracy across most joint metrics, while reducing inference time from 9 minutes to about 0.3 seconds per object.
Read the full abstract ▾
All authors Xuyang Wang, Zhenyu Li, Jian Ding, Habib Slim, Peter Wonka, Hongdong Li, Mohamed Elhoseiny
Computer Vision
CoSA: Correlation-Guided Change Attention with Learnable Residual Gating for Remote Sensing Change Detection
Abdirashid Omar and Jonghyuk Park
Remote sensing change detection (CD) from bi-temporal imagery is critical for applications such as urban monitoring, disaster assessment, and environmental management, yet robust localization remains challenging under sparse changes, noisy labels, and appearance variations. In this paper, we propose Context Sampling
Attention (CoSA), a lightweight decoder-side refinement module that explicitly leverages bi-temporal feature correlation as a control signal for adaptive change-aware feature enhancement. This differs from conventional
attention mechanisms that rely on implicit feature weighting without explicit temporal control. In the implemented FC-Siam setting, CoSA computes normalized same-location cross-correlation between paired decoder features, converts low correlation into a change gate, and injects the resulting gated residual at native 1/8 and 1/16 feature scales through learnable residual scaling. This design enables effective discrimination between stable and ambiguous regions without relying on computationally expensive global
attention. Extensive experiments on four
benchmark datasets (LEVIR-CD, S2Looking, DSIFN, and CLCD) demonstrate consistent improvements over strong baselines, achieving 1.5-2.6% gains in changed-class F1 while introducing negligible parameter overhead. Ablation studies confirm that multiscale placement and learnable residual gating are both important for peak performance. These results indicate that CoSA establishes a practical and effective refinement paradigm for enhancing temporal discriminability in Siamese change detection frameworks.
Read the full abstract ▾
Computer Vision
GTA-Net: Cooperative Game Theory for Vision-Language Alignment in Chest X-Ray Report Generation
Saif ur Rehman Khan, Imad Ahmed Waqar, Sebastian Vollmer et al. (+2 more)
Automated chest X-ray report generation requires precise cross-modal grounding to ensure clinically reliable descriptions. However, existing
vision-
language models rely on implicit
attention mechanisms that fail to enforce explicit region-word correspondence and disease-level consistency. We propose Game-Theoretic
Alignment Network (GTA-Net), a
vision-
language framework that formulates report generation as a cooperative game-theoretic
alignment problem. The model introduces a BinaryGameAligner that models interactions between image regions and text
tokens using similarity-based payoff
matrices with Shapley-inspired importance weighting. To enforce clinical semantics, we further develop a Disease-Aware Ternary Aligner, which captures joint interactions among images, reports, and structured disease concepts. GTA-Net combines a Swin-based visual encoder with a LoRA-adapted large
language model and is trained with a unified objective for generation and
alignment. Experiments on CheXpertPlus and IU-XRay demonstrate state-of-the-art performance across standard generation metrics and improved clinical consistency, highlighting the effectiveness of explicit game-theoretic
alignment for medical
vision-
language generation.
Read the full abstract ▾
All authors Saif ur Rehman Khan, Imad Ahmed Waqar, Sebastian Vollmer, Andreas Dengel, Muhammad Nabeel Asim
Computer Vision
Rethinking the Adaptation of Vision Foundation Models for Efficient Cell Segmentation
Qing Xu, Xiangjian He, Wenting Duan et al. (+2 more)
Cell segmentation is critical for computational pathology and biomedical discovery. While recent
Vision Foundation Models (VFMs) have demonstrated remarkable universal feature representations, unlocking their full potential for cellular imaging is currently bottlenecked by resource-intensive adaptation paradigms. Existing methods typically rely on fine-tuning heavy visual encoders, leading to extensive computational overhead and a dependency on large-scale annotations. To address this, we propose the EffiCell-Seg framework for highly efficient
cell segmentation without re-training the visual encoder. Our core insight is that pretrained VFMs intrinsically encode complementary structural priors: global saliency for localizing potential
cells, and local morphological patterns for delineating cellular structures. To harness these priors, we devise a
Cell Structure
Prompt Encoder (CSP-Encoder) that synthesizes semantic-aware saliency and principal morphological features from frozen VFM representations into explicit structural prior maps. Moreover, we propose a Synergistic Mask Decoder (SM-Decoder) that enforces contextual consistency by jointly predicting
geometric distance fields and semantic maps via mutual cross-guidance. Extensive experiments demonstrate that EffiCell-Seg outperforms state-of-the-art methods across diverse
cell imaging modalities while requiring only ~5M trainable parameters, over 130x fewer than fully fine-tuned VFM counterparts. The code is available at https://github.com/xq141839/EffiCell-Seg.
Read the full abstract ▾
All authors Qing Xu, Xiangjian He, Wenting Duan, Jiebo Luo, Zhen Chen
Computer Vision
Fidelity- and Perception-Aware Local Implicit Attention for Arbitrary-Scale Image Super-Resolution
Yu-Syuan Xu, Hao-Lun Sun, Hao-Wei Chen et al. (+2 more)
Arbitrary-scale image super-resolution (ASISR) aims to reconstruct high-resolution images from low-resolution inputs over a continuous range of upscaling factors. While traditional pixel-regression approaches often produce overly smooth results that lack realistic details, recent
diffusion methods can produce sharper and more realistic textures. However, these
diffusion techniques frequently introduce the risk of structural
hallucinations. To address these issues, we propose Fidelity- and Perception-Aware Local Implicit
Attention (FPLIA), a framework that effectively integrates fidelity-oriented features into a
diffusion pipeline to produce realistic and faithful reconstructions for ASISR. We introduce a Fidelity and Perception
Attention Module (FPAM), which applies both self-
attention and cross-
attention to fidelity-oriented and perceptual features to enhance representational capacity. To further exploit their complements, we design a Fidelity and Perception Select Module (FPSM) that adaptively selects the most representative features for RGB values prediction. We conduct extensive experiments to validate the effectiveness of these components. Both qualitative and quantitative results show that FPLIA delivers superior perceptual realism while maintaining reconstruction accuracy on standard ASISR
benchmarks. The source code is accessible at the following repository: https://github.com/XUSean0118/FPLIA.
Read the full abstract ▾
All authors Yu-Syuan Xu, Hao-Lun Sun, Hao-Wei Chen, Hsien-Kai Kuo, Chun-Yi Lee
Computer Vision · Cross-listed from cs.GR
Mesh2GS: White-Box 3DGS Construction via Plenoptic Sampling
Haoran Zhu, Youcheng Cai, Huangsheng Du et al. (+2 more)
3D Gaussian Splatting (3DGS) has emerged as a promising method for high-quality, real-time 3D reconstruction. To associate 3DGS with mesh representations, existing methods primarily focus on 3DGS-to-mesh reconstruction from multi-view images. In contrast, the problem of converting a mesh into 3DGS has received comparatively less
attention. Instead of relying on heuristic strategies that bind 3D Gaussians to the mesh, we propose a novel white-box 3DGS construction framework, termed Mesh2GS, which generates 3DGS directly from mesh geometry based on plenoptic sampling theory, achieving Nyquist-level performance for high-quality global illumination rendering. Firstly, we propose a plenoptic sampling guided 3DGS construction strategy that theoretically derives the minimum sampling rate of the sampled views and the distribution of 3D Gaussians. Second, we propose a novel 3DGS update procedure with albedo–shading decomposition for efficient global-illumination capture. Finally, we introduce a neural illumination enhancement module to handle non-Lambertian effects. Experimental results demonstrate that our method surpasses state-of-the-art baselines and is practically effective for both real-time shared rendering and non-Lambertian effects capturing specular highlights. The project code will be released upon acceptance.
Read the full abstract ▾
All authors Haoran Zhu, Youcheng Cai, Huangsheng Du, Jingyang Meng, Ligang Liu
Computer Vision · Cross-listed from cs.LG
AgroSense 2.0: Cross-Modal Transformer Fusion with Geospatial Raster Integration and Interpretable Multi-Task Learning for Precision Crop Recommendation
Vishal Pandey, Rishav Tewari and Ruzina Haque Laskar
Crop recommendation systems in precision agriculture have long suffered from a fundamental modality gap: visual soil characterization and chemical nutrient profiling are typically treated as independent inference problems, with fusion often reduced to late-stage feature concatenation. AgroSense~2.0 addresses this limitation through three architectural advances. First, we introduce continental-scale geospatial integration via a seven-band soil raster (india\_soil\_7bands.tif) spanning India, encoding Nitrogen, pH, SOC, Clay, Sand, Silt, and Bulk Density as $32\times32$ spatial patches, a modality entirely absent from prior work. Second, we replace naive feature concatenation with a cross-modal
Transformer fusion module, where tabular nutrient features attend over image representations via multi-head
attention, enabling richer inter-modal dependency modeling than shallow fusion. Third, we adopt a multi-task objective jointly optimizing soil classification and crop recommendation through a shared backbone, improving generalization via complementary cross-task signal. To enhance interpretability, we apply TreeSHAP to the tabular branch, revealing crop-conditioned nutrient sensitivity: humidity and rainfall emerge as the most influential features globally, while crop-specific profiles diverge meaningfully rainfall dominates rice, nitrogen and potassium dominate maize, and humidity and nitrogen dominate coffee. These explanations provide transparency into model decisions and surface both agronomically consistent patterns and
dataset-specific divergences worth further study. Together, these contributions establish AgroSense~2.0 as a more principled, interpretable, and geospatially grounded framework for precision agriculture.
Read the full abstract ▾
Computer Vision
Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation
Yanghui Song, Nanqing Liu, Haonan Yin et al. (+3 more)
Open-vocabulary semantic
segmentation (OVSS) in remote sensing images aims to segment categories beyond a fixed label space. Recent SAM 3-based methods provide a promising training-free foundation, yet three key issues remain: (1) a single class-name
prompt lacks sufficient semantic coverage for complex remote sensing categories; (2) expanding each category into multiple
prompts introduces redundant online text encoding; and (3) directly aggregating multiple
prompt responses propagates noisy activations into the final prediction. To address these issues, we propose ProC-SAM3, which calibrates SAM 3's
prompt interface for remote sensing OVSS from three complementary aspects. First, we construct an offline
prompt pool where a Category Matcher groups MLLM-generated candidates into per-category sets, and Expansion Constraints further refine each set using category-specific prior knowledge. Second, the resulting text
embeddings are cached and reused across all test images, eliminating repeated text encoding. Third, we introduce Presence-Guided Residual Fusion to gate unreliable decoder outputs by
prompt presence and confidence, followed by peak-preserving class aggregation that retains fine-grained activations for small and sparse objects. Experiments on eight
benchmarks show that ProC-SAM3 achieves an average mIoU of 56.1%, outperforming the previous best training-free method by 3.9 percentage points. Code will be available at https://github.com/YanghuiSong/ProC-SAM3.
Read the full abstract ▾
All authors Yanghui Song, Nanqing Liu, Haonan Yin, Yingjie Gao, Chengfu Yang, Qi Ming
Computer Vision
Zero-Shot Vision-Language Models for Classroom Engagement Recognition: A Benchmark Study of Prompt Sensitivity and Cross-Dataset Generalization
Aman Goyal, Kshama Nitin Shah and Kemmannu Vineet Venkatesh Rao
Automated classroom engagement recognition holds substantial promise for scalable learning analytics, yet the suitability of modern
Vision-
Language Models (VLMs) for this task under zero-shot conditions remains largely unexplored. We present a systematic
benchmark that evaluates five widely-used VLMs: CLIP, BLIP-VQA, GPT-4o, LLaVA-1.5-7B, and Qwen2.5VL-7B-Instruct across two complementary educational
datasets: DAiSEE, an individual-student
video dataset (300 sampled test clips), and the Student Classroom Behaviour
dataset (SCB, 1,168 scene-level images). Each model is probed with three
prompt variants spanning minimal, rubric-anchored, and chain-of-thought designs. Our experiments reveal three primary failure modes of zero-shot VLMs for engagement recognition: (1) near-random performance on individual students, with Cohen's kappa never exceeding 0.10 on DAiSEE; (2) severe class collapse, where models assign 85-100% of predictions to a single engagement level regardless of visual content; and (3) extreme
prompt sensitivity, with accuracy swings of up to 32 percentage points on identical images depending solely on
prompt phrasing. Remarkably, scene-level classification on SCB is substantially more tractable: CLIP and GPT-4o achieve kappa approximately 0.60 when prompted with behaviorally-grounded rubrics. We also document a practical barrier for deployment: GPT-4o's safety filters reject 98% of chain-of-thought requests involving individual student faces. Our findings provide a calibrated baseline and surface critical design considerations for the use of VLMs in educational observation systems.
Read the full abstract ▾