Computer Vision
SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation
Jinsheng Quan, Jianhua Li, Siyi Xie et al. (+7 more)
Spatial perception and
reasoning from visual observations require recovering
geometric structure, establishing correspondences, and understanding spatial relations. Existing approaches typically address these capabilities separately using task-specific architectures or external
geometric modules, limiting knowledge transfer among complementary representations of the same physical scene. We introduce SPARGen, a unified
multimodal framework that casts 3D reconstruction, dense correspondence, and spatial
reasoning as instruction-conditioned generation tasks. SPARGen serializes compact structured and linguistic outputs as
token sequences while generating dense
geometric fields in image-aligned forms, enabling spatial supervision to jointly shape shared representations within a native
multimodal generative model. Experiments across
benchmarks for 3D reconstruction, correspondence, and spatial
reasoning show that SPARGen achieves competitive performance across heterogeneous spatial tasks within a single native
multimodal generative framework.
Read the full abstract ▾
All authors Jinsheng Quan, Jianhua Li, Siyi Xie, Xuanke Shi, Kewang Deng, Zukai Chen, Feifei Shao, Lei Yang, Quan Wang, Yawei Luo
Computer Vision
HiCo-GS: Hierarchical Context Aggregation and Geometric Consistency for Octree Gaussian Splatting
Wei Zhang, Shengkai Yu, Shiqiang Gong et al. (+3 more)
Octree-based anchor Gaussian Splatting has emerged as a scalable representation for city-scale novel view synthesis, where multi-level anchors adaptively capture scene content from coarse building structures to fine architectural details. However, we identify a fundamental limitation in existing methods: cross-level feature isolation, where each level's anchor features are optimized independently with no inter-level communication, causing color drift on building facades and over-smoothing in textured regions. We present HiCo-GS, a high-fidelity reconstruction framework with two complementary modules. Cross-Level Context Aggregation (CLCA) enables bidirectional hierarchical prior injection by leveraging the octree's spatial containment structure to aggregate per-level context vectors into parent-self-child triplets, fused via a lightweight MLP with residual connection. Coarse-level structural priors flow down to inform fine-level anchors, while fine-level detail statistics feed back to prevent over-smoothing, at negligible computational overhead. Depth-Normal
Geometric Consistency (DNGC) regularization enforces agreement between rendered normals and depth-derived normals through an alpha-weighted consistency loss, complemented by edge-aware smoothness losses with progressive warmup that exploit the strong planar priors ubiquitous in urban geometry to suppress floating artifacts. We further introduce the China-Pagoda
dataset comprising 8 ancient Chinese pagodas with over 1,200 images each, featuring dense ornamental carvings, curved multi-layer eaves, and repetitive fine-grained textures. Extensive experiments on Mill19, UrbanScene3D, MatrixCity, and China-Pagoda demonstrate that HiCo-GS achieves state-of-the-art rendering quality and substantially cleaner geometry across real-world and synthetic urban
benchmarks.Code: https://github.com/WZ-CS/HiCo-GS.
Read the full abstract ▾
All authors Wei Zhang, Shengkai Yu, Shiqiang Gong, Qi Zhang, Qiang Li, Qi Wang
Computer Vision · Cross-listed from cs.MM
AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relations
Ying Huang, Wencan Zhang and Brian Y. Lim
Computer
vision models for generated facial content, such as face editing and
privacy protection, increasingly affect people, requiring similarity metrics that serve as faithful proxies for human perception. While perceptual evaluation has progressed from signal-based heuristics to representation-based metrics, current approaches are limited to behavioral modeling without cognitive
alignment. They rely on implicit and spurious relations while assuming a universal observer, failing to account for inherent variations across diverse human populations. This leads to inaccurate evaluative models of stakeholders and misleading guidance for generative model debugging. Rather than treating perception as a black box, we leverage scientific findings from cognitive psychology of human face similarity perception: dependence on facial featural and configural attributes, nonlinear psychophysical response scaling, and own-group biases. We introduce the FACETS
dataset and propose AlignFace, an interpretable, human-aligned, face similarity metric that encodes these cognitive principles through ante-hoc modeling. It employs visual-
language modeling (VLM) to encode paired face images and text-based attributes, gated cross-
attention (CA) to extract attribute-specific facial difference representations, concept bottleneck modeling (CBM) to constrain
reasoning via interpretable face attributes, and neural generalized additive model (GAM) to model their nonlinear influence. Experiments found AlignFace significantly improves
alignment with human subpopulation perceptions compared to baseline metrics, including recent domain-free learned perceptual metrics. By bridging learned representations and human cognitive processes, this work enables more transparent and aligned perceptual evaluation metrics for face images.
Read the full abstract ▾
Computer Vision
Fixed-Budget Gaussian Volume Encoding with Structure-Aware Allocation
Michael R. Martin, Joseph Insley, Victor A. Mateevitsi et al. (+2 more)
Scientific simulations often produce scalar volumes faster than they can be stored, transferred, and loaded, while in situ reduction must use only a limited share of simulation resources. This work encodes scalar fields as anisotropic Gaussian primitives under a fixed budget. The complete primitive set is allocated analytically from local field structure, including position, orientation, and shape, then refined directly against the scalar field without densification,
pruning, or count changes. The selected budget determines encoded storage before refinement and, together with the iteration schedule, provides a controllable refinement-time budget. In a controlled
benchmark, truncation-aware field evaluation reduces encoding time by up to 51x; 1.4 million Gaussians encode a billion-voxel volume in at most four minutes on one desktop GPU, with reduced-iteration refinement completing in under one minute. Across five
datasets spanning 2.1 million to 1.1 billion evaluated voxels, compression-useful configurations achieve 15.0-38.7 dB PSNR at compression ratios from 2.2x to over 40,000x. Pre-encoding structure statistics characterize fields for which one-shot allocation yields limited gains from additional capacity. Because primitives retain scalar attributes rather than baked appearance, a single compact model serves every subsequent visualization state - supporting post-hoc transfer-function, colormap, lighting, and viewpoint changes without re-encoding.
Read the full abstract ▾
All authors Michael R. Martin, Joseph Insley, Victor A. Mateevitsi, Silvio Rizzi, Kwan-Liu Ma
Computer Vision
CoDS: Robust Collaborative Perception via Expert-driven Detection and BEV Segmentation
Jinlong Wang, Yuang Jia, Junhong Lin et al. (+2 more)
Collaborative perception breaks through single-view limitations via multi-
agent information exchange. However, multi-source noise such as pose errors and communication delays degrades fusion feature quality, constraining perception performance. Joint training of detection and BEV
segmentation provides a natural remedy, where segmented road regions help constrain target distributions and detection bounding boxes help recover ambiguous
segmentation boundaries. To this end, we propose a robust Collaborative perception framework with expert-driven Detection and bev
Segmentation (CoDS). To address spatial inconsistency in fusion quality, we first introduce the Collaborative Reliability Map (CoRM) to explicitly quantify feature quality distribution. Based on CoRM, we design the Semantic Mixture-of-Experts (S-MoE) module to extract differentiated features for inconsistent feature demands. Finally, to further mitigate feature noise degradation, the Bidirectional Task Complementary Interaction (BTCI) refines task-aware features through bidirectional injection. Extensive experiments on OPV2V and V2V4Real
datasets show that our CoDS surpasses existing baselines on both tasks and maintains stable robustness under multi-source noise. Code: https://github.com/JinlongW128/CoDS and https://openi.pcl.ac.cn/OpenAIDriving/CoDS.
Read the full abstract ▾
All authors Jinlong Wang, Yuang Jia, Junhong Lin, Nannan Li, Wei Gao
Computer Vision
Owner3D: Ownership-Guided Style Writing for Training-Free Localized 3D Stylization
Suchang Tao, Kaifeng Shi, Zhiyan Liu et al. (+2 more)
Localized 3D stylization aims to modify the appearance of a specified object part while preserving the remaining surfaces. In large reconstruction models (LRMs), this task is challenging because style is injected into intermediate appearance representations before rendering, while compact triplane features are shared across target and non-target surfaces, causing style leakage and boundary ambiguity. We propose Owner3D, a training-free framework for localized 3D stylization that integrates localized appearance control directly into the LRM reconstruction process. Specifically, Owner3D introduces ownership-guided style writing to restrict reference-style injection to target regions, producing a single localized stylized triplane without additional training while avoiding separate global style and appearance representations. To resolve appearance ambiguity near semantic boundaries, we further introduce boundary dual slots that maintain separate local feature sources for target and non-target regions. Finally, a surface-first texture readout hierarchically combines surface, 3D, and triplane ownership evidence to robustly recover appearance under incomplete visibility. Experiments on a
benchmark constructed from Google Scanned Objects and PartNet demonstrate that Owner3D consistently outperforms existing 3D stylization methods in target-region style fidelity and non-target appearance preservation, reducing appearance leakage by 86.4% and 89.9% compared with StyleSplat and LAENeRF, respectively.
Read the full abstract ▾
All authors Suchang Tao, Kaifeng Shi, Zhiyan Liu, Zhuoyuan Jiang, Yuqi Ouyang
Computer Vision · Cross-listed from cs.AI
A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images
Jennifer D'Souza, Fahad Ahmed, Cecilia Andrea Bustamante Andrade et al. (+9 more)
Scientific figures and tables encode essential experimental evidence, yet remain difficult for digital libraries and
multimodal AI systems to retrieve and interpret. The ALD/E-ImageMiner
benchmark and ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching Scientific Figures provide 1,951 figures from 205 publications, expert-annotated for classification, data table extraction, summarization, and visual question answering. In these companion proceedings, we present a forward-looking perspective on how the
benchmark can guide future scientific-image challenges. We examine how its tasks probe capabilities from visual and quantitative reading to domain-grounded
reasoning and evidential justification, and how Bloom-informed question design can support deeper scientific understanding. We propose "scientific conceptual understanding from images" as a long-term
benchmark objective, with future directions including broader domains and figure types, contextual and cross-document synthesis, hypothesis evaluation, provenance, uncertainty, counterfactual grounding, and open-ended
multimodal research. This perspective connects the ICDAR 2026 challenge to a broader agenda for machine-actionable scientific visual knowledge and verifiable
multimodal scientific AI.
Read the full abstract ▾
All authors Jennifer D'Souza, Fahad Ahmed, Cecilia Andrea Bustamante Andrade, Lina Frolova, Poorani Gnanasambandan, Dilshad Hussain, Muhammad Uzair Khan, Nkembeng Kevin Nkengfoa, Paul Praveen J., Fabio Priante, Sjoerd Franciscus van der Werf, Thomas Frederik Jan van Roeden
Computer Vision
InstructVVT: Instruction-Driven Video Virtual Try-On without Auxiliary Spatial Priors
Dingbao Shao, Song Wu, Xinyu Chen et al. (+18 more)
Video virtual try-on is a highly constrained editing task requiring the precise replacement of a target person's clothing while strictly preserving the original
video's spatial structure and temporal dynamics. Existing methods heavily rely on auxiliary handcrafted spatial priors (e.g., masks, poses) for editing control. However, these priors are prone to failure in unconstrained real-world
videos and often compress rich visual context into incomplete structural signals. Furthermore, standard reconstruction objectives fail to fully capture try-on-specific human preferences. To address these challenges, we propose InstructVVT, an instruction-driven and reference-guided
video virtual try-on framework based on a
Diffusion Transformer (DiT) that operates without inference-time spatial priors. Our core insight is to recover fine-grained control directly from the input triplet (source
video, reference garment, and instruction) via a dual-level reference conditioning scheme. Specifically, an MLLM infers semantic edit
tokens for target disambiguation and structural preservation, while a lightweight conditioning pathway explicitly injects fine-grained visual garment details. Finally, we design a try-on-specific reward and utilize the DiffusionNFT
algorithm to align the model with human preferences. Extensive experiments on ViViD-S and TripVVT-Bench demonstrate that InstructVVT outperforms state-of-the-art open-source methods in garment fidelity, structural preservation, and temporal consistency, despite requiring fewer inference-time controls.
Read the full abstract ▾
All authors Dingbao Shao, Song Wu, Xinyu Chen, Qian Wang, Jiahang Li, Kuai Jiang, Jiang Lin, Yuhang Liu, Ziyu Chen, Duo Li, Jiaxin Hu, Shengrong Gu, Ziheng Tang, Rongrong Liu, Yanlun Peng, Liang Li, Junlan Feng, Lujia Jin, Ting Zhang, Jian Yang, Zili Yi
Computer Vision
Voxel-based 3D Facies Segmentation from Seismic Data: A Comparative Study
Duc-Thanh Pham, Minh-Tan Pham, Anh Nguyen et al. (+1 more)
Seismic facies
segmentation has emerged as a significant challenge in geophysics, requiring robust methods and systems to effectively identify geologically analogous facies with limited labeled data. Although existing studies have shown promising results in 2D facies
segmentation, they often preprocess the original 3D seismic volumes into sets of 2D slices, typically the inline and crossline directions, and treat this problem as a purely 2D
segmentation task. This simplification introduces discontinuities across slices and fails to preserve the spatial and structural continuity in 3D seismic data, thus limiting the model's ability to learn coherent geological patterns. In this work, we present a comparative and reproducible
benchmark for voxel-based 3D seismic facies
segmentation, built upon publicly available seismic volumes including the Netherlands F3 and the Parihaka
datasets, with standardized data splits and evaluation metrics. By evaluating the three representative families of modern 3D
segmentation architectures, we establish strong baseline results that highlight the potential and remaining challenges for future research in this domain.
Read the full abstract ▾
All authors Duc-Thanh Pham, Minh-Tan Pham, Anh Nguyen, Van Nguyen
Computer Vision
Discovery and Spatial Characterisation of Multiple Shortcut Groups for Auditing Vision Model Bias
Akshit Achara, Vishnunarayan Manickam, Thomas Day et al. (+3 more)
Deep learning models trained on
datasets with spurious correlations can achieve high average accuracy whilst relying on shortcut features that do not generalise out of distribution. Whilst out-of-distribution testing highlights subgroup performance disparities arising from shortcut learning, it does not localise the regions within images that are associated with it. Existing research mostly uses attribution maps from interpretability methods to understand the spatial nature of spurious correlations. For example, conditional
alignment methods separate task-relevant evidence from evidence tied to spurious correlations by comparing attribution maps from a task model, a sensitive attribute model, and a bias-reduced reference model. This yields shortcut-aligned and task-aligned contribution maps for each image. However, existing methods aggregate these maps across the
dataset, potentially masking recurring spatial shortcut patterns that occur only in subsets of images. We address this limitation by grouping per-image shortcut and task contribution maps into recurring spatial patterns using K-means and non-negative
matrix factorisation, and visualising the resulting shortcut groups through contribution maps and representative examples. Across CelebA, CheXpert, Waterbirds, Camelyon17, and ISIC2019, and across ResNet and ViT models, the discovered shortcut groups reveal both shared and distinct spatial patterns of shortcut and task contribution, with varying subgroup composition and error rates, enabling targeted inspection of image subsets with higher error rates. We perform input occlusion and internal test-time interventions to show that masking or suppressing task contribution regions substantially degrades the model classification performance and propose a combined shortcut suppression and task amplification feature intervention approach which generally reduces performance disparities.
Read the full abstract ▾
All authors Akshit Achara, Vishnunarayan Manickam, Thomas Day, Esther Puyol Anton, Alexander Hammers, Andrew P. King
Computer Vision · Cross-listed from cs.RO
Evolve Vision-Language-Action Model into an Agent with On-the-fly Tool-use
Ding Yi, Yanzhao Yu, Xili Dai et al. (+5 more)
This paper integrates end-to-end Visual-
Language-Action (VLA) models with agentic tool-use to propose Agentic
Robot with Tool-use (ART). ART is a tool-injection framework that tunes any VLA model to leverage off-the-shelf tool modules for low-level
vision, high-level affordance, and embodiment enhancement. Compared to vanilla VLA models with a whole continuous action solution space, ART reduces the complexity of the action solution space through tool-use, which not only improves generalizability across different tasks but also reduces data dependency. To demonstrate the advantages (high generalizability and low data dependency) of this framework, we first built a
dataset of 30K tool-use trajectories and action demonstrations, which is much smaller than those used by baseline methods. We then designed a training regimen for long-trajectory tool-use
reasoning in challenging environments. Experiments show that ART achieves a 20% higher success rate than mainstream baselines on simulation and real-world tasks, such as pick-and-place in the dark at novel viewpoints. Empirical results highlight the benefits of an
agent-based approach: modular tool utilization enables more efficient training, lightweight deployment, and scalable integration of new tools. This design fosters robustness, adaptability, and extensibility, paving the way for the practical deployment of VLA systems in complex real-world scenarios.
Read the full abstract ▾
All authors Ding Yi, Yanzhao Yu, Xili Dai, Xianbiao Qi, Peiwen Sun, Xueqian Wang, Xiangyu Yue, Jianan Wang
Computer Vision
Source-Agnostic Image Translation Based on Latent Aware Adaptive Masking
Tomislav Dobrički and Byung-Woo Hong
In this work, we propose a source-agnostic framework that dynamically refines a binary mask throughout the reverse
diffusion process by computing the discrepancies of a pretrained
diffusion model's prediction for each latent time step. Rather than relying on a fixed threshold, our method introduces a time-dependent statistical thresholding scheme derived from the empirical mean and standard deviation of prediction discrepancies across the latent noisy images from the target distribution. This allows the mask to adapt to the model's varying predictive confidence at different noise levels, effectively isolating domain-specific regions while preserving global structural coherence. Experimental results on the AFHQ and Celeba-HQ
datasets demonstrate that our approach outperforms state-of-the-art unsupervised Image-to-Image methods in both realism (FID, KID) and faithfulness (SSIM, LPIPS). By requiring only a pretrained model of the target domain, our approach enables precise, automated localization and seamless translation across diverse source distributions without any specialized training. The project source code is available at: https://github.com/dtoma95/PM-Edit
Read the full abstract ▾
Computer Vision
Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation
Yanbo Ding, Yijia Fan, Caihua Shan et al. (+8 more)
Diffusion Transformers (DiTs) have become the dominant paradigm for high-fidelity
video generation, yet their ability to perform high-level semantic planning remains limited. While hybrid architectures integrating MLLMs with
diffusion backbones have shown strong advantages in image synthesis, such designs remain underexplored in
video generation, where existing approaches often treat MLLMs primarily as frozen feature encoders rather than semantic generators. To fill this gap, we systematically study how an MLLM should be integrated with a DiT for
video generation by answering three questions: what intermediate representation should bridge the MLLM and DiT, how the MLLM should generate it, and how the DiT should incorporate it during
diffusion rendering. Our analysis reveals three key findings: (1) discrete semantic visual
tokens produced by an EMA-based tokenizer provide a stable and expressive interface, (2) autoregressive causal modeling is effective for generating these
tokens, and (3) explicit visual-
token conditioning is more effective than
prompt refinement or latent bridging. Based on these findings, we propose BiVidGen, a hybrid framework where an MLLM first generates semantic visual
tokens and a DiT renders
videos conditioned on both text and these
tokens via multi-layer cross-
attention. Extensive experiments show that BiVidGen improves semantic
alignment and temporal coherence over a fine-tuned DiT baseline, achieving stronger performance on VBench-Long. These results demonstrate that explicit MLLM-based visual planning provides an effective intermediate interface for text-to-
video generation beyond text-only conditioning.
Read the full abstract ▾
All authors Yanbo Ding, Yijia Fan, Caihua Shan, Yifan Yang, Yifei Shen, Weijie Wang, Xirui Hu, Dongsheng Li, Lili Qiu, Yuqing Yang, Yali Wang
Computer Vision
E-S2Feat:Semantic-Guided Spiking Local Feature Detection and Description for Event Cameras
Yang Yi, Juntao Hua, Jinpu Zhang et al. (+3 more)
Benefiting from high temporal resolution and dynamic range, event-based local feature methods have attracted increasing
attention. However, event sparsity, noise, and limited texture still hinder robust local feature learning. Deploying such methods on resource-constrained platforms such as unmanned aerial vehicles also requires balancing accuracy and energy efficiency. To address these challenges, this paper proposes E-S2Feat, a spiking neural network framework for event-based local feature detection and description. The framework jointly optimizes local feature learning from the perspectives of feature representation and selection. First, a module-specific spiking activation mechanism preserves fine-grained structural cues and discriminative information under low-bit, energy-efficient inference, thereby improving overall representation fidelity. Furthermore, a semantic-guided feature modulation mechanism leverages semantic priors to refine keypoint response distributions and enhance local descriptor discriminability, thereby guiding the model to extract local features with greater
geometric stability and stronger discriminative capability. Experiments on the ECD and EDS
datasets show that the proposed method significantly outperforms baseline methods such as SuperEvent in pose estimation accuracy. It also achieves accuracy comparable to its artificial neural network counterpart while delivering an approximately 4.8-fold improvement in theoretical computational energy efficiency. Visual-inertial odometry experiments on the TUM-VIE
dataset further verify the effectiveness and practical application potential of the proposed method in complete SLAM systems.
Read the full abstract ▾
All authors Yang Yi, Juntao Hua, Jinpu Zhang, Liangwei Fan, Hui Shen, Dewen Hu
Computer Vision
SSP: An Event-Matched Syn2Sim2Phy Cross-Domain Evaluation Framework for Autonomous Driving VLA Models
Haojie Feng, Peizhi Zhang, Xinrui Zhang et al. (+7 more)
Vision-
language-action (VLA) models for autonomous driving jointly produce scene interpretation,
language-based
reasoning, and driving trajectories. Existing evaluations often use independently selected synthetic, simulated, and physical data, so measured performance gaps can be confounded by changes in scenario content rather than genuine domain sensitivity. We propose SSP (Synthetic-Simulation-Physical), an event-matched Syn2Sim2Phy evaluation framework that anchors cross-domain comparison to the same safety-critical interaction. Starting from a synthetic long-tail
video, SSP builds a validated event specification that preserves road
topology, participant roles, relative motion, conflict
evolution, passing order, response constraints, and event phases. Platform-specific realizations are then constructed in CARLA and on a closed proving ground and are evaluated only after transfer audits confirm preservation of mandatory event properties. SSP maps heterogeneous outputs from OpenEMMA, LLaViDA, and Alpamayo-R1 into common semantic slots and a 1 s trajectory window to assess output validity, semantic accuracy, critical-interaction recognition, trajectory quality, and risk response. Across Cut-in and vulnerable-road-user crossing cases, the macro-averaged Integrated VLA Capability Scores are 0.259, 0.291, and 0.325 in the Synthetic, Simulation, and Physical domains, respectively, while the best domain varies by scenario. Alpamayo-R1, OpenEMMA, and LLaViDA obtain scores of 0.405, 0.338, and 0.131. SSP provides a reproducible scene-transfer chain and an evidence-qualified evaluation of VLA behavior without assuming that the Physical domain is universally superior.
Read the full abstract ▾
All authors Haojie Feng, Peizhi Zhang, Xinrui Zhang, Zhuoren Li, Junpeng Huang, Xiurong Wang, Dongxiao Yin, Yuxiang Zhang, Junfan Zhu, Lu Xiong
Computer Vision
ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Models
Xinye Li, Lingshuai Lin, Lei Wang et al. (+8 more)
Action-conditioned
video world models require low-latency causal generation and reliable responses to game-native controls. Although causal
distillation enables one- or few-step
video synthesis, extending it to interactive world models remains challenging, as discrete keyboard states and continuous mouse motion must remain aligned with temporally compressed latent chunks during causal training and autoregressive rollout. We introduce ForgeWM, a progressive framework that transforms a bidirectional action-conditioned
video generator into efficient few-step world models through domain adaptation, teacher-forced causal training, causal consistency
distillation, and on-policy distribution matching with a bidirectional teacher. The resulting budget-specialized students operate at steady-state denoising budgets of 1, 2, and 4 steps. ForgeWM further supports a dual-path deployment protocol combining latency-critical interaction with optional replay-time refinement, where the one-step student re-noises and refines its saved draft. On paired Minecraft trajectories, ForgeWM leads the evaluated systems in Imaging Quality, reference-aligned motion-profile agreement, action-sign accuracy, and mouse-control accuracy, while achieving the lowest reference LPIPS; the same four-stage recipe transfers to gamepad-controlled FPS gameplay. Replay-time refinement matches four-step reference quality while remaining roughly three times closer to the experienced trajectory than regeneration from noise. These results demonstrate ForgeWM's effectiveness for controllable few-step
video generation.
Read the full abstract ▾
All authors Xinye Li, Lingshuai Lin, Lei Wang, Liuzhou Zhang, Jialin Cui, Qingshan Li, Guanchu Wang, Qingbin Liu, Xi Chen, Jiang Bian, Wai Lam
Computer Vision
Content Based Video Narration of Gameplay with Vision Language Models
Mathew Varghese
Live game commentary is scarce: it exists for professional esports broadcasts and almost nowhere else. We present a content-based
video narration system that produces spoken, esports-style commentary for arbitrary gameplay recordings using a general-purpose
vision-
language model (VLM) and a text-to-
speech back end, with no game-specific instrumentation, no engine telemetry, and no task-specific training. Three mechanisms carry the system. Temporal mosaic packing arranges nine uniformly sampled frames into a single 3x3 image, letting an image-native VLM reason about motion while consuming one image payload per segment instead of nine. Context-conditioned prompting replays the K most recent narrations as assistant-role history, suppressing the repetition that dominates per-segment captioning of static scenes. Duration-conditioned generation and elastic
alignment constrain narration length in the
prompt, then time-scale or symmetrically pad the synthesized audio so each utterance fills its segment slot exactly, giving frame-accurate muxing without a forced aligner. The implementation supports either cloud TTS or a 6-bit quantized 4B-parameter on-device TTS model on Apple silicon, making the
speech stage fully local. We report a qualitative case study on real-time strategy footage, a cost model showing the mosaic reduces per-minute image payloads by 9x, and a candid account of observed failure modes - hallucinated game state, resolution loss from mosaicking, and prosody artifacts from time-scaling. We release the system as a reproducible baseline, with an evaluation protocol for the quantitative study a full version will report.
Read the full abstract ▾
Computer Vision
Yingying Fan, Penghui Du, Leyan Zhu et al. (+10 more)
Understanding tens-of-minutes
surgical videos requires long-horizon temporal
reasoning, answering what happens before, after, or across stages of a procedure by grounding the question in visual evidence spread across time. Existing approaches handle this poorly: a one-shot
vision-
language model (VLM) compresses the whole procedure to fit its context window and loses the detail a "before" or "after" question depends on, while
video agents that train the model where to look are data-hungry and transfer poorly to out-of-domain surgery. We build an
agent harness that separates
reasoning from perception and improves by evolving context rather than optimizing weights. A text-only orchestrator plans which evidence to gather and issues an auditable sequence of tool calls, while frozen
vision-
language sub-
agents execute each call over the pixels, viewing, cropping, inspecting frames, and retrieving external knowledge. We further propose a gradient-free, reward-gated Heuristic Skill
Distillation loop that mines the
agent's own low-scoring traces and keeps a candidate skill only when it raises a validation reward, yielding reusable
retrieval skills, notably directed re-look. Growing an external skill library rather than tuning weights, the loop adapts from only about 100 labeled examples, far fewer than supervised or reinforcement fine-tuning requires. To evaluate this
agent, we introduce MedClawBench, a de-leaked, doctor-grounded
benchmark of 1,123 questions over self-built long neurosurgery recordings and a held-out public lecture-
video test split. Across both
datasets and all four evaluation dimensions, our
agent consistently outperforms one-shot VLMs and general
video-
agent frameworks, with the largest gains on the long, out-of-domain neurosurgery
videos. Project page: https://fyycs.github.io/medclaw/.
Read the full abstract ▾
All authors Yingying Fan, Penghui Du, Leyan Zhu, Runze He, Zimeng Wu, Yuxuan Zhang, Liang Chen, Jiahao Xie, Jiangtang Wang, Shuai Shao, Anchao Yang, Yutong Bai, Yan Wang
Computer Vision · Cross-listed from eess.IV
A Subjective Study on a New Sharpness Informed Class of Metrics
Uditangshu Aurangabadkar, Vibhoothi Vibhoothi, Darren Ramsook et al. (+1 more)
Perceptual loss functions in Deep Neural Network (DNN) deblurring architectures improve the overall quality of restored images. However, few focus on explicitly targeting sharpness in the restorations. We conduct a subjective study of models trained with and without losses which explicitly target sharpness using a four-protocol approach, exploring preferred sharpness levels and effects on image quality. We introduce a novel
dataset of images with uniform sharpness increments along with Difference Mean Opinion Scores (DMOS). Additionally, we propose a novel class of Sharpness Informed (SI) Image Quality Assessment (IQA) metrics which properly penalize over-sharpening. Our new SI-PSNR metric outperforms all other PSNR variants in terms of correlation statistics on IQA benchmarking
datasets. We show that, on average, images restored using a sharpness-aware composite loss are preferred in 67% of binarized comparisons, as opposed to losses that do not explicitly target sharpness.
Read the full abstract ▾
All authors Uditangshu Aurangabadkar, Vibhoothi Vibhoothi, Darren Ramsook, Anil Kokaram
Computer Vision
FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation
Weidong Tang, Kaiyu Li, Yikai Wang et al. (+4 more)
Reasoning segmentation requires
multimodal large
language models (MLLMs) to translate implicit instructions into precise pixel-level masks. MLLMs encode an image as visual
tokens, each of which merges a group of image patches. In remote sensing images, small targets, thin structures, and adjacent instances can occupy different parts of the same visual
token. Assigning a single binary mask label to such a
token loses its internal spatial structure, causing nearby targets to merge and object boundaries to become coarse. To bridge this representational gap, we introduce FIRM, a Fine-grained Intra-
token Representation of Masks. For each visual
token, FIRM predicts a mask code that specifies an $r\times r$ binary sub-
cell pattern rather than a single foreground/background label. Given a target identified by the MLLM, the complete grid of mask codes is predicted in one mask pass. Fixed lookup converts the predicted codes into a discrete sub-
cell mask, while marginalizing the code distribution yields a soft structural field. To further recover fine-grained boundaries within each sub-
cell, we introduce a lightweight continuous renderer that refines this field using pre-merge visual features and image details. Across five
reasoning and referring
segmentation benchmarks on satellite and
UAV images, FIRM achieves leading results, including $70.5/80.5$ gIoU/cIoU on
LaSeRS and a $3.0$-point average gain on EarthReason. These results demonstrate the value of explicitly representing intra-
token mask patterns for fine-grained MLLM
segmentation.
Read the full abstract ▾
All authors Weidong Tang, Kaiyu Li, Yikai Wang, Yanan Wu, Haotian Gan, Shihong Wang, Xiangyong Cao
Computer Vision
ProFocus: Interpreting Affective Experience in Artistic Images with Progressive Visual Focusing
Zhiyan Zhang, Zicheng Yan, Jianqi Chen et al. (+3 more)
Interpreting the emotional responses triggered by images is central to achieving emotional intelligence. Compared with natural images, visual art is intentionally created to elicit emotional responses from its viewers through abstract concepts and visual metaphors, making affective interpretation particularly challenging. However, most existing methods rely on general-purpose visual
embeddings (e.g., CLIP), failing to capture the nuanced cues underlying artistic
emotion. To address this gap, we propose ProFocus, a novel framework that models affective experience in artistic images via progressive visual focusing. The key idea is to model visual representation learning inspired by a hierarchical cognitive theory of human aesthetic appreciation. Technically, ProFocus contains two core components: a Hierarchical Art Critic (HAC) and a Progressive Hint Fusion (PHF) module. HAC leverages
multimodal large
language models to generate structured linguistic priors at three cognitive levels–atmospheric style, narrative subjects, and concrete details–thereby translating artistic perception into coherent semantic guidance. Building upon these priors, PHF departs from conventional cross-modal fusion by sequentially injecting the hierarchical hints into visual features, enabling a progressive focusing process that mirrors human perception. This design allows the model to capture subtle affective cues and produce more faithful explanations. Extensive experiments on the ArtEmis v1.0 and v2.0
datasets demonstrate that ProFocus consistently outperforms state-of-the-art methods in both
emotion recognition and affective explanation. Project page: https://github.com/Zhang-Zhiyan/ProFocus.
Read the full abstract ▾
All authors Zhiyan Zhang, Zicheng Yan, Jianqi Chen, Peipei Song, Shanshan Wang, Xun Yang
Computer Vision
Rethinking Auxiliary Modalities in Multimodal Zero-shot Anomaly Detection: From Semantic Fusion to Conditional Modulation
Peng Wu, Xin Ge, Yujia Sun et al. (+1 more)
Recent foundation model-based methods have endowed RGB images with strong zero-shot anomaly detection (ZSAD) through
vision-
language pretraining. However, RGB observations alone remain limited in perceiving anomalies dominated by
geometric deformation, depth variation, or subtle surface changes. Auxiliary modalities can provide complementary structural information, but existing
multimodal methods typically fuse them directly into a shared semantic space, which may disturb the text-aligned anomaly semantics established by RGB foundation models and often requires modality-specific architectures. To address this issue, we propose a plug-and-play auxiliary-conditioned enhancement framework for zero-shot anomaly detection. Instead of reconstructing a joint
multimodal anomaly semantic space, our framework preserves the original RGB image-text anomaly matching pathway and uses auxiliary observations as conditional signals for RGB feature refinement, allowing auxiliary modalities to seamlessly enhance existing RGB-based zero-shot anomaly detectors. Specifically, a lightweight meta-learning module takes global RGB and auxiliary representations as input and generates sample-adaptive low-rank residual updates to determine how RGB features should be refined. We further construct uncertainty-aware spatial modulation from the initial RGB anomaly response and auxiliary reliability, which determines where local residual updates are strengthened or suppressed. This global-to-local conditional modulation enables selective
multimodal enhancement while preserving the original RGB anomaly semantics. Extensive experiments on MVTec 3D-AD and Eyecandies demonstrate that our framework consistently improves multiple popular RGB-based zero-shot anomaly detectors, achieving state-of-the-art performance for
multimodal zero-shot anomaly detection.
Read the full abstract ▾
All authors Peng Wu, Xin Ge, Yujia Sun, Guansong Pang
Computer Vision
PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuning
Liang Wang, Haoyang Li, Chao Wang et al. (+3 more)
Prompt tuning adapts CLIP-based
vision-
language models with few trainable parameters, yet its predictions remain sensitive to the spatial sampling imposed by a frozen
vision transformer. In particular, non-overlapping patch tokenization makes predictions depend on the
alignment (phase) between image and the patch
lattice. To reduce prediction sensitivity to patch-grid
alignment, we introduce Patch-Phase Orbit Marginalization (PPOM), a training-free inference operator that treats phase shift as a nuisance variable. Given a patch stride, PPOM evaluates the identity view and reflection-padded translations, pairs opposite shifts into horizontal, vertical, and diagonal antithetic families, and assigns equal mass to these families and the identity prediction to avoid view-count bias during phase integration. In summary, PPOM provides a deterministic interface between
prompt adaptation and patch-grid sensitivity. Across multiple
prompt-learning hosts, PPOM improves host performance without re-training.
Read the full abstract ▾
All authors Liang Wang, Haoyang Li, Chao Wang, Guodong Long, Jing Jiang, Yan Peng
Computer Vision
SAFE: Scene-Aware Feature Modulation for Color Constancy with Learned Color Space in Pure-Color Scenes
Yuan-Kang Lee, Kuan-Lin Chen, Chih-Heng Chang et al. (+1 more)
Color constancy on pure-color scenes is challenging: when most pixels share a narrow band of hues, every chromaticity-based cue collapses to a single point and standard
estimators become ambiguous. We propose a compact framework that couples two innovations: (i) SAFE, a Scene-Aware FeaturE modulation network that organizes illumination cues into a structured four-
token representation, which is then selectively reweighted based on scene complexity features; (ii) the Learned Color Space (LCS), a scene-dependent chromaticity normalization that directly addresses the chromaticity collapse problem for pure-color scenes. Experiment results show that SAFE consistently improves performance in pure-color scenes. Compared to the best-performing baseline in each metric, it reduces the mean angular error by 10%, the best-25% error by 20%, and the worst-25% error by 5.8%.
Read the full abstract ▾
All authors Yuan-Kang Lee, Kuan-Lin Chen, Chih-Heng Chang, Jian-Jiun Ding
Computer Vision
Fast Implicit Neural Light Field Representation via Geometric Decomposition and Multi-Resolution Low-Rank Features
Yao Guo, Ligen Shi, Shuchen Sun et al. (+2 more)
Implicit neural representations provide a compact and continuous way to reconstruct dense light fields from sampled ray coordinates. However, fast light field reconstruction remains challenging because a light field is a high-dimensional signal with strong spatial-angular redundancy and structured disparity variations. Directly fitting 4D ray coordinates with a neural network often requires considerable optimization time to recover both view appearance and cross-view consistency. To address this issue, this paper proposes a fast implicit light field representation based on
geometric decomposition and multi-resolution low-rank features. The proposed method decomposes a 4D light field into a horizontal disparity plane, a spatial texture plane, and a vertical disparity plane. Each plane is represented by a low-rank structure that combines a low-resolution 2D grid with the element-wise product of two high-resolution 1D line features at multiple resolution levels. The fused features are decoded by a lightweight multilayer perceptron to predict RGB values. Experiments on public light field
datasets show that the proposed method achieves competitive reconstruction quality while providing a better trade-off among model parameters, training time, and inference efficiency.
Read the full abstract ▾
All authors Yao Guo, Ligen Shi, Shuchen Sun, Jun Qiu, Chang Liu
Computer Vision
CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification
Bingxin Yu, Xueli Wang, Jerry Zhou et al. (+6 more)
Ultrasound is the primary imaging modality for assessing thyroid nodules, and the ACR TI-RADS framework standardizes diagnosis through five ultrasound feature categories that are aggregated into five risk levels (TR1-TR5). Although widely adopted in clinical practice, most deep learning approaches focus on binary malignancy classification, while multi-class prediction and explicit utilization of feature-level supervision remain underexplored, largely due to limited annotated data. In this study, we introduce the STN
dataset of 600 thyroid nodules with paired transverse and longitudinal ultrasound images, bounding box annotations, and complete labels for all five TI-RADS feature categories. Following the clinical decision process, we investigate how structured feature information can guide representation learning during training while requiring only images at inference. We demonstrate that text
embeddings derived from standardized feature descriptions form a stable surrogate representation for TI-RADS risk levels. Based on this observation, we propose CMCNet, which aligns image
embeddings to fixed textual
embeddings via a Center-Margin Contrastive Loss that simultaneously promotes intra-class compactness and inter-class separation. Experimental results show that this
embedding alignment strategy is more data-efficient and robust than direct multitask learning, and consistently outperforms InfoNCE, center loss, a strong multitask baseline, and a VQA-style
multimodal model, particularly in imbalanced settings. The
dataset is freely available at doi: 10.5281/zenodo.19125693 and the source code is available at: https://www.healthinformaticslab.org/supp/.
Read the full abstract ▾
All authors Bingxin Yu, Xueli Wang, Jerry Zhou, Wenyan Wang, Li Wen, Lan Huang, Xin Feng, Fengfeng Zhou, Kewei Li
Computer Vision
CoANeRV: Coordinate-Aware Token-Space Neural Video Representation
Jialong Guo, Ke Liu, Mengxuan Li et al. (+2 more)
Neural representations for
videos (NeRV) have shown strong reconstruction fidelity by storing
video-specific information in network weights. However, existing formulations typically require either costly per-
video optimization or
video-specific weight generation, making it difficult to scale to efficient amortized
video representation. We propose CoANeRV, a coordinate-aware
token-space framework that adapts the broader
token-conditioned neural-field paradigm to amortized
video representation. CoANeRV forms compact
video tokens in one feed-forward pass and uses a shared coordinate-conditioned decoder to reconstruct continuous spatio-temporal queries, avoiding per-
video decoder optimization or generation while retaining coordinate-level reconstruction flexibility. To make
token-space reconstruction effective, CoANeRV introduces a coordinate-aware decoding architecture that aligns spatio-temporal queries with
video tokens through axis-adaptive positional encoding and temperature-modulated cross-
attention. Block-wise coordinate querying further reduces peak
attention memory, making high-resolution reconstruction practical. Experiments on diverse
video datasets show that CoANeRV consistently improves reconstruction quality over prior feed-forward NeRV and INR baselines, reduces peak
memory compared with
attention-based coordinate decoders, and provides efficient amortized encoding without per-
video optimization. These results support the proposed
video-specific combination of feed-forward
token formation, spatio-temporal coordinate
retrieval, and
memory-bounded dense querying. The code is available at https://github.com/jialong2023/CoANeRV.
Read the full abstract ▾
All authors Jialong Guo, Ke Liu, Mengxuan Li, Jiajun Bu, Haishuai Wang
Computer Vision
RGBX-Next: Towards Realistic Generative Rendering from G-Buffers
Zheng Zeng, Marco Salvi, Lifan Wu et al. (+9 more)
Diffusion models have achieved impressive results in image,
video, and streaming generation. However, compared to traditional 3D rendering, they still lack precise control over the generated output. We believe a viable path forward is to use generative models as learned renderers conditioned on traditionally rendered G-buffers. We introduce RGBX-Next, a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images,
videos, and streams, and rendering realistic images,
videos, and streams from G-buffers. Our key contribution is a general recipe for finetuning
diffusion transformer (DiT) models into generative forward and inverse renderers. We show that the resulting models achieve high quality in both realistic generative rendering and intrinsic decomposition. We will make all our models publicly available. We believe that the design principles presented in this paper will benefit future research on controllable generative forward and inverse rendering.
Read the full abstract ▾
All authors Zheng Zeng, Marco Salvi, Lifan Wu, Jan Novák, Daqi Lin, Saeed Hadadan, Yichen Sheng, Robert Pottorff, Shiqiu Liu, Ravi Ramamoorthi, Ling-Qi Yan, Miloš Hašan
Computer Vision
OpenBelief-Nav: Evidence-Preserving Object Memory for Open-Vocabulary Language-Guided Navigation
Dinh Tuan Nguyen, Anh Dao, Phuong Nam Dang et al. (+4 more)
Open-vocabulary 3D scene graphs provide compact semantic
memory for
language-guided
navigation, but mapped objects are often exposed through a single fused feature or committed semantic label. Such commitment can remove minority yet task-relevant hypotheses from the task-time interface. We present OpenBelief-Nav, an evidence-preserving object
memory that retains observation-level phrases, reliability cues, and frame-mask provenance while maintaining separate aggregate
geometric and visual representations. Semantically related phrases are consolidated into a vocabulary-independent object belief from which task-specific readouts perform fixed-vocabulary projection or free-form
retrieval. On five ScanNet200 and eight Replica scenes, full-belief projection achieves mIoU scores of 0.2742 and 0.2912, compared with 0.2393 and 0.2701 for a matched early-commit readout. Across 78 HM3D-YCB
navigation trials, consensus and early-commit
retrieval each achieve 60/78 successes, compared with 58/78 for belief-weighted
retrieval and 55/78 for DualMap. Across 20 Unitree G1 runs organized as 10 matched evaluation cases, a correction policy permitting at most two verified candidate attempts improves target-confirmation success from 6/10 to 8/10 relative to top-1-only execution. Code will be released upon acceptance at https://openbelief-nav.github.io/.
Read the full abstract ▾
All authors Dinh Tuan Nguyen, Anh Dao, Phuong Nam Dang, Quan-Dung Pham, Tuyen P. Le, Truong Nguyen, Quan Nguyen
Computer Vision
Beyond Control Points: Arcsecond Relative-Motion Estimation of Vision Measurement Platforms With Incomplete or Absent Control Fields
Meng Lian, Jian Wang, Shuixin Pan et al. (+3 more)
Long-range
vision-based deformation monitoring is highly sensitive to motion of the camera platform. Absolute-pose differencing typically relies on dedicated control data and propagates two independent pose errors into the relative-motion estimate. We develop a control-adaptive differential framework that estimates inter-frame platform motion directly from image displacements and known 3D points. With no dedicated control point, the framework recovers platform rotation from measurement-point observations. One surveyed control point enables prior-constrained translation recovery, while two nonparallel control rays recover full 3D translation. The framework requires neither nonlinear optimization nor an initial pose estimate. Excluding control data from the rotation stage makes the rotation estimate exactly immune to contamination confined to the control field. The inherited differential formulation also cancels translational extrinsic errors exactly. We derive the rotation observability condition, a leakage bound for unmodeled translation and nonrigid point motion, and the single-point axial-prior bias law. Under 0.5-pixel image noise, attitude changes of up to 30~arcmin, and 3D point perturbations of up to 2~mm, the multi-camera
estimator achieves a rotation RMSE of 2.97~arcsec and an average runtime of 0.46~ms. With one surveyed control point, its prior-constrained translation RMSE is 1.19~mm. In a bridge experiment without a stable control field, the median coordinate-wise displacement RMSE relative to total-station measurements is 0.85~mm. The
estimator also maintains zero divergence under the tested 3D coordinate perturbations on public RGB-D and stereo sequences. These results establish state-of-the-art accuracy, calibration robustness, and computational efficiency among the evaluated methods.
Read the full abstract ▾
All authors Meng Lian, Jian Wang, Shuixin Pan, Haibo Liu, Yueqiang Zhang, Yulan Guo