Computer Vision
Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models
Nicoleta-Nina Basoc, Adrian Cosma and Emilian Radoi
Vision-
language models (VLMs) achieve strong
benchmark results, yet can exhibit systematic perceptual weaknesses: structured, large changes to pixel values can cause confident yet nonsensical predictions, even when the underlying scene remains easily recognizable to humans. We study this gap using Spatial Colour Mixing, a programmatic family of colour distortions that overlays structured patterns (in both RGB and Ostwald colour systems) onto natural images. We introduce a framework of eight spatial colour mixing variants and evaluate nine VLMs across three model families on four
datasets. Across models and
datasets, accuracy degrades sharply with increasing distortion, and scaling the
language model does not reliably mitigate the failure. In a human study with 61 participants on an animal recognition
dataset, humans substantially outperform VLMs under the same distortions. Finally, we show that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types, motivating perception-aware preprocessing and tool-use as practical strategies for improving VLM robustness.
Read the full abstract ▾
Computer Vision
Place-it-R1: Unlocking Environment-aware Reasoning Potential of MLLM for Video Object Insertion
Bohai Gu, Taiyi Wu, Dazhao Du et al. (+5 more)
Video object insertion is fundamental to
video editing, yet existing
diffusion methods often produce visually plausible but physically inconsistent results. We present Place-it-R1, an end-to-end framework for physically plausible
video object insertion driven by environment-aware MLLM
reasoning. Rather than treating
reasoning as a generic text
prompt, Place-it-R1 uses the MLLM to analyze the target environment, infer object-scene interactions, and determine where an insertion is physically valid. The resulting
reasoning is translated into two complementary forms of guidance for
video diffusion: semantic guidance that describes the intended physical interaction and spatial guidance that provides a valid insertion region in each frame. To further align generation with local physical realism, we introduce Spatial Direct Preference Optimization, which leverages an MLLM to rank generated candidates, and introduces a region-aware preference objective that explicitly localizes physical-violation penalties to the inserted-object region. Place-it-R1 further offers flexible and standard modes to trade off environment adaptation and scene preservation. Extensive experiments show that Place-it-R1 produces more physically coherent and visually natural insertions than state-of-the-art methods and achieves competitive results against commercial systems.
Read the full abstract ▾
All authors Bohai Gu, Taiyi Wu, Dazhao Du, Jian Liu, Shuai Yang, Xiaotong Zhao, Alan Zhao, Song Guo
Computer Vision
Cross-Resolution Distribution Matching for Diffusion Distillation
Feiyang Chen, Hongpeng Pan, Haonan Xu et al. (+3 more)
Diffusion distillation is central to accelerating image and
video generation, yet existing methods are fundamentally limited by the denoising process, where step reduction has largely saturated. Partial timestep low-resolution generation can further accelerate inference, but it suffers noticeable quality degradation due to cross-resolution distribution gaps. We propose Cross-Resolution Distribution Matching
Distillation (RMD), a novel
distillation framework that bridges cross-resolution distribution gaps for high-fidelity, few-step multi-resolution cascaded inference. Specifically, RMD divides the timestep intervals for each resolution using logarithmic signal-to-noise ratio (logSNR) curves, and introduces logSNR-based mapping to compensate for resolution-induced shifts. Distribution matching is conducted along resolution trajectories to reduce the gap between low-resolution generator distributions and the teacher's high-resolution distribution. In addition, a predicted-noise re-injection mechanism is incorporated during upsampling to stabilize training and improve synthesis quality. Quantitative and qualitative results show that RMD preserves high-fidelity generation while accelerating inference across various backbones. Notably, RMD achieves up to 33.4X speedup on SDXL and 25.6X on Wan2.1-14B, while preserving high visual fidelity.
Read the full abstract ▾
All authors Feiyang Chen, Hongpeng Pan, Haonan Xu, Xinyu Duan, Zhefeng Wang, Yang Yang
Computer Vision
FedARKS: Federated Aggregation via Robust and Discriminative Knowledge Selection and Integration for Person Re-identification
Xin Xu, Binchang Ma, Zhixi Yu et al. (+1 more)
The application of
federated domain generalization in person re-identification (FedDG-ReID) aims to enhance the model's generalization ability in unseen domains while protecting client data
privacy. However, existing mainstream methods typically rely on global feature representations and simple averaging operations for model aggregation, leading to two limitations in domain generalization: (1) Using only global features makes it difficult to capture subtle, domain-invariant local details (such as accessories or textures); (2) Uniform parameter averaging treats all clients as equivalent, ignoring their differences in robust feature extraction capabilities, thereby diluting the contributions of high quality clients. To address these issues, we propose a novel
federated learning framework,
Federated Aggregation via Robust and Discriminative Knowledge Selection and Integration (FedARKS), comprising two mechanisms: RK (Robust Knowledge) and KS (Knowledge Selection).
Read the full abstract ▾
All authors Xin Xu, Binchang Ma, Zhixi Yu, Wei Liu
Computer Vision · Cross-listed from eess.IV
Enhancing Neural Video Compression of Static Scenes with Positive-Incentive Noise
Cheng Yuan, Zhenyu Jia, Jiawei Shao et al. (+1 more)
Static scene
videos, such as surveillance feeds and videotelephony streams, constitute a dominant share of storage consumption and network traffic. However, both traditional standardized codecs and neural
video compression (NVC) methods struggle to encode these
videos efficiently due to inadequate usage of temporal redundancy and severe distribution gaps between training and test data, respectively. While recent generative compression methods improve perceptual quality, they introduce hallucinated details that are unacceptable in authenticity-critical applications. To overcome these limitations, we propose a positive-incentive camera (PIC) framework for static scene
videos, where short-term temporal changes are reinterpreted as positive-incentive noise to facilitate NVC model finetuning. By disentangling transient variations from the persistent background, structured prior information is internalized in the compression model. During inference, the invariant component requires minimal signaling, thus reducing data transmission while maintaining pixel-level fidelity. Experiment results show that PIC achieves visually lossless reconstruction for static scenes at an extremely low compression rate of 0.009%, while the DCVC-FM baseline requires 20.5% higher Bjøntegaard delta (BD) rate. Our method provides an effective solution to trade computation for bandwidth, enabling robust
video transmission under adverse network conditions and economic long-term retention of surveillance footage.
Read the full abstract ▾
All authors Cheng Yuan, Zhenyu Jia, Jiawei Shao, Xuelong Li
Computer Vision
DeepSight: Bridging Depth Maps and Language with a Depth-Driven Multimodal Model
Hao Yang, Hongbo Zhang, Yanyan Zhao et al. (+1 more)
Multimodal large
language models (MLLMs) have achieved impressive performance across various tasks such as image captioning and visual question answer(VQA); however, they often struggle to accurately interpret depth information inherent in visual data. In this work, we introduce DeepSight, the first dedicated depth MLLM designed to enhance three-dimensional scene understanding. Unlike conventional methods that align RGB image encodings with text, our approach takes advantage of the unique characteristics of depth images: single-channel grayscale images where the pixel values directly reflect depth cues to improve spatial
reasoning. To address challenges associated with limited depth data and the inadequacy of simple channel replication, we construct a novel depth image-text pair
dataset and a depth instruction
dataset. Depth maps are generated from visual images using the GLPN model, and GPT-4 is employed to curate corresponding depth instructions, an approach validated by LLaVA. Additionally, we modify the ViT encoder in CLIP to incorporate local object information, thereby capturing the subtle continuous variations of depth more effectively. To evaluate the performance of our model, we develop a comprehensive depth question answer
benchmark based on existing depth image
datasets, which rigorously assesses understanding in typical depth map scenarios. Experimental results demonstrate that DeepSight significantly enhances depth perception and downstream task performance, marking a substantial step forward in
multimodal three-dimensional understanding.
Read the full abstract ▾
All authors Hao Yang, Hongbo Zhang, Yanyan Zhao, Bing Qin
Computer Vision
Lyapunov Probes for Hallucination Detection in Large Foundation Models
Bozhi Luan, Gen Li, Yalan Qin et al. (+6 more)
We address
hallucination detection in Large
Language Models (LLMs) and
Multimodal Large
Language Models (MLLMs) by framing the problem through the lens of dynamical systems stability theory. Rather than treating
hallucination as a straightforward classification task, we conceptualize (M)LLMs as dynamical systems, where factual knowledge is represented by stable
equilibrium points within the representation space. Our main insight is that
hallucinations tend to arise at the boundaries of knowledge-transition regions separating stable and unstable zones. To capture this phenomenon, we propose Lyapunov Probes: lightweight networks trained with derivative-based stability constraints that enforce a monotonic decay in confidence under input perturbations. By performing systematic perturbation analysis and applying a two-stage training process, these probes reliably distinguish between stable factual regions and unstable,
hallucination-prone regions. Experiments on diverse
datasets and models demonstrate consistent improvements over existing baselines.
Read the full abstract ▾
All authors Bozhi Luan, Gen Li, Yalan Qin, Jifeng Guo, Yun Zhou, Faguo Wu, Hongwei Zheng, Wenjun Wu, Zhaoxin Fan
Computer Vision
Text-Driven Emotionally Continuous Talking Face Generation
Hao Yang, Yanyan Zhao, Tian Zheng et al. (+7 more)
Talking Face Generation (TFG) strives to create realistic and emotionally expressive digital faces. While previous TFG works have mastered the creation of naturalistic facial movements, they typically express a fixed target
emotion in synthetic
videos and lack the ability to exhibit continuously changing and natural expressions like humans do when conveying information. To synthesize realistic
videos, we propose a novel task called Emotionally Continuous Talking Face Generation (EC-TFG), which takes a text segment and an
emotion description with varying
emotions as driving data, aiming to generate a
video where the person speaks the text while reflecting the emotional changes within the description. Alongside this, we introduce a customized model, i.e., Temporal-Intensive
Emotion Modulated Talking Face Generation (TIE-TFG), which innovatively manages dynamic emotional variations by employing Temporal-Intensive
Emotion Fluctuation Modeling, allowing it to provide
emotion variation sequences corresponding to the input text to drive continuous facial expression changes in synthesized
videos. Extensive evaluations demonstrate our method's exceptional ability to produce smooth
emotion transitions and uphold high-quality visuals and motion authenticity across diverse emotional states.
Read the full abstract ▾
All authors Hao Yang, Yanyan Zhao, Tian Zheng, Hongbo Zhang, Bichen Wang, Di Wu, Xing Fu, Xuda Zhi, Yongbo Huang, Hao He
Computer Vision
Transforming Omnidirectional RGB-LiDAR data into 3D Gaussian Splatting
Semin Bae, Hansol Lim and Jongseong Brad Choi
The demand for large-scale digital twins is rapidly growing in robotics and autonomous driving. However, constructing these environments with 3D Gaussian Splatting (3DGS) usually requires expensive, purpose-built data collection. Meanwhile, deployed platforms routinely collect extensive omnidirectional RGB and
LiDAR logs, but a significant portion of these sensor data is directly discarded or strictly underutilized due to transmission constraints and the lack of scalable reuse pipeline. In this paper, we present an omnidirectional RGB-
LiDAR reuse pipeline that transforms these archived logs into robust initialization assets for 3DGS. Direct conversion of such raw logs introduces practical bottlenecks: inherent non-linear distortion leads to unreliable Structure-from-Motion (SfM) tracking, and dense, unorganized
LiDAR clouds cause computational overhead during 3DGS optimization. To overcome these challenges, our pipeline strategically integrates an ERP-to-cubemap conversion module for deterministic spatial anchoring, alongside PRISM-a color stratified downsampling strategy. By bridging these multi-modal inputs via Fast Point Feature Histograms (FPFH) based global registration and Iterative Closest Point (ICP), our pipeline successfully repurposes a considerable fraction of discarded data into usable SfM geometry. Furthermore, our
LiDAR-reinforced initialization consistently enhances the final 3DGS rendering fidelity in structurally complex scenes compared to
vision-only baselines. Ultimately, this work provides a deterministic workflow for creating simulation-grade digital twins from standard archived sensor logs.
Read the full abstract ▾
Computer Vision
TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation
Soumya Mazumdar and Vineet Kumar Rakesh
Diffusion models have recently advanced photorealistic human synthesis, although practical talking-head generation (THG) remains constrained by high inference latency, temporal instability such as flicker and identity drift, and imperfect audio-visual
alignment under challenging
speech conditions. This paper introduces TempoSyncDiff, a reference-conditioned latent
diffusion framework that explores few-step inference for efficient audio-driven talking-head generation. The approach adopts a teacher-student
distillation formulation in which a
diffusion teacher trained with a standard noise prediction objective guides a lightweight student denoiser capable of operating with significantly fewer inference steps to improve generation stability. The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker during synthesis, while viseme-based audio conditioning provides coarse lip motion control. Experiments on the LRS3
dataset report denoising-stage component-level metrics relative to VAE reconstructions and preliminary latency characterization, including CPU-only and edge computing measurements and feasibility estimates for edge deployment. The results suggest that distilled
diffusion models can retain much of the reconstruction behaviour of a stronger teacher while enabling substantially lower latency inference. The study is positioned as an initial step toward practical
diffusion-based talking-head generation under constrained computational settings. GitHub: https://mazumdarsoumya.github.io/TempoSyncDiff
Read the full abstract ▾
Computer Vision
Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving
Nikos Theodoridis, Reenu Mohandas, Ganesh Sistu et al. (+3 more)
The use of
Vision-
Language Models (VLMs) in automated driving applications is becoming increasingly common, with the aim of leveraging their
reasoning and generalisation capabilities to handle long-tail scenarios. However, these models often fail on simple visual questions that are highly relevant to automated driving, and the reasons behind these failures remain poorly understood. In this work, we examine the intermediate activations of VLMs and assess the extent to which specific visual concepts are linearly encoded, with the goal of identifying bottlenecks in the flow of visual information. Specifically, we create counterfactual image sets that differ only in a targeted visual concept and then train linear probes to distinguish between them using the activations of five state-of-the-art (SOTA) VLMs, including two training variants for one model. Our results show that concepts such as the presence of an object or
agent in a scene are explicitly and linearly encoded, whereas other spatial visual concepts, such as the orientation of an object or
agent, are only implicitly encoded by the spatial structure retained by the
vision encoder, or are not linearly encoded at all. In parallel, we observe that in certain cases, even when a concept is linearly encoded in the model's activations, the model still fails to answer correctly. This leads us to identify two failure modes. The first is perceptual failure, where the visual information required to answer a question is not linearly encoded in the model's activations. The second is cognitive failure, where the visual information is present but the model fails to align it correctly with
language semantics. Finally, we show that increasing the distance of the object in question quickly degrades the linear separability of the corresponding visual concept...
Read the full abstract ▾
All authors Nikos Theodoridis, Reenu Mohandas, Ganesh Sistu, Anthony Scanlan, Ciarán Eising, Tim Brophy
Computer Vision
Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models
Canyu Chen, Yuguang Yang, Zhewen Tan et al. (+10 more)
We identify a fundamental Narrow Policy limitation undermining the performance of autonomous VLA models, where driving Imitation Learning (IL) tends to collapse exploration and limit the potential of subsequent Reinforcement Learning (RL) stages, which often saturate prematurely due to insufficient feedback diversity. Thereby, we propose Curious-VLA, a framework that alleviates the exploit-explore dilemma through a two-stage design. During IL, we introduce a Feasible Trajectory Expansion (FTE) strategy to generate multiple physically valid trajectories and a step-wise normalized trajectory representation to adapt this diverse data. In the RL stage, we present Adaptive Diversity-Aware Sampling (ADAS) that prioritizes high-diversity samples and introduce Spanning Driving Reward (SDR) with a focal style weighting to amplify reward's value span for improving sensitivity to driving quality. On the Navsim
benchmark, Curious-VLA achieves SoTA results (PDMS 90.3, EPDMS 85.4) and a Best-of-N PDMS of 94.8, demonstrating its effectiveness in unlocking the exploratory potential of VLA models. Code: https://github.com/Mashiroln/curious_vla.git.
Read the full abstract ▾
All authors Canyu Chen, Yuguang Yang, Zhewen Tan, Yizhi Wang, Ruiyi Zhan, Haiyan Liu, Xuanyao Mao, Jason Bao, Xinyue Tang, Linlin Yang, Bingchuan Sun, Yan Wang, Baochang Zhang
Computer Vision
GenHOI: Towards Object-Consistent Hand-Object Interaction with Temporally Balanced and Spatially Selective Object Injection
Xuan Huang, Mochu Xiang, Zhelun Shen et al. (+9 more)
Hand-Object Interaction (HOI) remains a core challenge in digital human
video synthesis, where models must generate physically plausible contact and preserve object identity across frames. Although recent HOI reenactment approaches have achieved progress, they are typically trained and evaluated in-domain and fail to generalize to complex, in-the-wild scenarios. In contrast, all-in-one
video editing models exhibit broader robustness but still struggle with HOI-specific issues such as inconsistent object appearance. In this paper, we present GenHOI, a lightweight augmentation to pretrained
video generation models that injects reference-object information in a temporally balanced and spatially selective manner. For temporal balancing, we propose Head-Sliding RoPE, which assigns head-specific temporal offsets to reference
tokens, distributing their influence evenly across frames and mitigating the temporal decay of 3D RoPE to improve long-range object consistency. For spatial selectivity, we design a two-level spatial
attention gate that concentrates object-conditioned
attention on HOI regions and adaptively scales its strength, preserving background realism while enhancing interaction fidelity. Extensive qualitative and quantitative evaluations on unseen, in-the-wild scenes demonstrate that GenHOI significantly outperforms state-of-the-art HOI reenactment and all-in-one
video editing methods. Project page: https://xuanhuang0.github.io/GenHOI/
Read the full abstract ▾
All authors Xuan Huang, Mochu Xiang, Zhelun Shen, Jinbo Wu, Chenming Wu, Chen Zhao, Kaisiyuan Wang, Hang Zhou, Shanshan Liu, Haocheng Feng, Wei He, Jingdong Wang
Computer Vision
Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models
Jiadong Pan, Liang Li, Yuxin Peng et al. (+6 more)
Recently, unified
multimodal models (UMMs) have made remarkable progress in integrating visual understanding and generation, demonstrating strong potential for complex text-to-image (T2I) tasks. Despite their theoretical promise, a persistent capability gap exists: UMMs typically exhibit superior visual understanding but comparatively weaker generative capabilities. This discrepancy arises largely from the intrinsic decoupling between the understanding and generation processes. While a UMM can accurately interpret fine-grained visual details, it often struggles to produce semantically coherent images from complex textual
prompts. To address this challenge, we explore UMMs' internal understanding capability to enhance generation quality. We propose a
token-level intrinsic text-image
alignment reward mechanism, GvU, enabling the UMM to act simultaneously as teacher and student: it evaluates its own outputs using the understanding branch to guide the generations accordingly. Building upon this, we design a self-supervised reinforcement learning framework, allowing UMMs to iteratively improve their generation quality through understanding-based intrinsic reward signals–without reliance on external supervision. Experimental results show that our method substantially boosts UMMs' generation, which in turn strengthens their fine-grained visual understanding, narrowing the capability gap between UMMs' visual understanding and generation.
Read the full abstract ▾
All authors Jiadong Pan, Liang Li, Yuxin Peng, Yu-Ming Tang, Shuohuan Wang, Yu Sun, Hua Wu, Qingming Huang, Haifeng Wang
Computer Vision
FontUse: A Data-Centric Approach to Style- and Use-Case-Conditioned In-Image Typography
Xia Xin, Yuki Endo and Yoshihiro Kanamori
Recent text-to-image models can generate high-quality images from natural-
language prompts, yet controlling typography remains challenging: requested typographic appearance is often ignored or only weakly followed. We address this limitation with a data-centric approach that trains image generation models using targeted supervision derived from a structured annotation pipeline specialized for typography. Our pipeline constructs a large-scale typography-focused
dataset, FontUse, consisting of about 70K images annotated with user-friendly
prompts, text-region locations, and OCR-recognized strings. The annotations are automatically produced using
segmentation models and
multimodal large
language models (MLLMs). The
prompts explicitly combine font styles (e.g., serif, script, elegant) and use cases (e.g., wedding invitations, coffee-shop menus), enabling intuitive specification even for novice users. Fine-tuning existing generators with these annotations allows them to consistently interpret style and use-case conditions as textual
prompts without architectural modification. For evaluation, we introduce a Long-CLIP-based metric that measures
alignment between generated typography and requested attributes. Experiments across diverse
prompts and layouts show that models trained with our pipeline produce text renderings more consistent with
prompts than competitive baselines. The source code for our annotation pipeline is available at https://github.com/xiaxinz/FontUSE.
Read the full abstract ▾
Computer Vision
Ensemble Learning with Sparse Hypercolumns
Julia Dietlmeier, Vayangi Ganepola, Oluwabukola G. Adegboro et al. (+3 more)
Directly inspired by findings in biological
vision, high-dimensional hypercolumns are feature vectors built by concatenating multi-scale activations of convolutional neural networks for a single image pixel location. Together with powerful classifiers, they can be used for image
segmentation i.e. pixel classification. However, in practice, there are only very few works dedicated to the use of hypercolumns. One reason is the computational complexity of processing concatenated dense hypercolumns that grows linearly with the size $N$ of the training set. In this work, we address this challenge by applying stratified subsampling to the VGG16 based hypercolumns. Furthermore, we investigate the performance of ensemble learning on sparse hypercolumns. Our experiments on a
brain tumor
dataset show that stacking and voting ensembles deliver competitive performance, but in the extreme low-shot case of $N \leq 20$, a simple Logistic Regression classifier is the most effective method. For 10% stratified subsampling rate, our best average Dice score is 0.66 for $N=20$. This is a statistically significant improvement of 24.53% over the standard multi-scale UNet baseline ($p$-value = $[3.07e-11]$, Wilcoxon signed-rank test), which is less effective due to overfitting.
Read the full abstract ▾
All authors Julia Dietlmeier, Vayangi Ganepola, Oluwabukola G. Adegboro, Mayug Maniparambil, Claudia Mazo, Noel E. O'Connor
Computer Vision
Occlusion-Aware SORT: Observing Occlusion for Robust Multi-Object Tracking
Chunjiang Li, Jianbo Ma, Li Shen et al. (+2 more)
Multi-object tracking (MOT) involves analyzing object trajectories and counting the number of objects in
video sequences. However, 2D MOT faces challenges due to positional cost confusion arising from partial occlusion. To address this issue, we present the novel Occlusion-Aware SORT (OA-SORT) framework, a plug-and-play and training-free framework that includes the Occlusion-Aware Module (OAM), the Occlusion-Aware Offset (OAO), and the Bias-Aware Momentum (BAM). Specifically, OAM analyzes the occlusion status of objects, where a Gaussian Map (GM) is introduced to reduce background influence. In contrast, OAO and BAM leverage the OAM-described occlusion status to mitigate cost confusion and suppress estimation instability. Comprehensive evaluations on the DanceTrack, SportsMOT, and MOT17
datasets demonstrate the importance of occlusion handling in MOT. On the DanceTrack test set, OA-SORT achieves 63.1% and 64.2% in HOTA and IDF1, respectively. Furthermore, integrating the Occlusion-Aware framework into the four additional trackers improves HOTA and IDF1 by an average of 2.08% and 3.05%, demonstrating the reusability of the occlusion awareness.
Read the full abstract ▾
All authors Chunjiang Li, Jianbo Ma, Li Shen, Yanru Chen, Liangyin Chen
Computer Vision
StruVis: Enhancing Reasoning-based Text-to-Image Generation via Thinking with Structured Vision
Yuanhuiyi Lyu, Kaiyu Lei, Ziqiao Weng et al. (+7 more)
Reasoning-based text-to-image (T2I) generation requires models to interpret complex
prompts accurately. Existing
reasoning frameworks can be broadly categorized into two types: (1) Text-Only
Reasoning, which is computationally efficient but lacks access to visual context, often resulting in the omission of critical spatial and visual elements; and (2) Text-Image Interleaved
Reasoning, which leverages a T2I generator to provide visual references during the
reasoning process. While this approach enhances visual grounding, it incurs substantial computational costs and constrains the
reasoning capacity of MLLMs to the representational limitations of the generator. To this end, we propose StruVis, a novel framework that enhances T2I generation through Thinking with Structured
Vision. Instead of relying on intermediate image generation, StruVis employs text-based structured visual representations as intermediate
reasoning states, thereby enabling the MLLM to effectively "perceive" visual structure within a purely text-based
reasoning process. Powered by this, the
reasoning potential for T2I generation of the MLLM is unlocked through structured-
vision-guided
reasoning. Additionally, as a generator-agnostic
reasoning framework, our proposed StruVis can be seamlessly integrated with diverse T2I generators and efficiently enhance their performance in
reasoning-based T2I generation. Extensive experiments demonstrate that StruVis achieves significant performance improvements on
reasoning-based T2I
benchmarks, e.g., a 4.61% gain on T2I-ReasonBench and a 4% gain on WISE.
Read the full abstract ▾
All authors Yuanhuiyi Lyu, Kaiyu Lei, Ziqiao Weng, Xu Zheng, Lutao Jiang, Teng Li, Yangfu Li, Ziyuan Huang, Linfeng Zhang, Xuming Hu
Computer Vision · Cross-listed from cs.CL
ViewFusion: Structured Spatial Thinking Chains for Multi-View Reasoning
Xingjian Tao, Yiwei Wang, Yujun Cai et al. (+2 more)
Multi-view spatial
reasoning remains difficult for current
vision-
language models. Even when multiple viewpoints are available, models often underutilize cross-view relations and instead rely on single-image shortcuts, leading to fragile performance on viewpoint transformation and occlusion-sensitive cases. We present ViewFusion, a two-stage framework that explicitly separates cross-view spatial pre-
alignment from question answering. In the first stage, the model performs deliberate spatial pre-thinking to infer viewpoint relations and spatial transformations across views, forming an intermediate workspace that goes beyond a simple re-description. In the second stage, the model conducts question-driven
reasoning conditioned on this workspace to produce the final prediction. We train ViewFusion with synthetic
reasoning supervision followed by reinforcement learning using GRPO, which improves answer correctness while stabilizing the intended two-stage generation behavior. On MMSI-Bench, ViewFusion improves accuracy by 5.3% over Qwen3-VL-4B-Instruct, with the largest gains on examples that require genuine cross-view
alignment.
Read the full abstract ▾
All authors Xingjian Tao, Yiwei Wang, Yujun Cai, Yifan Song, Jing Tang
Computer Vision
MOSIV: Multi-Object System Identification from Videos
Chunjiang Liu, Xiaoyuan Wang, Qingran Lin et al. (+9 more)
We introduce the challenging problem of multi-object system identification from
videos, for which prior methods are ill-suited due to their focus on single-object scenes or discrete material classification with a fixed set of material prototypes. To address this, we propose MOSIV, a new framework that directly optimizes for continuous, per-object material parameters using a differentiable simulator guided by
geometric objectives derived from
video. We also present a new synthetic
benchmark with contact-rich, multi-object interactions to facilitate evaluation. On this
benchmark, MOSIV substantially improves grounding accuracy and long-horizon simulation fidelity over adapted baselines, establishing it as a strong baseline for this new task. Our analysis shows that object-level fine-grained supervision and geometry-aligned objectives are critical for stable optimization in these complex, multi-object settings. The source code and
dataset will be released.
Read the full abstract ▾
All authors Chunjiang Liu, Xiaoyuan Wang, Qingran Lin, Albert Xiao, Haoyu Chen, Shizheng Wen, Hao Zhang, Lu Qi, Ming-Hsuan Yang, Laszlo A. Jeni, Min Xu, Yizhou Zhao
Computer Vision
EffectMaker: Unifying Reasoning and Generation for Customized Visual Effect Creation
Shiyuan Yang, Ruihuang Li, Jiale Tao et al. (+3 more)
Visual effects (VFX) are essential for enhancing the expressiveness and creativity of
video content, yet producing high-quality effects typically requires expert knowledge and costly production pipelines. Existing AIGC systems face significant challenges in VFX generation due to the scarcity of effect-specific data and the inherent difficulty of modeling supernatural or stylized effects. Moreover, these approaches often require per-effect fine-tuning, which severely limits their scalability and generalization to novel VFX. In this work, we present EffectMaker, a unified
reasoning-generation framework that enables reference-based VFX customization. EffectMaker employs a
multimodal large
language model to interpret high-level effect semantics and reason about how they should adapt to a target subject, while a
diffusion transformer leverages in-context learning to capture fine-grained visual cues from reference
videos. These two components form a semantic-visual dual-path guidance mechanism that enables accurate, controllable, and effect-consistent synthesis without per-effect fine-tuning. Furthermore, we construct EffectData, the largest high-quality synthetic
dataset containing 130k
videos across 3k VFX categories, to improve generalization and scalability. Experiments show that EffectMaker achieves superior visual quality and effect consistency over state-of-the-art baselines, offering a scalable and flexible paradigm for customized VFX generation. Project page: https://effectmaker.github.io
Read the full abstract ▾
All authors Shiyuan Yang, Ruihuang Li, Jiale Tao, Shuai Shao, Qinglin Lu, Jing Liao
Computer Vision
Demystifying KAN for Vision Tasks: The RepKAN Approach
Minjong Cheon
Remote sensing image classification is essential for Earth observation, yet standard CNNs and
Transformers often function as uninterpretable black-boxes. We propose RepKAN, a novel architecture that integrates the structural efficiency of CNNs with the non-linear representational power of KANs. By utilizing a dual-path design – Spatial Linear and Spectral Non-linear – RepKAN enables the autonomous discovery of class-specific spectral fingerprints and physical interaction
manifolds. Experimental results on the EuroSAT and NWPU-RESISC45
datasets demonstrate that RepKAN provides explicit physically interpretable
reasoning while outperforming state-of-the-art models. These findings indicate that RepKAN holds significant potential to serve as the backbone for future interpretable visual foundation models.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration
Ninghao Zhang, Bin Zhu, Shijie Zhou et al. (+1 more)
Vision-
Language-Action (VLA) models enable
robots to perform manipulation tasks directly from natural
language instructions and are increasingly viewed as a foundation for generalist robotic policies. However, their reliability under Out-of-Distribution (OOD) instructions remains underexplored. In this paper, we reveal a critical failure mode in which VLA policies continue executing visually plausible actions even when the
language instruction contradicts the scene. We refer to this phenomenon as linguistic blindness, where VLA policies prioritize visual priors over instruction semantics during action generation. To systematically analyze this issue, we introduce ICBench, a diagnostic
benchmark constructed from the LIBERO
dataset that probes
language-action coupling by injecting controlled OOD instruction contradictions while keeping the visual environment unchanged. Evaluations on three representative VLA architectures, including Pi0, Pi0.5 and OpenVLA OFT, show that these models frequently succeed at tasks despite logically impossible instructions, revealing a strong visual bias in action generation. To mitigate this issue, we propose Instruction-Guided
Attention Recalibration (IGAR), a train-free inference-time mechanism that rebalances
attention distributions to restore the influence of
language instructions. IGAR operates without retraining or architectural modification and can be directly applied to existing VLA models. Experiments across 30 LIBERO tasks demonstrate that IGAR substantially reduces erroneous execution under OOD contradictory instructions while preserving baseline task performance. We additionally validate the approach on a real Franka robotic arm, where IGAR effectively prevents manipulation triggered by inconsistent instructions.
Read the full abstract ▾
All authors Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen
Computer Vision
RePer-360: Releasing Perspective Priors for 360$^\circ$ Depth Estimation via Self-Modulation
Cheng Guan, Chunyu Lin, Zhijie Shen et al. (+2 more)
Recent depth foundation models trained on perspective imagery achieve strong performance, yet generalize poorly to 360$^\circ$ images due to the substantial
geometric discrepancy between perspective and panoramic domains. Moreover, fully fine-tuning these models typically requires large amounts of panoramic data. To address this issue, we propose RePer-360, a distortion-aware self-modulation framework for
monocular panoramic depth estimation that adapts depth foundation models while preserving powerful pretrained perspective priors. Specifically, we design a lightweight geometry-aligned guidance module to derive a modulation signal from two complementary projections (i.e., ERP and CP) and use it to guide the model toward the panoramic domain without overwriting its pretrained perspective knowledge. We further introduce a Self-Conditioned AdaLN-Zero mechanism that produces pixel-wise scaling factors to reduce the feature distribution gap between the perspective and panoramic domains. In addition, a cubemap-domain consistency loss further improves training stability and cross-projection
alignment. By shifting the focus from complementary-projection fusion to panoramic domain adaptation under preserved pretrained perspective priors, RePer-360 surpasses standard fine-tuning methods while using only 1% of the training data. Under the same in-domain training setting, it further achieves an approximately 20% improvement in RMSE. The code is available at https://github.com/munimo/RePer360.
Read the full abstract ▾
All authors Cheng Guan, Chunyu Lin, Zhijie Shen, Junsong Zhang, Jiyuan Wang
Computer Vision
MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs
Zhi Lei, Chenxi Liu, Hao Miao et al. (+3 more)
Irregularly sampled time series (ISTS) are widespread in real-world scenarios, exhibiting asynchronous observations on uneven time intervals across diverse variables. Existing ISTS
forecasting methods often solely utilize historical observations to predict future ones while falling short in learning contextual semantics and fine-grained temporal patterns. To address these problems, we propose MM-ISTS, a
multimodal ISTS
forecasting framework augmented by
vision-text large
language models, which bridges temporal, visual, and textual modalities. MM-ISTS encompasses a two-stage encoding mechanism. In particular, a Cross-Modal
Vision-Text Encoding module is proposed to automatically generate informative visual images and textual data, enabling the capture of intricate temporal patterns and comprehensive contextual understanding, in collaboration with
multimodal LLMs (MLLMs). In parallel, ISTS encoding extracts complementary yet enriched temporal features from historical ISTS observations, including multi-view
embedding fusion and a Temporal-Variable Encoder. Further, we propose an Adaptive Query-Based Feature Extractor to compress MLLM
token embeddings while preserving useful knowledge, which in turn reduces computational costs. In addition, a
Multimodal Alignment module with Modality-Aware Gating is designed to alleviate the modality gaps. Extensive experiments on real data offer insight into the effectiveness of the proposed solutions.
Read the full abstract ▾
All authors Zhi Lei, Chenxi Liu, Hao Miao, Wanghui Qiu, Bin Yang, Chenjuan Guo
Computer Vision
Technical Report: Automated Optical Inspection of Surgical Instruments
Zunaira Shafqat, Atif Aftab Ahmed Jilani and Qurrat Ul Ain
In the dynamic landscape of modern healthcare, maintaining the highest standards in
surgical instruments is critical for clinical success. This report explores the diverse realm of
surgical instruments and their associated manufacturing defects, emphasizing their pivotal role in ensuring the safety of
surgical procedures. With potentially fatal consequences arising from even minor defects, precision in manufacturing is paramount.The report addresses the identification and rectification of critical defects such as cracks, rust, and structural irregularities. Such scrutiny prevents substantial financial losses for manufacturers and, more crucially, safeguards patient lives. The collaboration with industry leaders Daddy D Pro and Dr. Frigz International, renowned trailblazers in the Sialkot
surgical cluster, provides invaluable insights into the analysis of defects in Pakistani-made instruments. This partnership signifies a commitment to advancing automated defect detection methodologies, specifically through the integration of deep learning architectures including YOLOv8, ResNet-152, and EfficientNet-b4, thereby elevating quality standards in the manufacturing process. The scope of this report is to identify various
surgical instruments manufactured in Pakistan and analyze their associated defects using a newly developed
dataset of 4,414 high-resolution images. By focusing on quality assurance through Automated Optical Inspection (AOI) tools, this document serves as a resource for manufacturers, healthcare professionals, and regulatory bodies. The insights gained contribute to the enhancement of instrument standards, ensuring a more reliable healthcare environment through industry expertise and cutting-edge technology.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
HarvestFlex: Strawberry Harvesting via Vision-Language-Action Policy Adaptation in the Wild
Ziyang Zhao, Shuheng Wang, Zhonghua Miao et al. (+1 more)
This work presents the first study on transferring
vision-
language-action (VLA) policies to real greenhouse tabletop strawberry harvesting, a long-horizon, unstructured task challenged by occlusion and specular reflections. We built an end-to-end closed-loop system on the HarvestFlex platform using three-view RGB sensing (two fixed scene views plus a wrist-mounted view) and intentionally avoided depth clouds and explicit
geometric calibration. We collected 3.71 h of VR teleoperated demonstrations (227 episodes) and fine-tuned pi_0, pi_0.5, and WALL-OSS with full fine-tuning and LoRA. Under a unified 50 trials real-greenhouse protocol and metrics spanning completion, pi_0.5 with full fine-tuning achieved success rate of 74.0% with 32.6 s/pick and damage rate of 4.1%. Asynchronous inference-control decoupling further improved performance over synchronous deployment. Results showed non-trivial closed-loop picking with fewer than four hours of real data, while remaining limited by close-range observability loss and contact-dynamics mismatch. A demonstration
video is available at: https://youtu.be/bN8ZowZKPMI.
Read the full abstract ▾
All authors Ziyang Zhao, Shuheng Wang, Zhonghua Miao, Ya Xiong
Computer Vision
Towards High-resolution and Disentangled Reference-based Sketch Colorization
Dingkun Yan, Xinrui Wang, Ru Wang et al. (+5 more)
Sketch colorization is a critical task for automating and assisting in the creation of animations and digital illustrations. Previous research identified the primary difficulty as the distribution shift between semantically aligned training data and highly diverse test data, and focused on mitigating the artifacts caused by the distribution shift instead of fundamentally resolving the problem. In this paper, we present a framework that directly minimizes the distribution shift, thereby achieving superior quality, resolution, and controllability of colorization. We propose a dual-branch framework to explicitly model the data distributions of the training process and inference process with a semantic-aligned branch and a semantic-misaligned branch, respectively. A Gram Regularization Loss is applied across the feature maps of both branches, effectively enforcing cross-domain distribution coherence and stability. Furthermore, we adopt an anime-specific Tagger Network to extract fine-grained attributions from reference images and modulate SDXL's conditional encoders to ensure precise control, and a plugin module to enhance texture transfer. Quantitative and qualitative comparisons, alongside user studies, confirm that our method effectively overcomes the distribution shift challenge, establishing State-of-the-Art performance across both quality and controllability metrics. Ablation study reveals the influence of each component.
Read the full abstract ▾
All authors Dingkun Yan, Xinrui Wang, Ru Wang, Zhuoru Li, Jinze Yu, Yusuke Iwasawa, Yutaka Matsuo, Jiaxian Guo
Computer Vision
Breaking Smooth-Motion Assumptions: A UAV Benchmark for Multi-Object Tracking in Complex and Adverse Conditions
Jingtao Ye, Kexin Zhang, Xunchi Ma et al. (+6 more)
The rapid movements and agile maneuvers of unmanned aerial vehicles (
UAVs) induce significant observational challenges for multi-object tracking (MOT). However, existing
UAV-perspective MOT
benchmarks often lack these complexities, featuring predominantly predictable camera dynamics and linear motion patterns. To address this gap, we introduce DynUAV, a new
benchmark for dynamic
UAV-perspective MOT, characterized by intense ego-motion and the resulting complex apparent trajectories. The
benchmark comprises 42
video sequences with over 1.7 million bounding box annotations, covering vehicles, pedestrians, and specialized industrial categories such as excavators, bulldozers and cranes. Compared to existing
benchmarks, DynUAV introduces substantial challenges arising from ego-motion, including drastic scale changes and viewpoint changes, as well as motion blur. Comprehensive evaluations of state-of-the-art trackers on DynUAV reveal their limitations, particularly in managing the intertwined challenges of detection and association under such dynamic conditions, thereby establishing DynUAV as a rigorous
benchmark. We anticipate that DynUAV will serve as a demanding testbed to spur progress in real-world
UAV-perspective MOT, and we will make all resources available at link.
Read the full abstract ▾
All authors Jingtao Ye, Kexin Zhang, Xunchi Ma, Yuehan Li, Guangming Zhu, Peiyi Shen, Linhua Jiang, Xiangdong Zhang, Liang Zhang
Computer Vision
Imagine How To Change: Explicit Procedure Modeling for Change Captioning
Jiayang Sun, Zixin Guo, Min Cao et al. (+2 more)
Change captioning generates descriptions that explicitly describe the differences between two visually similar images. Existing methods operate on static image pairs, thus ignoring the rich temporal dynamics of the change procedure, which is the key to understand not only what has changed but also how it occurs. We introduce ProCap, a novel framework that reformulates change modeling from static image comparison to dynamic procedure modeling. ProCap features a two-stage design: The first stage trains a procedure encoder to learn the change procedure from a sparse set of keyframes. These keyframes are obtained by automatically generating intermediate frames to make the implicit procedural dynamics explicit and then sampling them to mitigate redundancy. Then the encoder learns to capture the latent dynamics of these keyframes via a caption-conditioned, masked reconstruction task. The second stage integrates this trained encoder within an encoder-decoder model for captioning. Instead of relying on explicit frames from the previous stage – a process incurring computational overhead and sensitivity to visual noise – we introduce learnable procedure queries to
prompt the encoder for inferring the latent procedure representation, which the decoder then translates into text. The entire model is then trained end-to-end with a captioning loss, ensuring the encoder's output is both temporally coherent and captioning-aligned. Experiments on three
datasets demonstrate the effectiveness of ProCap. Code and pre-trained models are available at https://github.com/BlueberryOreo/ProCap
Read the full abstract ▾
All authors Jiayang Sun, Zixin Guo, Min Cao, Guibo Zhu, Jorma Laaksonen