Computer Vision
SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decoding
Zhaoxu Li, Chenqi Kong, Peijun Bao et al. (+5 more)
Hallucinations in Large
Vision-
Language Models (LVLMs) pose significant security and reliability risks in real-world applications. Inspired by the observation that humans are more error-prone when uncertain or hesitant, we investigate how instability in a model 's internal knowledge contributes to LVLM
hallucinations. We conduct extensive empirical analyses from three perspectives, namely
attention heads, model layers, and decoding
tokens, and identify three key
hallucination patterns: (i) visual activation drift across
attention heads, (ii) pronounced knowledge fluctuations across layers, and (iii) visual focus distraction between neighboring output
tokens. Building on these findings, we propose Stability-Aware Knowledge-Enhanced Decoding (SAKED), which introduces a layer-wise Knowledge Stability Score (KSS) to quantify knowledge stability throughout the model. By contrasting the most stability-aware and stability-agnostic layers, SAKED suppresses decoding noise and dynamically leverages the most reliable internal knowledge for faithful
token generation. Moreover, SAKED is training-free and can be seamlessly integrated into different architectures. Extensive experiments demonstrate that SAKED achieves state-of-the-art performance for
hallucination mitigation on various models, tasks, and
benchmarks.
Read the full abstract ▾
All authors Zhaoxu Li, Chenqi Kong, Peijun Bao, Song Xia, Yi Tu, Yi Yu, Xinghao Jiang, Xudong Jiang
Computer Vision
CompSplat: Compression-aware 3D Gaussian Splatting for Real-world Video
Hojun Song, Heejung Choi, Aro Kim et al. (+5 more)
High-quality novel view synthesis (NVS) from real-world
videos is crucial for applications such as cultural heritage preservation, digital twins, and immersive media. However, real-world
videos typically contain long sequences with irregular camera trajectories and unknown poses, leading to pose drift, feature misalignment, and
geometric distortion during reconstruction. Moreover, lossy compression amplifies these issues by introducing inconsistencies that gradually degrade geometry and rendering quality. While recent studies have addressed either long-sequence NVS or unposed reconstruction, compression-aware approaches still focus on specific artifacts or limited scenarios, leaving diverse compression patterns in long
videos insufficiently explored. In this paper, we propose CompSplat, a compression-aware training framework that explicitly models frame-wise compression characteristics to mitigate inter-frame inconsistency and accumulated
geometric errors. CompSplat incorporates compression-aware frame weighting and an adaptive
pruning strategy to enhance robustness and
geometric consistency, particularly under heavy compression. Extensive experiments on challenging
benchmarks, including Tanks and Temples, Free, and Hike, demonstrate that CompSplat achieves state-of-the-art rendering quality and pose accuracy, significantly surpassing most recent state-of-the-art NVS approaches under severe compression conditions.
Read the full abstract ▾
All authors Hojun Song, Heejung Choi, Aro Kim, Chae-yeong Song, Gahyeon Kim, Soo Ye Kim, Jaehyup Lee, Sang-hyo Park
Computer Vision
SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing
Tong Zhang, Honglin Lin, Zhou Liu et al. (+2 more)
Scientific diagrams convey explicit structural information, yet modern text-to-image models often produce visually plausible but structurally incorrect results. Existing
benchmarks either rely on image-centric or subjective metrics insensitive to structure, or evaluate intermediate symbolic representations rather than final rendered images, leaving pixel-based diagram generation underexplored. We introduce SciFlow-Bench, a structure-first
benchmark for evaluating scientific diagram generation directly from pixel-level outputs. Built from real scientific PDFs, SciFlow-Bench pairs each source framework figure with a canonical ground-truth graph and evaluates models as black-box image generators under a closed-loop, round-trip protocol that inverse-parses generated diagram images back into structured graphs for comparison. This design enforces evaluation by structural recoverability rather than visual similarity alone, and is enabled by a hierarchical multi-
agent system that coordinates planning, perception, and structural
reasoning. Experiments show that preserving structural correctness remains a fundamental challenge, particularly for diagrams with complex
topology, underscoring the need for structure-aware evaluation.
Read the full abstract ▾
All authors Tong Zhang, Honglin Lin, Zhou Liu, Chong Chen, Wentao Zhang
Computer Vision
Where Do Images Come From? Analyzing Captions to Geographically Profile Datasets
Abhipsa Basu, Yugam Bahl, Kirti Bhagat et al. (+3 more)
Recent studies show that text-to-image models often fail to generate geographically representative images, raising concerns about the representativeness of their training data and motivating the question: which parts of the world do these training examples come from? We geographically profile large-scale
multimodal datasets by mapping image-caption pairs to countries based on location information extracted from captions using LLMs. Studying English captions from three widely used
datasets (Re-LAION, DataComp1B, and Conceptual Captions) across $20$ common entities (e.g., house, flag), we find that the United States, the United Kingdom, and Canada account for $48.0\%$ of samples, while South American and African countries are severely under-represented with only $1.8\%$ and $3.8\%$ of images, respectively. We observe a strong correlation between a country's GDP and its representation in the data ($\rho = 0.82$). Examining non-English subsets for $4$
languages from the Re-LAION
dataset, we find that representation skews heavily toward countries where these
languages are predominantly spoken. Additionally, we find that higher representation does not necessarily translate to greater visual or semantic diversity. Finally, analyzing country-specific images generated by Stable
Diffusion v1.3 trained on Re-LAION, we show that while generations appear realistic, they are severely limited in their coverage compared to real-world images.
Read the full abstract ▾
All authors Abhipsa Basu, Yugam Bahl, Kirti Bhagat, Preethi Seshadri, R. Venkatesh Babu, Danish Pruthi
Computer Vision
Self-Supervised Learning as Discrete Communication
Kawtar Zaher, Ilyass Moummad, Olivier Buisson et al. (+1 more)
Most self-supervised learning (SSL) methods learn continuous visual representations by aligning different views of the same input, offering limited control over how information is structured across representation dimensions. In this work, we frame visual self-supervised learning as a discrete communication process between a teacher and a student network, where semantic information is transmitted through a fixed-capacity binary channel. Rather than aligning continuous features, the student predicts multi-label binary messages produced by the teacher. Discrete agreement is enforced through an element-wise binary cross-
entropy objective, while a coding-rate regularization term encourages effective utilization of the constrained channel, promoting structured representations. We further show that periodically reinitializing the projection head strengthens this effect by encouraging
embeddings that remain predictive across multiple discrete encodings. Extensive experiments demonstrate consistent improvements over continuous agreement baselines on image classification,
retrieval, and dense visual prediction tasks, as well as under domain shift through self-supervised adaptation. Beyond backbone representations, we analyze the learned binary codes and show that they form a compact and informative discrete
language, capturing semantic factors reusable across classes.
Read the full abstract ▾
All authors Kawtar Zaher, Ilyass Moummad, Olivier Buisson, Alexis Joly
Computer Vision
Robust Vision Systems for Connected and Autonomous Vehicles: Security Challenges and Attack Vectors
Sandeep Gupta and Roberto Passerone
Connected and Autonomous Vehicles (CAVs), which is critical for achieving Level-5 autonomous driving. Safe and reliable CAV
navigation depends on robust
vision systems that enable accurate detection of objects, lane markings, and traffic signs. This survey presents a reference architecture for CAV
vision systems (CAVVS) and uses it to derive a system-level threat model that maps assets, vulnerabilities, and attack points to three concrete attack surfaces (data, model, input) across the perception lifecycle. We examine attack vectors targeting each surface, rigorously evaluating their implications for confidentiality, integrity, and availability (CIA) and comparing them by adversary capability and practical exposure. We further outline atomic road events that can impact CAVVS robustness and propose a
benchmark-oriented framework of
datasets, metrics, and evaluation protocols for standardized future assessment. Together, these contributions provide a structured basis for vulnerability assessment, principled defense design, and reproducible robustness evaluation in real-world CAV
vision systems.
Read the full abstract ▾
Computer Vision
Toward Fine-Grained Facial Control in 3D Talking Head Generation
Shaoyang Xie, Xiaofeng Cong, Baosheng Yu et al. (+4 more)
Audio-driven talking head generation is a core component of digital avatars, and 3D Gaussian Splatting has shown strong performance in real-time rendering of high-fidelity talking heads. However, achieving precise control over fine-grained facial movements remains a significant challenge, particularly due to lip-synchronization inaccuracies and facial jitter, both of which can contribute to the uncanny valley effect. To address these challenges, we propose Fine-Grained 3D Gaussian Splatting (FG-3DGS), a novel framework that enables temporally consistent and high-fidelity talking head generation. Our method introduces a frequency-aware disentanglement strategy to explicitly model facial regions based on their motion characteristics. Low-frequency regions, such as the cheeks, nose, and forehead, are jointly modeled using a standard MLP, while high-frequency regions, including the eyes and mouth, are captured separately using a dedicated network guided by facial area masks. The predicted motion dynamics, represented as Gaussian deltas, are applied to the static Gaussians to generate the final head frames, which are rendered via a rasterizer using frame-specific camera parameters. Additionally, a high-frequency-refined post-rendering
alignment mechanism, learned from large-scale audio-
video pairs by a pretrained model, is incorporated to enhance per-frame generation and achieve more accurate lip synchronization. Extensive experiments on widely used
datasets for talking head generation demonstrate that our method outperforms recent state-of-the-art approaches in producing high-fidelity, lip-synced talking head
videos.
Read the full abstract ▾
All authors Shaoyang Xie, Xiaofeng Cong, Baosheng Yu, Zhipeng Gui, Jie Gui, Yuan Yan Tang, James Tin-Yau Kwok
Computer Vision
Allure of Craquelure: A Variational-Generative Approach to Crack Detection in Paintings
Laura Paul, Holger Rauhut, Martin Burger et al. (+2 more)
Recent advances in imaging technologies, deep learning and numerical performance have enabled non-invasive detailed analysis of artworks, supporting their documentation and conservation. In particular, automated detection of craquelure in digitized paintings is crucial for assessing degradation and guiding restoration, yet remains challenging due to the possibly complex scenery and the visual similarity between cracks and crack-like artistic features such as brush strokes or hair. We propose a hybrid approach that models crack detection as an inverse problem, decomposing an observed image into a crack-free painting and a crack component. A deep generative model is employed as powerful prior for the underlying artwork, while crack structures are captured using a Mumford–Shah-type variational functional together with a crack prior. Joint optimization yields a pixel-level map of crack localizations in the painting.
Read the full abstract ▾
All authors Laura Paul, Holger Rauhut, Martin Burger, Samira Kabri, Tim Roith
Computer Vision
From Lightweight CNNs to SpikeNets: Benchmarking Accuracy-Energy Tradeoffs with Pruned Spiking SqueezeNet
Radib Bin Kabir, Tawsif Tashwar Dipto, Mehedi Ahamed et al. (+2 more)
Spiking Neural Networks (SNNs) are increasingly studied as energy-efficient alternatives to Convolutional Neural Networks (CNNs), particularly for edge intelligence. However, prior work has largely emphasized large-scale models, leaving the design and evaluation of lightweight CNN-to-SNN pipelines underexplored. In this paper, we present the first systematic
benchmark of lightweight SNNs obtained by converting compact CNN architectures into spiking networks, where activations are modeled with Leaky-Integrate-and-Fire (LIF)
neurons and trained using surrogate gradient descent under a unified setup. We construct spiking variants of ShuffleNet, SqueezeNet, MnasNet, and MixNet, and evaluate them on CIFAR-10, CIFAR-100, and TinyImageNet, measuring accuracy, F1-score, parameter count, computational complexity, and energy consumption. Our results show that SNNs can achieve up to 15.7x higher energy efficiency than their CNN counterparts while retaining competitive accuracy. Among these, the SNN variant of SqueezeNet consistently outperforms other lightweight SNNs. To further optimize this model, we apply a structured
pruning strategy that removes entire redundant modules, yielding a pruned architecture, SNN-SqueezeNet-P. This pruned model improves CIFAR-10 accuracy by 6% and reduces parameters by 19% compared to the original SNN-SqueezeNet. Crucially, it narrows the gap with CNN-SqueezeNet, achieving nearly the same accuracy (only 1% lower) but with an 88.1% reduction in energy consumption due to sparse spike-driven computations. Together, these findings establish lightweight SNNs as practical, low-power alternatives for edge deployment, highlighting a viable path toward deploying high-performance, low-power intelligence on the edge.
Read the full abstract ▾
All authors Radib Bin Kabir, Tawsif Tashwar Dipto, Mehedi Ahamed, Sabbir Ahmed, Md Hasanul Kabir
Computer Vision
Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion models
Ruisi Zhao, Haoren Zheng, Zongxin Yang et al. (+2 more)
Rigged 3D assets are fundamental to 3D deformation and animation. However, existing 3D generation methods face challenges in generating animatable geometry, while rigging techniques lack fine-grained structural control over skeleton creation. To address these limitations, we introduce Stroke3D, a novel framework that directly generates rigged meshes from user inputs: 2D drawn strokes and a descriptive text
prompt. Our approach pioneers a two-stage pipeline that separates the generation into: 1) Controllable Skeleton Generation, we employ the Skeletal Graph VAE (Sk-VAE) to encode the skeleton's graph structure into a latent space, where the Skeletal Graph DiT (Sk-DiT) generates a skeletal
embedding. The generation process is conditioned on both the text for semantics and the 2D strokes for explicit structural control, with the VAE's decoder reconstructing the final high-quality 3D skeleton; and 2) Enhanced Mesh Synthesis via TextuRig and SKA-DPO, where we then synthesize a textured mesh conditioned on the generated skeleton. For this stage, we first enhance an existing skeleton-to-mesh model by augmenting its training data with TextuRig: a
dataset of textured and rigged meshes with captions, curated from Objaverse-XL. Additionally, we employ a preference optimization strategy, SKA-DPO, guided by a skeleton-mesh
alignment score, to further improve
geometric fidelity. Together, our framework enables a more intuitive workflow for creating ready to animate 3D content. To the best of our knowledge, our work is the first to generate rigged 3D meshes conditioned on user-drawn 2D strokes. Extensive experiments demonstrate that Stroke3D produces plausible skeletons and high-quality meshes.
Read the full abstract ▾
All authors Ruisi Zhao, Haoren Zheng, Zongxin Yang, Hehe Fan, Yi Yang
Computer Vision · Cross-listed from cs.LG
Physics-informed diffusion models in spectral space
Davide Gallon, Philippe von Wurstemberger, Patrick Cheridito et al. (+1 more)
We propose physics-informed spectral
diffusion (PISD), a methodology that combines generative latent
diffusion models with physics-informed machine learning to generate solutions of partial differential equations (PDEs) conditioned on partial observations, which includes, in particular, forward and inverse PDE problems. We learn the joint distribution of PDE parameters and solutions via a
diffusion process in a latent space of scaled spectral representations, where Gaussian noise corresponds to functions with controlled regularity. This spectral formulation enables significant dimensionality reduction compared to grid-based
diffusion models and ensures that the induced process in function space remains within a class of functions for which the PDE operators are well defined. Building on
diffusion posterior sampling, we enforce physics-informed constraints and measurement conditions during inference, applying Adam-based updates at each
diffusion step. We evaluate the proposed approach on Poisson, Helmholtz, and incompressible Navier-Stokes equations, demonstrating improved accuracy and computational efficiency compared with existing
diffusion-based PDE solvers, which are state of the art for sparse observations. Code is available at https://github.com/deeplearningmethods/PISD.
Read the full abstract ▾
All authors Davide Gallon, Philippe von Wurstemberger, Patrick Cheridito, Arnulf Jentzen
Computer Vision
GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentation
Sandesh Hegde, Jaison Saji Chacko, Debarshi Banerjee et al. (+1 more)
We study fine-grained referring image
segmentation via a decoupled reason-then-segment pipeline. A
vision-
language model (VLM) receives an image and a natural-
language query, reasons about the scene, and emits structured spatial
prompts: a bounding box plus two interior keypoints for every referred instance. A frozen promptable segmenter (SAM 2) converts these
prompts into high-quality masks. Within our GenSeg-R1 framework we finetune Qwen3-VL models (4B and 8B parameters) using Group Relative Policy Optimization (GRPO), requiring no supervised
reasoning-chain annotations. On RefCOCOg validation our best model (GenSeg-R1-8B) achieves 0.7127 cIoU and 0.7382 mIoU, substantially outperforming the corresponding Qwen3-VL Instruct baselines (+15.3 and +21.9 points, respectively) and surpassing Seg-Zero-7B [3] by +3.3 cIoU under identical evaluation. We further introduce GenSeg-R1-G, a variant trained on GRefCOCO [9] with a SAM 2 in-the-loop reward that directly optimizes mask quality. On GRefCOCO validation GenSeg-R1-G achieves 76.69% target mIoU with 82.40% accuracy on negative (no-target)
prompts, substantially outperforming Seg-R1-7B and Seg-Zero-7B, which lack no-target detection capability. On ReasonSeg test, GenSeg-R1-4B reaches 68.40% mIoU, surpassing Seg-Zero-7B by +7.0 and Seg-R1-7B by +10.7 points.
Read the full abstract ▾
All authors Sandesh Hegde, Jaison Saji Chacko, Debarshi Banerjee, Uma Mahesh
Computer Vision
Semi-supervised Liver Segmentation and Patch-based Fibrosis Staging with Registration-aided Multi-parametric MRI
Boya Wang, Ruizhe Li, Chao Chen et al. (+1 more)
Liver fibrosis poses a substantial challenge in clinical practice, emphasizing the necessity for precise liver
segmentation and accurate disease staging. Based on the CARE Liver 2025 Track 4 Challenge, this study introduces a multi-task deep learning framework developed for liver
segmentation (LiSeg) and liver fibrosis staging (LiFS) using multiparametric MRI. The LiSeg phase addresses the challenge of limited annotated images and the complexities of multi-parametric MRI data by employing a semi-supervised learning model that integrates image
segmentation and registration. By leveraging both labeled and unlabeled data, the model overcomes the difficulties introduced by domain shifts and variations across modalities. In the LiFS phase, we employed a patchbased method which allows the visualization of liver fibrosis stages based on the classification outputs. Our approach effectively handles multimodality imaging data, limited labels, and domain shifts. The proposed method has been tested by the challenge organizer on an independent test set that includes in-distribution (ID) and out-of-distribution (OOD) cases using three-channel MRIs (T1, T2, DWI) and seven-channel MRIs (T1, T2, DWI, GED1-GED4). The code is freely available. Github link: https://github.com/mileywang3061/Care-Liver
Read the full abstract ▾
All authors Boya Wang, Ruizhe Li, Chao Chen, Xin Chen
Computer Vision
TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution
Deyang Jiang, Jing Huang, Xuanle Zhao et al. (+6 more)
Effectively scaling GUI automation is essential for computer-use
agents (CUAs); however, existing work primarily focuses on scaling GUI grounding rather than the more crucial GUI planning, which requires more sophisticated data collection. In reality, the exploration process of a CUA across apps/desktops/web pages typically follows a tree structure, with earlier functional entry points often being explored more frequently. Thus, organizing large-scale trajectories into tree structures can reduce data cost and streamline the data scaling of GUI planning. In this work, we propose TreeCUA to efficiently scale GUI automation with tree-structured verifiable
evolution. We propose a multi-
agent collaborative framework to explore the environment, verify actions, summarize trajectories, and evaluate quality to generate high-quality and scalable GUI trajectories. To improve efficiency, we devise a novel tree-based
topology to store and replay duplicate exploration nodes, and design an adaptive exploration
algorithm to balance the depth (i.e., trajectory difficulty) and breadth (i.e., trajectory diversity). Moreover, we develop world knowledge guidance and global
memory backtracking to avoid low-quality generation. Finally, we naturally extend and propose the TreeCUA-DPO method from abundant tree node information, improving GUI planning capability by referring to the branch information of adjacent trajectories. Experimental results show that TreeCUA and TreeCUA-DPO offer significant improvements, and out-of-domain (OOD) studies further demonstrate strong generalization. All trajectory node information and code will be available at https://github.com/UITron-hub/TreeCUA.
Read the full abstract ▾
All authors Deyang Jiang, Jing Huang, Xuanle Zhao, Lei Chen, Liming Zheng, Fanfan Liu, Haibo Qiu, Peng Shi, Zhixiong Zeng
Computer Vision
Time2General: Learning Spatiotemporal Invariant Representations for Domain-Generalization Video Semantic Segmentation
Siyu Chen, Ting Han, Haoling Huang et al. (+5 more)
Domain Generalized
Video Semantic
Segmentation (DGVSS) is trained on a single labeled driving domain and is directly deployed on unseen domains without target labels and test-time adaptation while maintaining temporally consistent predictions over
video streams. In practice, both domain shift and temporal-sampling shift break correspondence-based propagation and fixed-stride temporal aggregation, causing severe frame-to-frame flicker even in label-stable regions. We propose Time2General, a DGVSS framework built on Stability Queries. Time2General introduces a Spatio-Temporal
Memory Decoder that aggregates multi-frame context into a clip-level spatio-temporal
memory and decodes temporally consistent per-frame masks without explicit correspondence propagation. To further suppress flicker and improve robustness to varying sampling rates, the Masked Temporal Consistency Loss is proposed to regularize temporal prediction discrepancies across different strides, and randomize training strides to expose the model to diverse temporal gaps. Extensive experiments on multiple driving
benchmarks show that Time2General achieves a substantial improvement in cross-domain accuracy and temporal stability over prior DGSS and VSS baselines while running at up to 18 FPS. Code will be released after the review process.
Read the full abstract ▾
All authors Siyu Chen, Ting Han, Haoling Huang, Chaolei Wang, Chengzheng Fu, Duxin Zhu, Guorong Cai, Jinhe Su
Computer Vision
VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Model
Hanqing Wang, Mingyu Liu, Xiaoyu Chen et al. (+9 more)
3D affordance grounding aims to highlight the actionable regions on 3D objects, which is crucial for robotic manipulation. Previous research primarily focused on learning affordance knowledge from static cues such as
language and images, which struggle to provide sufficient dynamic interaction context that can reveal temporal and causal cues. To alleviate this predicament, we collect a comprehensive
video-based 3D affordance
dataset, VIDA, which contains 38K human-object-interaction
videos covering 16 affordance types, 38 object categories, and 22K point clouds. Based on VIDA, we propose a strong baseline: VideoAfford, which activates
multimodal large
language models with additional affordance
segmentation capabilities, enabling both world knowledge
reasoning and fine-grained affordance grounding within a unified framework. To enhance action understanding capability, we leverage a latent action encoder to extract dynamic interaction priors from HOI
videos. Moreover, we introduce a spatial-aware loss function to enable VideoAfford to obtain comprehensive 3D spatial knowledge. Extensive experimental evaluations demonstrate that our model significantly outperforms well-established methods and exhibits strong open-world generalization with affordance
reasoning abilities. All
datasets and code will be publicly released to advance research in this area.
Read the full abstract ▾
All authors Hanqing Wang, Mingyu Liu, Xiaoyu Chen, Chengwei MA, Yiming Zhong, Wenti Yin, Yuhao Liu, Zhiqing Cui, Jiahao Yuan, Lu Dai, Zhiyuan Ma, Hui Xiong
Computer Vision
Towards Training-free Multimodal Hate Localisation with Large Language Models
Yueming Sun, Long Yang, Jianbo Jiao et al. (+1 more)
The proliferation of hateful content in online
videos poses severe threats to individual well-being and societal harmony. However, existing solutions for
video hate detection either rely heavily on large-scale human annotations or lack fine-grained temporal precision. In this work, we propose LELA, the first training-free Large
Language Model (LLM) based framework for hate
video localization. Distinct from state-of-the-art models that depend on supervised pipelines, LELA leverages LLMs and modality-specific captioning to detect and temporally localize hateful content in a training-free manner. Our method decomposes a
video into five modalities, including image,
speech, OCR, music, and
video context, and uses a multi-stage prompting scheme to compute fine-grained hateful scores for each frame. We further introduce a composition matching mechanism to enhance cross-modal
reasoning. Experiments on two challenging
benchmarks, HateMM and MultiHateClip, demonstrate that LELA outperforms all existing training-free baselines by a large margin. We also provide extensive ablations and qualitative visualizations, establishing LELA as a strong foundation for scalable and interpretable hate
video localization.
Read the full abstract ▾
All authors Yueming Sun, Long Yang, Jianbo Jiao, Zeyu Fu
Computer Vision · Cross-listed from cs.RO
AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception
Ruoxuan Feng, Yuxuan Zhou, Siyu Mei et al. (+6 more)
Real-world contact-rich manipulation demands
robots to perceive temporal tactile feedback, capture subtle surface deformations, and reason about object properties as well as force dynamics. Although optical tactile sensors are uniquely capable of providing such rich information, existing tactile
datasets and models remain limited. These resources primarily focus on object-level attributes (e.g., material) while largely overlooking fine-grained tactile temporal dynamics during physical interactions. We consider that advancing dynamic tactile perception requires a systematic hierarchy of dynamic perception capabilities to guide both data collection and model design. To address the lack of tactile data with rich dynamic information, we present ToucHD, a large-scale hierarchical tactile
dataset spanning tactile atomic actions, real-world manipulations, and touch-force paired data. Beyond scale, ToucHD establishes a comprehensive tactile dynamic data ecosystem that explicitly supports hierarchical perception capabilities from the data perspective. Building on it, we propose AnyTouch 2, a general tactile representation learning framework for diverse optical tactile sensors that unifies object-level understanding with fine-grained, force-aware dynamic perception. The framework captures both pixel-level and action-specific deformations across frames, while explicitly modeling physical force dynamics, thereby learning multi-level dynamic perception capabilities from the model perspective. We evaluate our model on
benchmarks that covers static object properties and dynamic physical attributes, as well as real-world manipulation tasks spanning multiple tiers of dynamic perception capabilities-from basic object-level understanding to force-aware dexterous manipulation. Experimental results demonstrate consistent and strong performance across sensors and tasks.
Read the full abstract ▾
All authors Ruoxuan Feng, Yuxuan Zhou, Siyu Mei, Dongzhan Zhou, Pengwei Wang, Shaowei Cui, Bin Fang, Guocai Yao, Di Hu
Computer Vision
AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models
Yue Li, Xin Yi, Dongsheng Shi et al. (+3 more)
Watermarking has emerged as a pivotal solution for content traceability and intellectual property protection in large
vision language models (LVLMs). However,
vision-agnostic watermarks may introduce visually irrelevant
tokens and disrupt visual grounding by enforcing indiscriminate pseudo-random biases. Additionally, current
vision-specific watermarks rely on a static, one-time estimation of
vision-critical weights and ignore the weight distribution density when determining the proportion of protected
tokens. This design fails to account for dynamic changes in visual dependence during generation and may introduce low-quality
tokens in the long tail. To address these challenges, we propose
Attention-Guided Dynamic Watermarking (AGMark), a novel framework that embeds detectable signals while largely preserving visual-semantic fidelity. At each decoding step, AGMark first dynamically identifies semantic-critical evidence based on
attention weights for visual relevance, together with context-aware coherence cues, resulting in a more adaptive and well-calibrated evidence-weight distribution. It then determines the proportion of semantic-critical
tokens by jointly considering uncertainty awareness (
token entropy) and evidence calibration (weight density), thereby enabling more reliable adaptive vocabulary partitioning to avoid irrelevant
tokens. Empirical results consistently confirm that AGMark outperforms conventional methods, substantially improving generation quality and yielding particularly strong gains in visual semantic fidelity in the later stages of generation. Our framework maintains highly competitive detection performance (at least 99.36% AUC) and robust attack resilience (at least 88.61% AUC) without sacrificing inference efficiency, taking a significant step toward reliability-preserving
multimodal watermarking.
Read the full abstract ▾
All authors Yue Li, Xin Yi, Dongsheng Shi, Yongyi Cui, Gerard de Melo, Linlin Wang
Computer Vision
Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing
Jialun Liu, Tian Li, Xiao Cao et al. (+20 more)
Recent advances in
diffusion-based
video generation have substantially improved visual fidelity and temporal coherence. However, most existing approaches remain task-specific and rely primarily on textual instructions, limiting their ability to handle
multimodal inputs, contextual references, and diverse
video generation and editing scenarios within a unified framework. Moreover, many
video editing methods depend on carefully engineered pipelines tailored to individual operations, which hinders scalability and composability. In this paper, we propose Tele-Omni, a unified
multimodal framework for
video generation and editing that follows
multimodal instructions, including text, images, and reference
videos, within a single model. Tele-Omni leverages pretrained
multimodal large
language models to parse heterogeneous instructions and infer structured generation or editing intents, while
diffusion-based generators perform high-quality
video synthesis conditioned on these structured signals. To enable joint training across heterogeneous
video tasks, we introduce a task-aware data processing pipeline that unifies
multimodal inputs into a structured instruction format while preserving task-specific constraints. Tele-Omni supports a wide range of
video-centric tasks, including text-to-
video generation, image-to-
video generation, first-last-frame
video generation, in-context
video generation, and in-context
video editing. By decoupling instruction parsing from
video synthesis and combining it with task-aware data design, Tele-Omni achieves flexible
multimodal control while maintaining strong temporal coherence and visual consistency. Experimental results demonstrate that Tele-Omni achieves competitive performance across multiple tasks.
Read the full abstract ▾
All authors Jialun Liu, Tian Li, Xiao Cao, Yukuo Ma, Gonghu Shang, Haibin Huang, Chi Zhang, Xiangzhen Chang, Zhiyong Huang, Jiakui Hu, Zuoxin Li, Yuanzhi Liang, Cong Liu, Junqi Liu, Robby T. Tan, Haitong Tang, Qizhen Weng, Yifan Xu, Liying Yang, Xiaoyan Yang, Peng Yu, Shiwen Zhang, Xuelong Li
Computer Vision
Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures
Yuxi Wang, Wenqi Ouyang, Tianyi Wei et al. (+3 more)
Egocentric interactive world models are essential for augmented reality and
embodied AI, where visual generation must respond to user input with low latency,
geometric consistency, and long-term stability. We study
egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic
videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion. This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in
monocular views, and the need for arbitrary-length
video generation. We present Hand2World, a unified autoregressive framework that addresses these challenges through occlusion-invariant hand conditioning based on projected 3D hand meshes, allowing visibility and occlusion to be inferred from scene context rather than encoded in the control signal. To stabilize
egocentric viewpoint changes, we inject explicit camera geometry via per-pixel Plücker-ray
embeddings, disentangling camera motion from hand motion and preventing background drift. We further develop a fully automated
monocular annotation pipeline and distill a bidirectional
diffusion model into a causal generator, enabling arbitrary-length synthesis. Experiments on three
egocentric interaction
benchmarks show substantial improvements in perceptual quality and 3D consistency while supporting camera control and long-horizon interactive generation.
Read the full abstract ▾
All authors Yuxi Wang, Wenqi Ouyang, Tianyi Wei, Yi Dong, Zhiqi Shen, Xingang Pan
Computer Vision
MieDB-100k: A Comprehensive Dataset for Medical Image Editing
Yongfan Lai, Wen Qian, Bo Liu et al. (+5 more)
The scarcity of high-quality data remains a primary bottleneck in adapting
multimodal generative models for medical image editing. Existing medical image editing
datasets often suffer from limited diversity, neglect of medical image understanding and inability to balance quality with scalability. To address these gaps, we propose MieDB-100k, a large-scale, high-quality and diverse
dataset for text-guided medical image editing. It categorizes editing tasks into perspectives of Perception, Modification and Transformation, considering both understanding and generation abilities. We construct MieDB-100k via a data curation pipeline leveraging both modality-specific expert models and rule-based data synthetic methods, followed by rigorous manual inspection to ensure clinical fidelity. Extensive experiments demonstrate that model trained with MieDB-100k consistently outperform both open-source and proprietary models while exhibiting strong generalization ability. We anticipate that this
dataset will serve as a cornerstone for future advancements in specialized medical image editing.
Read the full abstract ▾
All authors Yongfan Lai, Wen Qian, Bo Liu, Hongyan Li, Hao Luo, Fan Wang, Bohan Zhuang, Shenda Hong
Computer Vision
Delving into Spectral Clustering with Vision-Language Representations
Bo Peng, Yuanwei Hu, Bo Liu et al. (+3 more)
Spectral clustering is known as a powerful technique in unsupervised data analysis. The vast majority of approaches to spectral clustering are driven by a single modality, leaving the rich information in multi-modal representations untapped. Inspired by the recent success of
vision-
language pre-training, this paper enriches the landscape of spectral clustering from a single-modal to a multi-modal regime. Particularly, we propose Neural Tangent Kernel Spectral Clustering that leverages cross-modal
alignment in pre-trained
vision-
language models. By anchoring the neural tangent kernel with positive nouns, i.e., those semantically close to the images of interest, we arrive at formulating the affinity between images as a coupling of their visual proximity and semantic overlap. We show that this formulation amplifies within-cluster connections while suppressing spurious ones across clusters, hence encouraging block-diagonal structures. In addition, we present a regularized affinity
diffusion mechanism that adaptively ensembles affinity
matrices induced by different
prompts. Extensive experiments on 16
benchmarks – including classical, large-scale, fine-grained and domain-shifted
datasets – manifest that our method consistently outperforms the state-of-the-art by a large margin.
Read the full abstract ▾
All authors Bo Peng, Yuanwei Hu, Bo Liu, Ling Chen, Jie Lu, Zhen Fang
Computer Vision · Cross-listed from cs.LG
ECG-IMN: Interpretable Mesomorphic Neural Networks for 12-Lead Electrocardiogram Interpretation
Vajira Thambawita, Jonas L. Isaksen, Jørgen K. Kanters et al. (+2 more)
Deep learning has achieved expert-level performance in automated electrocardiogram (ECG) diagnosis, yet the "black-box" nature of these models hinders their clinical deployment. Trust in medical AI requires not just high accuracy but also transparency regarding the specific physiological features driving predictions. Existing explainability methods for ECGs typically rely on post-hoc approximations (e.g., Grad-CAM and SHAP), which can be unstable, computationally expensive, and unfaithful to the model's actual decision-making process. In this work, we propose the ECG-IMN, an Interpretable Mesomorphic Neural Network tailored for high-resolution 12-lead ECG classification. Unlike standard classifiers, the ECG-IMN functions as a hypernetwork: a deep convolutional backbone generates the parameters of a strictly linear model specific to each input sample. This architecture enforces intrinsic interpretability, as the decision logic is mathematically transparent and the generated weights (W) serve as exact, high-resolution feature attribution maps. We introduce a transition decoder that effectively maps latent features to sample-wise weights, enabling precise localization of pathological evidence (e.g., ST-elevation, T-wave inversion) in both time and lead dimensions. We evaluate our approach on the PTB-XL
dataset for classification tasks, demonstrating that the ECG-IMN achieves competitive predictive performance (AUROC comparable to black-box baselines) while providing faithful, instance-specific explanations. By explicitly decoupling parameter generation from prediction execution, our framework bridges the gap between deep learning capability and clinical trustworthiness, offering a principled path toward "white-box" cardiac diagnostics.
Read the full abstract ▾
All authors Vajira Thambawita, Jonas L. Isaksen, Jørgen K. Kanters, Hugo L. Hammer, Pål Halvorsen
Computer Vision
Ziqiang Shi, Rujie Liu, Shanshan Yu et al. (+2 more)
Rapid progress in large
vision-
language models (LVLMs) has achieved unprecedented performance in
vision-
language tasks. However, due to the strong prior of large
language models (LLMs) and misaligned
attention across modalities, LVLMs often generate outputs inconsistent with visual content - termed
hallucination. To address this, we propose Scalpel, a method that reduces
hallucination by refining
attention activation distributions toward more credible regions. Scalpel predicts trusted
attention directions for each head in
Transformer layers during inference and adjusts activations accordingly. It employs a Gaussian mixture model to capture multi-peak distributions of
attention in trust and
hallucination manifolds, and uses entropic optimal transport (equivalent to Schrödinger bridge problem) to map Gaussian components precisely. During mitigation, Scalpel dynamically adjusts intervention strength and direction based on component membership and mapping relationships between
hallucination and trust activations. Extensive experiments across multiple
datasets and
benchmarks demonstrate that Scalpel effectively mitigates
hallucinations, outperforming previous methods and achieving state-of-the-art performance. Moreover, Scalpel is model- and data-agnostic, requiring no additional computation, only a single decoding step.
Read the full abstract ▾
All authors Ziqiang Shi, Rujie Liu, Shanshan Yu, Satoshi Munakata, Koichi Shirahata
Computer Vision
AUHead: Realistic Emotional Talking Head Generation via Action Units Control
Jiayi Lyu, Leigang Qu, Wenjing Zhang et al. (+6 more)
Realistic talking-head
video generation is critical for virtual avatars, film production, and interactive systems. Current methods struggle with nuanced emotional expressions due to the lack of fine-grained
emotion control. To address this issue, we introduce a novel two-stage method (AUHead) to disentangle fine-grained
emotion control, i.e. , Action Units (AUs), from audio and achieve controllable generation. In the first stage, we explore the AU generation abilities of large audio-
language models (ALMs), by spatial-temporal AU tokenization and an "
emotion-then-AU" chain-of-thought mechanism. It aims to disentangle AUs from raw
speech, effectively capturing subtle emotional cues. In the second stage, we propose an AU-driven controllable
diffusion model that synthesizes realistic talking-head
videos conditioned on AU sequences. Specifically, we first map the AU sequences into the structured 2D facial representation to enhance spatial fidelity, and then model the AU-
vision interaction within cross-
attention modules. To achieve flexible AU-quality trade-off control, we introduce an AU disentanglement guidance strategy during inference, further refining the emotional expressiveness and identity consistency of the generated
videos. Results on
benchmark datasets demonstrate that our approach achieves competitive performance in emotional realism, accurate lip synchronization, and visual coherence, significantly surpassing existing techniques. Our implementation is available at https://github.com/laura990501/AUHead_ICLR
Read the full abstract ▾
All authors Jiayi Lyu, Leigang Qu, Wenjing Zhang, Hanyu Jiang, Kai Liu, Zhenglin Zhou, Xiaobo Xia, Jian Xue, Tat-Seng Chua
Computer Vision
RAD: Retrieval-Augmented Monocular Metric Depth Estimation for Underrepresented Classes
Michael Baltaxe, Dan Levi and Sagie Benaim
Monocular Metric Depth Estimation (MMDE) is essential for physically intelligent systems, yet accurate depth estimation for underrepresented classes in complex scenes remains a persistent challenge. To address this, we propose RAD, a
retrieval-augmented framework that approximates the benefits of multi-view stereo by utilizing retrieved neighbors as structural
geometric proxies. Our method first employs an uncertainty-aware
retrieval mechanism to identify low-confidence regions in the input and retrieve RGB-D context samples containing semantically similar content. We then process both the input and retrieved context via a dual-stream network and fuse them using a matched cross-
attention module, which transfers
geometric information only at reliable point correspondences. Evaluations on NYU Depth v2, KITTI, and Cityscapes demonstrate that RAD significantly outperforms state-of-the-art baselines on underrepresented classes, reducing relative absolute error by 29.2% on NYU Depth v2, 13.3% on KITTI, and 7.2% on Cityscapes, while maintaining competitive performance on standard in-domain
benchmarks.
Read the full abstract ▾
Computer Vision
DR.Experts: Differential Refinement of Distortion-Aware Experts for Blind Image Quality Assessment
Bohan Fu, Guanyi Qin, Fazhan Zhang et al. (+3 more)
Blind Image Quality Assessment, aiming to replicate human perception of visual quality without reference, plays a key role in
vision tasks, yet existing models often fail to effectively capture subtle distortion cues, leading to a misalignment with human subjective judgments. We identify that the root cause of this limitation lies in the lack of reliable distortion priors, as methods typically learn shallow relationships between unified image features and quality scores, resulting in their insensitive nature to distortions and thus limiting their performance. To address this, we introduce DR.Experts, a novel prior-driven BIQA framework designed to explicitly incorporate distortion priors, enabling a reliable quality assessment. DR.Experts begins by leveraging a degradation-aware
vision-
language model to obtain distortion-specific priors, which are further refined and enhanced by the proposed Distortion-Saliency Differential Module through distinguishing them from semantic attentions, thereby ensuring the genuine representations of distortions. The refined priors, along with semantics and bridging representation, are then fused by a proposed mixture-of-experts style module named the Dynamic Distortion Weighting Module. This mechanism weights each distortion-specific feature as per its perceptual impact, ensuring that the final quality prediction aligns with human perception. Extensive experiments conducted on five challenging BIQA
benchmarks demonstrate the superiority of DR.Experts over current methods and showcase its excellence in terms of generalization and data efficiency.
Read the full abstract ▾
All authors Bohan Fu, Guanyi Qin, Fazhan Zhang, Zihao Huang, Mingxuan Li, Runze Hu
Computer Vision
SCA-Net: Spatial-Contextual Aggregation Network for Enhanced Small Building and Road Change Detection
Emad Gholibeigi, Abbas Koochari and Azadeh ZamaniFar
Automated change detection in remote sensing imagery is critical for urban management, environmental monitoring, and disaster assessment. While deep learning models have advanced this field, they often struggle with challenges like low sensitivity to small objects and high computational costs. This paper presents SCA-Net, an enhanced architecture built upon the Change-
Agent framework for precise building and road change detection in bi-temporal images. Our model incorporates several key innovations: a novel Difference Pyramid Block for multi-scale change analysis, an Adaptive Multi-scale Processing module combining shape-aware and high-resolution enhancement blocks, and multi-level
attention mechanisms (PPM and CSAGate) for joint contextual and detail processing. Furthermore, a dynamic composite loss function and a four-phase training strategy are introduced to stabilize training and accelerate convergence. Comprehensive evaluations on the LEVIR-CD and LEVIR-MCI
datasets demonstrate SCA-Net's superior performance over Change-
Agent and other state-of-the-art methods. Our approach achieves a significant 2.64% improvement in mean Intersection over Union (mIoU) on LEVIR-MCI and a remarkable 57.9% increase in IoU for small buildings, while reducing the training time by 61%. This work provides an efficient, accurate, and robust solution for practical change detection applications.
Read the full abstract ▾
Computer Vision
SchröMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schrödinger Bridge Problem
Ziqiang Shi, Rujie Liu, Shanshan Yu et al. (+2 more)
Recent advancements in
Multimodal Large
Language Models (MLLMs) have achieved significant success across various domains. However, their use in high-stakes fields like healthcare remains limited due to persistent
hallucinations, where generated text contradicts or ignores visual input. We contend that MLLMs can comprehend images but struggle to produce accurate
token sequences. Minor perturbations can shift
attention from truthful to untruthful states, and the autoregressive nature of text generation often prevents error correction. To address this, we propose SchröMind-a novel framework reducing
hallucinations via solving the Schrödinger bridge problem. It establishes a
token-level mapping between hallucinatory and truthful activations with minimal transport cost through lightweight training, while preserving the model's original capabilities. Extensive experiments on the POPE and MME
benchmarks demonstrate the superiority of Schrödinger, which achieves state-of-the-art performance while introducing only minimal computational overhead.
Read the full abstract ▾
All authors Ziqiang Shi, Rujie Liu, Shanshan Yu, Satoshi Munakata, Koichi Shirahata