Computer Vision
Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding
Yang Du, Siyuan Dai, Yonghao Song et al. (+3 more)
Neural visual decoding is a central problem in
brain-computer interface research, aiming to reconstruct human visual perception and to elucidate the structure of neural representations. Recent contrastive neural visual decoding methods commonly align neural signals with the final
embeddings of pretrained
vision encoders. However, such representations are optimized for high-level semantic invariance, whereas
EEG/MEG signals contain information spanning multiple levels of visual abstraction, potentially creating a representational granularity mismatch. Motivated by prior evidence that
brain representations correspond to multiple levels of the DNN hierarchy, we propose Shallow
Alignment, a granularity-calibration framework that systematically explores intermediate visual representations as
alignment targets for neural decoding. Extensive experiments across multiple
benchmarks demonstrate that Shallow
Alignment significantly outperforms standard final-layer
alignment, with performance gains ranging from 22% to 58% across diverse
vision backbones. Notably, our approach reveals a positive scaling trend in neural visual decoding, enabling decoding performance to improve consistently with the capacity of pre-trained
vision backbones. We further conduct systematic empirical analyses to shed light on the mechanisms underlying the observed performance gains. Code is available at https://github.com/yangdu-neuroai/shallow-
alignment.
Read the full abstract ▾
All authors Yang Du, Siyuan Dai, Yonghao Song, Paul M. Thompson, Haoteng Tang, Liang Zhan
Computer Vision
BookNet: Book Image Rectification via Cross-Page Attention Network
Shaokai Liu, Hao Feng, Bozhi Luan et al. (+3 more)
Book image rectification presents unique challenges due to its complex
geometric distortions from binding constraints, where left and right pages exhibit distinctly asymmetric
curvature patterns. However, existing single-page document image rectification methods fail to capture the coupled
geometric relationships between adjacent pages in books. In this work, we introduce BookNet, the first end-to-end deep learning framework specifically designed for dual-page book image rectification. BookNet adopts a dual-branch architecture with cross-page
attention mechanisms, enabling it to estimate warping flows for both \rev{individual pages and the book spread}, explicitly modeling how left and right pages influence each other. Moreover, to address the absence of specialized
datasets, we present Book3D, a large-scale synthetic
dataset for training, and Book100, a comprehensive real-world
benchmark for evaluation. Extensive experiments demonstrate that BookNet outperforms existing state-of-the-art methods on book image rectification. \rev{Code and
dataset are available at https://github.com/Liu-Shaokai/BookNet.
Read the full abstract ▾
All authors Shaokai Liu, Hao Feng, Bozhi Luan, Min Hou, Jiajun Deng, Wengang Zhou
Computer Vision
Just Noticeable Difference Modeling for Deep Visual Features
Rui Zhao, Wenrui Li, Lin Zhu et al. (+2 more)
Deep visual features are increasingly used as the interface in
vision systems, motivating the need to describe feature characteristics and control feature quality for machine perception. Just noticeable difference (JND) characterizes the maximum imperceptible distortion for images under human or machine
vision. Extending it to deep visual features naturally meets the above demand by providing a task-aligned tolerance boundary in feature space, offering a practical reference for controlling feature quality under constrained resources. We propose FeatJND, a task-aligned JND formulation that predicts the maximum tolerable per-feature perturbation map while preserving downstream task performance. We propose a FeatJND
estimator at standardized split points and validate it across image classification, detection, and instance
segmentation. Under matched distortion strength, FeatJND-based distortions consistently preserve higher task performance than unstructured Gaussian perturbations, and attribution visualizations suggest FeatJND can suppress non-critical feature regions. As an application, we further apply FeatJND to
token-wise dynamic quantization and show that FeatJND-guided step-size allocation yields clear gains over random step-size permutation and global uniform step size under the same noise budget. Our code will be released after publication.
Read the full abstract ▾
All authors Rui Zhao, Wenrui Li, Lin Zhu, Yajing Zheng, Weisi Lin
Computer Vision
Zero-Shot Video Restoration and Enhancement with Assistance of Video Diffusion Models
Cong Cao, Huanjing Yue, Shangbin Xie et al. (+2 more)
Although
diffusion-based zero-shot image restoration and enhancement methods have achieved great success, applying them to
video restoration or enhancement will lead to severe temporal flickering. In this paper, we propose the first framework that utilizes the rapidly-developed
video diffusion model to assist the image-based method in maintaining more temporal consistency for zero-shot
video restoration and enhancement. We propose homologous latents fusion, heterogenous latents fusion, and a COT-based fusion ratio strategy to utilize both homologous and heterogenous text-to-
video diffusion models to complement the image method. Moreover, we propose temporal-strengthening post-processing to utilize the image-to-
video diffusion model to further improve temporal consistency. Our method is training-free and can be applied to any
diffusion-based image restoration and enhancement methods. Experimental results demonstrate the superiority of the proposed method.
Read the full abstract ▾
All authors Cong Cao, Huanjing Yue, Shangbin Xie, Xin Liu, Jingyu Yang
Computer Vision
VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models
Yunhao Li, Sijing Wu, Zhilin Gao et al. (+5 more)
Large
multimodal models (LMMs) have demonstrated outstanding capabilities in various visual perception tasks, which has in turn made the evaluation of LMMs significant. However, the capability of
video aesthetic quality assessment, which is a fundamental ability for human, remains underexplored for LMMs. To address this, we introduce VideoAesBench, a comprehensive
benchmark for evaluating LMMs' understanding of
video aesthetic quality. VideoAesBench has several significant characteristics: (1) Diverse content including 1,804
videos from multiple
video sources including user-generated (UGC), AI-generated (AIGC), compressed, robotic-generated (RGC), and game
videos. (2) Multiple question formats containing traditional single-choice questions, multi-choice questions, True or False questions, and a novel open-ended questions for
video aesthetics description. (3) Holistic
video aesthetics dimensions including visual form related questions from 5 aspects, visual style related questions from 4 aspects, and visual affectiveness questions from 3 aspects. Based on VideoAesBench, we
benchmark 23 open-source and commercial large
multimodal models. Our findings show that current LMMs only contain basic
video aesthetics perception ability, their performance remains incomplete and imprecise. We hope our VideoAesBench can be served as a strong testbed and offer insights for explainable
video aesthetics assessment. The data will be released on https://github.com/michaelliyunhao/VideoAesBench
Read the full abstract ▾
All authors Yunhao Li, Sijing Wu, Zhilin Gao, Zicheng Zhang, Qi Jia, Huiyu Duan, Xiongkuo Min, Guangtao Zhai
Computer Vision
Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learning
Hanmo Chen, Guangtao Lyu, Chenghao Xu et al. (+3 more)
As a foundational task in human-centric cross-modal intelligence, motion-
language retrieval aims to bridge the semantic gap between natural
language and human motion, enabling intuitive motion analysis, yet existing approaches predominantly focus on aligning entire motion sequences with global textual representations. This global-centric paradigm overlooks fine-grained interactions between local motion segments and individual body joints and text
tokens, inevitably leading to suboptimal
retrieval performance. To address this limitation, we draw inspiration from the pyramidal process of human motion perception (from joint dynamics to segment coherence, and finally to holistic comprehension) and propose a novel Pyramidal Shapley-Taylor (PST) learning framework for fine-grained motion-
language retrieval. Specifically, the framework decomposes human motion into temporal segments and spatial body joints, and learns cross-modal correspondences through progressive joint-wise and segment-wise
alignment in a pyramidal fashion, effectively capturing both local semantic details and hierarchical structural relationships. Extensive experiments on multiple public
benchmark datasets demonstrate that our approach significantly outperforms state-of-the-art methods, achieving precise
alignment between motion segments and body joints and their corresponding text
tokens. The code of this work will be released upon acceptance.
Read the full abstract ▾
All authors Hanmo Chen, Guangtao Lyu, Chenghao Xu, Jiexi Yan, Xu Yang, Cheng Deng
Computer Vision
TraceRouter: Robust Safety for Large Foundation Models via Path-Level Intervention
Chuancheng Shi, Shangze Li, Wenjun Lu et al. (+5 more)
Despite their capabilities, large foundation models (LFMs) remain susceptible to
adversarial manipulation. Current defenses predominantly rely on the "locality hypothesis", suppressing isolated
neurons or features. However, harmful semantics act as distributed, cross-layer
circuits, rendering such localized interventions brittle and detrimental to utility. To bridge this gap, we propose TraceRouter, a path-level framework that traces and disconnects the causal propagation
circuits of illicit semantics. TraceRouter operates in three stages: (1) it pinpoints a sensitive onset layer by analyzing
attention divergence; (2) it leverages sparse autoencoders (SAEs) and differential activation analysis to disentangle and isolate malicious features; and (3) it maps these features to downstream causal pathways via feature influence scores (FIS) derived from zero-out interventions. By selectively suppressing these causal chains, TraceRouter physically severs the flow of harmful information while leaving orthogonal computation routes intact. Extensive experiments demonstrate that TraceRouter significantly outperforms state-of-the-art baselines, achieving a superior trade-off between
adversarial robustness and general utility. Our code will be publicly released. WARNING: This paper contains unsafe model responses.
Read the full abstract ▾
All authors Chuancheng Shi, Shangze Li, Wenjun Lu, Wenhua Wu, Cong Wang, Zifeng Cheng, Fei Shen, Tat-Seng Chua
Computer Vision
Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusion
Hanmo Chen, Chenghao Xu, Xu Yang et al. (+2 more)
Video generation is pivotal to digital media creation, and recent advances in autoregressive
video generation have markedly enhanced the efficiency of real-time
video synthesis. However, existing approaches generally rely on heuristic KV Cache policies, which ignore differences in
token importance in long-term
video generation. This leads to the loss of critical spatiotemporal information and the accumulation of redundant, invalid cache, thereby degrading
video generation quality and efficiency. To address this limitation, we first observe that
token contributions to
video generation are highly time-heterogeneous and accordingly propose a novel Past- and Future-Informed KV Cache Policy (PaFu-KV). Specifically, PaFu-KV introduces a lightweight Salience Estimation Head distilled from a bidirectional teacher to estimate salience scores, allowing the KV cache to retain informative
tokens while discarding less relevant ones. This policy yields a better quality-efficiency trade-off by shrinking KV cache capacity and reducing
memory footprint at inference time. Extensive experiments on
benchmarks demonstrate that our method preserves high-fidelity
video generation quality while enables accelerated inference, thereby enabling more efficient long-horizon
video generation. Our code will be released upon paper acceptance.
Read the full abstract ▾
All authors Hanmo Chen, Chenghao Xu, Xu Yang, Xuan Chen, Cheng Deng
Computer Vision
Improving Classifier-Free Guidance of Flow Matching via Manifold Projection
Jian-Feng Cai, Haixia Liu, Zhengyi Su et al. (+1 more)
Classifier-free guidance (CFG) is a widely used technique for controllable generation in
diffusion and flow-based models. Despite its empirical success, CFG relies on a heuristic linear extrapolation that is often sensitive to the guidance scale. In this work, we provide a principled interpretation of CFG through the lens of optimization. We demonstrate that the velocity field in flow matching corresponds to the gradient of a sequence of smoothed distance functions, which guides latent variables toward the scaled target image set. This perspective reveals that the standard CFG formulation is an approximation of this gradient, where the prediction gap, the discrepancy between conditional and unconditional outputs, governs guidance sensitivity. Leveraging this insight, we reformulate the CFG sampling as a homotopy optimization with a
manifold constraint. This formulation necessitates a
manifold projection step, which we implement via an incremental gradient descent scheme during sampling. To improve computational efficiency and stability, we further enhance this iterative process with Anderson Acceleration without requiring additional model evaluations. Our proposed methods are training-free and consistently refine generation fidelity,
prompt alignment, and robustness to the guidance scale. We validate their effectiveness across diverse
benchmarks, demonstrating significant improvements on large-scale models such as DiT-XL-2-256, Flux, and Stable
Diffusion 3.5.
Read the full abstract ▾
All authors Jian-Feng Cai, Haixia Liu, Zhengyi Su, Chao Wang
Computer Vision
Dynamics-Inspired Diffusion for Foreground-Preserving Document Background Editing
Taewon Kang, Yu Shen and Ming C. Lin
We revisit
diffusion-based generation for structured visual content and identify a fundamental limitation of existing approaches: foreground preservation and background stylization are typically enforced through external interventions, such as hard masking or corrective post-processing, rather than arising from the generative process itself. Here, we define background as the generative content outside designated foreground regions (e.g., text and layout elements), while preserving the structural integrity of the foreground. We propose a dynamical systems perspective on
diffusion, in which controllable generation is formulated as trajectory shaping in latent space. Under this view, we introduce Auxiliary Context
Diffusion (ACD), a state-space control framework that integrates heterogeneous signals (layout-derived foreground indicators, document summaries, and style representations) directly into the
diffusion dynamics. This formulation induces time-scale separation in the generative process, where foreground regions become dynamically stabilized while background regions remain expressive. To address stylistic drift across multi-page documents, we further introduce style directions as persistent latent constraints that guide
diffusion trajectories within a shared stylistic subspace. Unlike prior approaches that entangle style with
prompt conditioning, our formulation enables reusable and consistent style control across pages. We validate the proposed perspective through controlled experiments on synthetic document
benchmarks, demonstrating that trajectory-level control provides a unified and extensible mechanism for structured generation without retraining, hard masking, or corrective post-processing. These results suggest a new direction for controllable
diffusion in document-centric and
multimodal applications.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Blind Ultrasound Image Enhancement via Self-Supervised Physics-Guided Degradation Modeling
Shujaat Khan, Syed Muhammad Atif, Jaeyoung Huh et al. (+1 more)
Ultrasound (US) interpretation is hampered by multiplicative speckle, acquisition blur from the point-spread function (PSF), and scanner- and operator-dependent artifacts. Supervised enhancement methods assume access to clean targets or known degradations; conditions rarely met in practice. We present a blind, self-supervised enhancement framework that jointly deconvolves and denoises B-mode images using a Swin Convolutional U-Net trained with a physics-guided degradation model. From each training frame, we extract rotated/cropped patches and synthesize inputs by (i) convolving with a Gaussian PSF surrogate and (ii) injecting noise via either spatial additive Gaussian noise or complex
Fourier-domain perturbations that emulate phase/magnitude distortions. For US scans, clean-like targets are obtained via non-local low-rank (NLLR) denoising, removing the need for ground truth; for natural images, the originals serve as targets. Trained and validated on UDIAT~B, JNU-IFM, and XPIE Set-P, and evaluated additionally on a 700-image PSFHS test set, the method achieves the highest PSNR/SSIM across Gaussian and speckle noise levels, with margins that widen under stronger corruption. Relative to MSANN, Restormer, and DnCNN, it typically preserves an extra $\sim$1–4\,dB PSNR and 0.05–0.15 SSIM in heavy Gaussian noise, and $\sim$2–5\,dB PSNR and 0.05–0.20 SSIM under severe speckle. Controlled PSF studies show reduced FWHM and higher peak gradients, evidence of resolution recovery without edge erosion. Used as a plug-and-play preprocessor, it consistently boosts Dice for fetal head and pubic symphysis
segmentation. Overall, the approach offers a practical, assumption-light path to robust US enhancement that generalizes across
datasets, scanners, and degradation types.
Read the full abstract ▾
All authors Shujaat Khan, Syed Muhammad Atif, Jaeyoung Huh, Syed Saad Azhar
Computer Vision
MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods
Honglin Lin, Zheng Liu, Yun Zhu et al. (+6 more)
Recent advances in
Vision Language Models (VLMs) have driven significant progress in visual
reasoning. However, open-source VLMs still lag behind proprietary systems, largely due to the lack of high-quality
reasoning data. Existing
datasets offer limited coverage of challenging domains such as STEM diagrams and visual puzzles, and lack consistent, long-form Chain-of-Thought (CoT) annotations essential for eliciting strong
reasoning capabilities. To bridge this gap, we introduce MMFineReason, a large-scale
multimodal reasoning dataset comprising 1.8M samples and 5.1B solution
tokens, featuring high-quality
reasoning annotations distilled from Qwen3-VL-235B-A22B-Thinking. The
dataset is established via a systematic three-stage pipeline: (1) large-scale data collection and standardization, (2) CoT rationale generation, and (3) comprehensive selection based on
reasoning quality and difficulty awareness. The resulting
dataset spans STEM problems, visual puzzles, games, and complex diagrams, with each sample annotated with visually grounded
reasoning traces. We fine-tune Qwen3-VL-Instruct on MMFineReason to develop MMFineReason-2B/4B/8B versions. Our models establish new state-of-the-art results for their size class. Notably, MMFineReason-4B succesfully surpasses Qwen3-VL-8B-Thinking, and MMFineReason-8B even outperforms Qwen3-VL-30B-A3B-Thinking while approaching Qwen3-VL-32B-Thinking, demonstrating remarkable parameter efficiency. Crucially, we uncover a "less is more" phenomenon via our difficulty-aware filtering strategy: a subset of just 7% (123K samples) achieves performance comparable to the full
dataset. Notably, we reveal a synergistic effect where
reasoning-oriented data composition simultaneously boosts general capabilities.
Read the full abstract ▾
All authors Honglin Lin, Zheng Liu, Yun Zhu, Chonghan Qin, Juekai Lin, Xiaoran Shang, Conghui He, Wentao Zhang, Lijun Wu
Computer Vision
CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models
Junming Huang, Chi Wang, Letian Li et al. (+5 more)
Large
Language Models(LLMs) have revolutionized text generation and
multimodal perception,but their capabilities in 3D content generation remain underexplored. Existing methods compromise by producing either low-resolution meshes or coarse structural proxies, failing to capture finegrained geometry natively. In this paper, we propose CG-MLLM, a novel Multi-modal Large
Language Model (MLLM) capable of 3D captioning and high-resolution 3D generation in a single framework. Leveraging the Mixture-ofTransformer architecture, CG-MLLM decouples disparate modeling needs, where the
Token-level Autoregressive (TokenAR)
Transformer handles
token-level content, and the Block-level Autoregressive (BlockAR)
Transformer handles blocklevel content. By integrating a pre-trained visionlanguage backbone with a specialized 3D VAE latent space, CG-MLLM facilitates long-context interactions between standard
tokens and spatial blocks within a single integrated architecture. Experimental results show that CG-MLLM significantly outperforms existing MLLMs in generating high-fidelity 3D objects, effectively bringing high-resolution 3D content creation into the mainstream LLM paradigm. Beyond generation, we further observe that learning to produce 3D content transfers back to perception, strengthening the model's image-based 3D understanding.
Read the full abstract ▾
All authors Junming Huang, Chi Wang, Letian Li, Guangkai Xu, Donglin Huang, Hao Chen, Qiang Dai, Weiwei Xu
Computer Vision
Synthetic-to-Real Domain Bridging for Single-View 3D Reconstruction of Ships for Maritime Monitoring
Borja Carrillo-Perez, Felix Sattler, Angel Bueno Rodriguez et al. (+2 more)
Three-dimensional (3D) reconstruction of ships is an important part of maritime monitoring, allowing improved visualization, inspection, and decision-making in real-world monitoring environments. However, most state-ofthe-art 3D reconstruction methods require multi-view supervision, annotated 3D ground truth, or are computationally intensive, making them impractical for real-time maritime deployment. In this work, we present an efficient pipeline for single-view 3D reconstruction of real ships by training entirely on synthetic data and requiring only a single view at inference. Our approach uses the Splatter Image network, which represents objects as sparse sets of 3D Gaussians for rapid and accurate reconstruction from single images. The model is first fine-tuned on synthetic ShapeNet vessels and further refined with a diverse custom
dataset of 3D ships, bridging the domain gap between synthetic and real-world imagery. We integrate a state-of-the-art
segmentation module based on YOLOv8 and custom preprocessing to ensure compatibility with the reconstruction network. Postprocessing steps include real-world scaling, centering, and orientation
alignment, followed by georeferenced placement on an interactive web map using AIS metadata and homography-based mapping. Quantitative evaluation on synthetic validation data demonstrates strong reconstruction fidelity, while qualitative results on real maritime images from the ShipSG
dataset confirm the potential for transfer to operational maritime settings. The final system provides interactive 3D inspection of real ships without requiring real-world 3D annotations. This pipeline provides an efficient, scalable solution for maritime monitoring and highlights a path toward real-time 3D ship visualization in practical applications. Interactive demo: https://dlr-mi.github.io/ship3d-demo/.
Read the full abstract ▾
All authors Borja Carrillo-Perez, Felix Sattler, Angel Bueno Rodriguez, Maurice Stephan, Sarah Barnes
Computer Vision
Jiankun Peng, Jianyuan Guo, Ying Xu et al. (+5 more)
Vision-
Language Navigation in Continuous Environments (VLN-CE) presents a core challenge: grounding high-level linguistic instructions into precise, safe, and long-horizon spatial actions. Explicit topological maps have proven to be a vital solution for providing robust spatial
memory in such tasks. However, existing topological planning methods suffer from a "Granularity
Rigidity" problem. Specifically, these methods typically rely on fixed
geometric thresholds to sample nodes, which fails to adapt to varying environmental complexities. This
rigidity leads to a critical mismatch: the model tends to over-sample in simple areas, causing computational redundancy, while under-sampling in high-uncertainty regions, increasing collision risks and compromising precision. To address this, we propose DGNav, a framework for Dynamic Topological
Navigation, introducing a context-aware mechanism to modulate map density and connectivity on-the-fly. Our approach comprises two core innovations: (1) A Scene-Aware Adaptive Strategy that dynamically modulates graph construction thresholds based on the dispersion of predicted waypoints, enabling "densification on demand" in challenging environments; (2) A Dynamic Graph
Transformer that reconstructs graph connectivity by fusing visual, linguistic, and
geometric cues into dynamic edge weights, enabling the
agent to filter out topological noise and enhancing instruction adherence. Extensive experiments on the R2R-CE and RxR-CE
benchmarks demonstrate DGNav exhibits superior
navigation performance and strong generalization capabilities. Furthermore, ablation studies confirm that our framework achieves an optimal trade-off between
navigation efficiency and safe exploration. The code is available at https://github.com/shannanshouyin/DGNav.
Read the full abstract ▾
All authors Jiankun Peng, Jianyuan Guo, Ying Xu, Yue Liu, Jiashuang Yan, Xuanwei Ye, Houhua Li, Xiaoming Wang
Computer Vision
From Global to Granular: Revealing IQA Model Performance via Correlation Surface
Baoliang Chen, Danni Huang, Hanwei Zhu et al. (+5 more)
Evaluation of Image Quality Assessment (IQA) models has long been dominated by global correlation metrics, such as Pearson Linear Correlation Coefficient (PLCC) and Spearman Rank-Order Correlation Coefficient (SRCC). While widely adopted, these metrics reduce performance to a single scalar, failing to capture how ranking consistency varies across the local quality spectrum. For example, two IQA models may achieve identical SRCC values, yet one ranks high-quality images (related to high Mean Opinion Score, MOS) more reliably, while the other better discriminates image pairs with small quality/MOS differences (related to $|\Delta$MOS$|$). Such complementary behaviors are invisible under global metrics. Moreover, SRCC and PLCC are sensitive to test-sample quality distributions, yielding unstable comparisons across test sets. To address these limitations, we propose Granularity-Modulated Correlation (GMC), which provides a structured, fine-grained analysis of IQA performance. GMC includes: (1) a Granularity Modulator that applies Gaussian-weighted correlations conditioned on absolute MOS values and pairwise MOS differences ($|\Delta$MOS$|$) to examine local performance variations, and (2) a Distribution Regulator that regularizes correlations to mitigate biases from non-uniform quality distributions. The resulting correlation surface maps correlation values as a joint function of MOS and $|\Delta$MOS$|$, providing a 3D representation of IQA performance. Experiments on standard
benchmarks show that GMC reveals performance characteristics invisible to scalar metrics, offering a more informative and reliable paradigm for analyzing, comparing, and deploying IQA models. Codes are available at https://github.com/Dniaaa/GMC.
Read the full abstract ▾
All authors Baoliang Chen, Danni Huang, Hanwei Zhu, Lingyu Zhu, Wei Zhou, Shiqi Wang, Yuming Fang, Weisi Lin
Computer Vision
DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learning
Mingshuang Luo, Shuang Liang, Zhengkun Rong et al. (+7 more)
Character image animation aims to synthesize high-fidelity
videos by transferring motion from a driving sequence to a static reference image. Despite recent advancements, existing methods suffer from two fundamental challenges: (1) suboptimal motion injection strategies that lead to a trade-off between identity preservation and motion consistency, manifesting as a "see-saw", and (2) an over-reliance on explicit pose priors (e.g., skeletons), which inadequately capture intricate dynamics and hinder generalization to arbitrary, non-
humanoid characters. To address these challenges, we present DreamActor-M2, a universal animation framework that reimagines motion conditioning as an in-context learning problem. Our approach follows a two-stage paradigm. First, we bridge the input modality gap by fusing reference appearance and motion cues into a unified latent space, enabling the model to jointly reason about spatial identity and temporal dynamics by leveraging the generative prior of foundational models. Second, we introduce a self-bootstrapped data synthesis pipeline that curates pseudo cross-identity training pairs, facilitating a seamless transition from pose-dependent control to direct, end-to-end RGB-driven animation. This strategy significantly enhances generalization across diverse characters and motion scenarios. To facilitate comprehensive evaluation, we further introduce AW Bench, a versatile
benchmark encompassing a wide spectrum of characters types and motion scenarios. Extensive experiments demonstrate that DreamActor-M2 achieves state-of-the-art performance, delivering superior visual fidelity and robust cross-domain generalization. Project Page: https://grisoon.github.io/DreamActor-M2/
Read the full abstract ▾
All authors Mingshuang Luo, Shuang Liang, Zhengkun Rong, Yuxuan Luo, Tianshu Hu, Ruibing Hou, Hong Chang, Yong Li, Yuan Zhang, Mingyuan Gao
Computer Vision
ChartE$^{3}$: A Comprehensive Benchmark for End-to-End Chart Editing
Shuo Li, Jiajun Sun, Zhekai Wang et al. (+9 more)
Charts are a fundamental visualization format for structured data analysis. Enabling end-to-end chart editing according to user intent is of great practical value, yet remains challenging due to the need for both fine-grained control and global structural consistency. Most existing approaches adopt pipeline-based designs, where natural
language or code serves as an intermediate representation, limiting their ability to faithfully execute complex edits. We introduce ChartE$^{3}$, an End-to-End Chart Editing
benchmark that directly evaluates models without relying on intermediate natural
language programs or code-level supervision. ChartE$^{3}$ focuses on two complementary editing dimensions: local editing, which involves fine-grained appearance changes such as font or color adjustments, and global editing, which requires holistic, data-centric transformations including data filtering and trend line addition. ChartE$^{3}$ contains over 1,200 high-quality samples constructed via a well-designed data pipeline with human curation. Each sample is provided as a triplet of a chart image, its underlying code, and a
multimodal editing instruction, enabling evaluation from both objective and subjective perspectives. Extensive benchmarking of state-of-the-art
multimodal large
language models reveals substantial performance gaps, particularly on global editing tasks, highlighting critical limitations in current end-to-end chart editing capabilities.
Read the full abstract ▾
All authors Shuo Li, Jiajun Sun, Zhekai Wang, Xiaoran Fan, Hui Li, Dingwen Yang, Zhiheng Xi, Yijun Wang, Zifei Shan, Tao Gui, Qi Zhang, Xuanjing Huang
Computer Vision
Multimodal Visual Surrogate Compression for Alzheimer's Disease Classification
Dexuan Ding, Ciyuan Peng, Endrowednes Kuantama et al. (+6 more)
High-dimensional structural MRI (sMRI) images are widely used for Alzheimer's Disease (AD) diagnosis. Most existing methods for sMRI representation learning rely on 3D architectures (e.g., 3D CNNs), slice-wise feature extraction with late aggregation, or apply training-free feature extractions using 2D foundation models (e.g., DINO). However, these three paradigms suffer from high computational cost, loss of cross-slice relations, and limited ability to extract discriminative features, respectively. To address these challenges, we propose
Multimodal Visual Surrogate Compression (MVSC). It learns to compress and adapt large 3D sMRI volumes into compact 2D features, termed as visual surrogates, which are better aligned with frozen 2D foundation models to extract powerful representations for final AD classification. MVSC has two key components: a Volume Context Encoder that captures global cross-slice context under textual guidance, and an Adaptive Slice Fusion module that aggregates slice-level information in a text-enhanced, patch-wise manner. Extensive experiments on three large-scale Alzheimer's disease
benchmarks demonstrate our MVSC performs favourably on both binary and multi-class classification tasks compared against state-of-the-art methods.
Read the full abstract ▾
All authors Dexuan Ding, Ciyuan Peng, Endrowednes Kuantama, Jingcai Guo, Jia Wu, Jian Yang, Amin Beheshti, Ming-Hsuan Yang, Yuankai Qi
Computer Vision
Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusion
Zixuan Xia, Hao Wang, Pengcheng Weng et al. (+4 more)
Multimodal fusion is often treated as an optimization-balancing problem, where training signals are adjusted to prevent one modality from dominating the others. However, balanced optimization does not fully determine the geometry of intermediate representations. Supervised
multimodal models may still learn low-diversity modality-specific
embeddings or allow paired cross-modal observations to drift excessively apart, weakening both unimodal robustness and
multimodal fusion. We introduce \regName, a lightweight plug-and-play
geometric regularization framework for
multimodal representation learning. Rather than enforcing rigid cross-modal
alignment, \regName follows a bounded-agreement principle: preserve modality-specific diversity while softly constraining only the portion of paired cross-modal drift that exceeds an admissible agreement band. Operationally, \regName combines a dispersion term that mitigates spectral concentration with an agreement-band anchoring term that controls excessive paired drift, requiring no architectural modification or inference-time overhead. Experiments across audio-visual, image-text, and RF-based
benchmarks show that \regName consistently improves
multimodal performance and often strengthens unimodal representations. These results suggest that explicitly regulating representation geometry is an effective complement to optimization balancing, and provide evidence that geometry-aware regularization can improve
multimodal learning across diverse architectures and domains.
Read the full abstract ▾
All authors Zixuan Xia, Hao Wang, Pengcheng Weng, Yanyu Qian, Yangxin Xu, William Dan, Fei Wang
Computer Vision · Cross-listed from cs.RO
From Instruction to Event: Sound-Triggered Mobile Manipulation
Hao Ju, Shaofei Huang, Hongyu Li et al. (+4 more)
Current mobile manipulation research predominantly follows an instruction-driven paradigm, where
robots rely on predefined textual commands to execute tasks. However, this setting confines
robots to a passive role, limiting robotic autonomy and the ability to react to dynamic environmental events. To address these limitations, we introduce Sound-Triggered Mobile Manipulation (STMM), where
robots must actively perceive and interact with sound-emitting objects without explicit action instructions. To support STMM, we develop Habitat-Echo, a simulation platform that integrates sound rendering with physical interaction. We further propose a hierarchical baseline that translates high-level planning into low-level executions, where a task planner predicts a skill chain for policy models to execute sequentially. Experiments indicate the feasibility of perceiving auditory events and executing corresponding physical interactions without explicit instructions. Notably, in challenging multi-event scenarios, the
robot successfully isolates the primary sources from overlapping acoustic interference to execute the first interactions, and subsequently proceeds to manipulate the secondary objects. These new challenges of planning-to-execution position STMM as a measurable research direction.The code and
datasets will be released upon acceptance.
Read the full abstract ▾
All authors Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, Zhedong Zheng
Computer Vision · Cross-listed from cs.AI
SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum et al. (+2 more)
Multimodal Large
Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, while their ability to process sequential audio-
video data remains underexplored. This gap highlights the need for a high-quality
benchmark to systematically evaluate MLLM performance in a real-world setting. We introduce SONIC-O1, a comprehensive, fully human-verified
benchmark of 60 hours (231 clips) spanning 13 real-world conversational domains with 4,958 annotations and demographic metadata. SONIC-O1 evaluates three capabilities: open-ended summarization, multiple-choice question (MCQ) answering, and temporal localization with supporting rationales (
reasoning). Across closed- and open-source models, we find that the MCQ accuracy shows the smallest gap between model families, but the best closed-source model outperforms the best open-source model by 22.6% on temporal localization. We further observe accuracy gaps of up to 21.4% on temporal localization across demographic groups, indicating persistent disparities in model behaviour. SONIC-O1 provides an open evaluation suite for temporally grounded and demographically robust
multimodal understanding. SONIC-O1 is publicly available for research: Project page (https://vectorinstitute.github.io/sonic-o1/),
Dataset (https://huggingface.co/
datasets/vector-institute/sonic-o1), GitHub (https://github.com/vectorinstitute/sonic-o1), Leaderboard (https://huggingface.co/spaces/vector-institute/sonic-o1-leaderboard).
Read the full abstract ▾
All authors Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum, Deval Pandya, Shaina Raza
Computer Vision
Few-Shot Domain Adaptation with Temporal References and Static Priors for Glacier Calving Front Delineation
Marcel Dreier, Nora Gourmelon, Dakota Pyles et al. (+4 more)
During benchmarking, the state-of-the-art model for glacier calving front delineation achieves near-human performance. However, when applied in a real-world setting at a novel study site, its delineation accuracy is insufficient for calving front products intended for further scientific analyses. This site represents an out-of-distribution domain for a model trained solely on the
benchmark dataset. By employing a few-shot domain adaptation strategy, incorporating spatial static prior knowledge, and including summer reference images in the input time series, the delineation error is reduced from 1131.6 m to 68.7 m without any architectural modifications. These methodological advancements establish a framework for applying deep learning-based calving front
segmentation to novel study sites, enabling calving front monitoring on a global scale.
Read the full abstract ▾
All authors Marcel Dreier, Nora Gourmelon, Dakota Pyles, Thorsten Seehaus, Matthias H. Braun, Andreas Maier, Vincent Christlein
Computer Vision
CAF-Mamba: Mamba-Based Cross-Modal Adaptive Attention Fusion for Multimodal Depression Detection
Bowen Zhou, Marc-André Fiedler and Ayoub Al-Hamadi
Depression is a prevalent mental health disorder that severely impairs daily functioning and quality of life. While recent deep learning approaches for depression detection have shown promise, most rely on limited feature types, overlook explicit cross-modal interactions, and employ simple concatenation or static weighting for fusion. To overcome these limitations, we propose CAF-Mamba, a novel Mamba-based cross-modal adaptive
attention fusion framework. CAF-Mamba not only captures cross-modal interactions explicitly and implicitly, but also dynamically adjusts modality contributions through a modality-wise
attention mechanism, enabling more effective
multimodal fusion. Experiments on two in-the-wild
benchmark datasets, LMVD and D-Vlog, demonstrate that CAF-Mamba consistently outperforms existing methods and achieves state-of-the-art performance. Our code is available at https://github.com/zbw-zhou/CAF-Mamba.
Read the full abstract ▾
Computer Vision
OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models
Yufeng Zhong, Lei Chen, Xuanle Zhao et al. (+7 more)
The development of large
vision language models drives the demand for managing, and applying massive amounts of
multimodal data, making OCR technology, which extracts information from visual images, increasingly popular. However, existing OCR methods primarily focus on recognizing text elements from images or scanned documents (Text-centric OCR), neglecting the identification of visual elements from visually information-dense image sources (
Vision-centric OCR), such as charts, web pages and science plots. In reality, these visually information-dense images are widespread on the internet and have significant real-world application value, such as data visualization and web page analysis. In this technical report, we propose OCRVerse, the first holistic OCR method in end-to-end manner that enables unified text-centric OCR and
vision-centric OCR. To this end, we constructe comprehensive data engineering to cover a wide range of text-centric documents, such as newspapers, magazines and books, as well as
vision-centric rendered composites, including charts, web pages and scientific plots. Moreover, we propose a two-stage SFT-RL multi-domain training method for OCRVerse. SFT directly mixes cross-domain data to train and establish initial domain knowledge, while RL focuses on designing personalized reward strategies for the characteristics of each domain. Specifically, since different domains require various output formats and expected outputs, we provide sufficient flexibility in the RL stage to customize flexible reward signals for each domain, thereby improving cross-domain fusion and avoiding data conflicts. Experimental results demonstrate the effectiveness of OCRVerse, achieving competitive results across text-centric and
vision-centric data types, even comparable to large-scale open-source and closed-source models.
Read the full abstract ▾
All authors Yufeng Zhong, Lei Chen, Xuanle Zhao, Wenkang Han, Liming Zheng, Jing Huang, Deyang Jiang, Yilin Cao, Lin Ma, Zhixiong Zeng
Computer Vision
RSGround-R1: Rethinking Remote Sensing Visual Grounding through Spatial Reasoning
Shiqi Huang, Shuting He and Bihan Wen
Remote Sensing Visual Grounding (RSVG) aims to localize target objects in large-scale aerial imagery based on natural
language descriptions. Owing to the vast spatial scale and high semantic ambiguity of remote sensing scenes, these descriptions often rely heavily on positional cues, posing unique challenges for
Multimodal Large
Language Models (MLLMs) in spatial
reasoning. To leverage this unique feature, we propose a
reasoning-guided, position-aware post-training framework, dubbed RSGround-R1, to progressively enhance spatial understanding. Specifically, we first introduce Chain-of-Thought Supervised Fine-Tuning (CoT-SFT) using synthetically generated RSVG
reasoning data to establish explicit position awareness. Reinforcement Fine-Tuning (RFT) is then applied, augmented by our newly designed positional reward that provides continuous and distance-aware guidance toward accurate localization. Moreover, to mitigate incoherent localization behaviors across rollouts, we introduce a spatial consistency guided optimization scheme that dynamically adjusts policy updates based on their spatial coherence, ensuring stable and robust convergence. Extensive experiments on RSVG
benchmarks demonstrate superior performance and generalization of our model.
Read the full abstract ▾
Computer Vision
A Tilted Seesaw: Revisiting Autoencoder Trade-off for Controllable Diffusion
Pu Cao, Yiyang Ma, Feng Zhou et al. (+3 more)
In latent
diffusion models, the autoencoder (AE) is typically expected to balance two capabilities: faithful reconstruction and a generation-friendly latent space (e.g., low gFID). In recent ImageNet-scale AE studies, we observe a systematic bias toward generative metrics in handling this trade-off: reconstruction metrics are increasingly under-reported, and ablation-based AE selection often favors the best-gFID configuration even when reconstruction fidelity degrades. We theoretically analyze why this gFID-dominant preference can appear unproblematic for ImageNet generation, yet becomes risky when scaling to controllable
diffusion: AEs can induce condition drift, which limits achievable condition
alignment. Meanwhile, we find that reconstruction fidelity, especially instance-level measures, better indicates controllability. We empirically validate the impact of tilted autoencoder evaluation on controllability by studying several recent ImageNet AEs. Using a multi-dimensional condition-drift evaluation protocol reflecting controllable generation tasks, we find that gFID is only weakly predictive of condition preservation, whereas reconstruction-oriented metrics are substantially more aligned. ControlNet experiments further confirm that controllability tracks condition preservation rather than gFID. Overall, our results expose a gap between ImageNet-centric AE evaluation and the requirements of scalable controllable
diffusion, offering practical guidance for more reliable benchmarking and model selection.
Read the full abstract ▾
All authors Pu Cao, Yiyang Ma, Feng Zhou, Xuedan Yin, Qing Song, Lu Yang
Computer Vision
Similarity of Processing Steps in Vision Model Representations
Matéo Mahaut and Marco Baroni
Recent literature suggests that the bigger the model, the more likely it is to converge to similar, “universal” representations, despite different training objectives,
datasets, or modalities. While this literature shows that there is an area where model representations are similar, we study here how
vision models might get to those representations – in particular, do they also converge to the same intermediate steps and operations? We therefore study the processes that lead to convergent representations in different models. First, we quantify distance between different model representations at different stages. We follow the
evolution of distances between models throughout processing, identifying the processing steps which are most different between models. We find that while layers at similar positions in different models have the most similar representations, strong differences remain. Classifier models, unlike the others, will discard information about low-level image statistics in their final layers. CNN- and
transformer-based models also behave differently, with
transformer models applying smoother changes to representations from one layer to the next. These distinctions clarify the level and nature of convergence between model representations, and enables a more qualitative account of the underlying processes in image models.
Read the full abstract ▾
Computer Vision
PathReasoner-R1: Instilling Structured Reasoning into Pathology Vision-Language Model via Knowledge-Guided Policy Optimization
Songhan Jiang, Fengchun Liu, Ziyue Wang et al. (+2 more)
Vision-
Language Models (VLMs) are advancing computational pathology with superior visual understanding capabilities. However, current systems often reduce diagnosis to directly output conclusions without verifiable evidence-linked
reasoning, which severely limits clinical trust and hinders expert error rectification. To address these barriers, we construct PathReasoner, the first large-scale
dataset of whole-slide image (WSI)
reasoning. Unlike previous work reliant on unverified
distillation, we develop a rigorous knowledge-guided generation pipeline. By leveraging medical knowledge graphs, we explicitly align structured pathological findings and clinical
reasoning with diagnoses, generating over 20K high-quality instructional samples. Based on the database, we propose PathReasoner-R1, which synergizes trajectory-masked supervised fine-tuning with
reasoning-oriented reinforcement learning to instill structured chain-of-thought capabilities. To ensure medical rigor, we engineer a knowledge-aware multi-granular reward function incorporating an Entity Reward mechanism strictly aligned with knowledge graphs. This effectively guides the model to optimize for logical consistency rather than mere outcome matching, thereby enhancing robustness. Extensive experiments demonstrate that PathReasoner-R1 achieves state-of-the-art performance on both PathReasoner and public
benchmarks across various image scales, equipping pathology models with transparent, clinically grounded
reasoning capabilities.
Dataset and code are available at https://github.com/cyclexfy/PathReasoner-R1.
Read the full abstract ▾
All authors Songhan Jiang, Fengchun Liu, Ziyue Wang, Linghan Cai, Yongbing Zhang
Computer Vision
WMVLM: Evaluating Diffusion Model Image Watermarking via Vision-Language Models
Zijin Yang, Yu Sun, Kejiang Chen et al. (+4 more)
Digital watermarking is essential for securing generated images from
diffusion models. Accurate watermark evaluation is critical for
algorithm development, yet existing methods have significant limitations: they lack a unified framework for both residual and semantic watermarks, provide results without interpretability, neglect comprehensive security considerations, and often use inappropriate metrics for semantic watermarks. To address these gaps, we propose WMVLM, the first unified and interpretable evaluation framework for
diffusion model image watermarking via
vision-
language models (VLMs). We redefine quality and security metrics for each watermark type: residual watermarks are evaluated by artifact strength and erasure resistance, while semantic watermarks are assessed through latent distribution shifts. Moreover, we introduce a three-stage training strategy to progressively enable the model to achieve classification, scoring, and interpretable text generation. Experiments show WMVLM outperforms state-of-the-art VLMs with strong generalization across
datasets,
diffusion models, and watermarking methods.
Read the full abstract ▾
All authors Zijin Yang, Yu Sun, Kejiang Chen, Jiawei Zhao, Jun Jiang, Weiming Zhang, Nenghai Yu