Computer Vision · Cross-listed from cs.LG
IDER: IDempotent Experience Replay for Reliable Continual Learning
Zhanwang Liu, Yuting Li, Haoyuan Gao et al. (+4 more)
Catastrophic forgetting, the tendency of neural networks to forget previously learned knowledge when learning new tasks, has been a major challenge in continual learning (CL). To tackle this challenge, CL methods have been proposed and shown to reduce forgetting. Furthermore, CL models deployed in mission-critical settings can benefit from uncertainty awareness by calibrating their predictions to reliably assess their confidences. However, existing uncertainty-aware continual learning methods suffer from high computational overhead and incompatibility with mainstream replay methods. To address this, we propose idempotent experience replay (IDER), a novel approach based on the idempotent property where repeated function applications yield the same output. Specifically, we first adapt the training loss to make model idempotent on current data streams. In addition, we introduce an idempotence
distillation loss. We feed the output of the current model back into the old checkpoint and then minimize the distance between this reprocessed output and the original output of the current model. This yields a simple and effective new baseline for building reliable continual learners, which can be seamlessly integrated with other CL approaches. Extensive experiments on different CL
benchmarks demonstrate that IDER consistently improves prediction reliability while simultaneously boosting accuracy and reducing forgetting. Our results suggest the potential of idempotence as a promising principle for deploying efficient and trustworthy continual learning systems in real-world applications.Our code is available at https://github.com/YutingLi0606/Idempotent-Continual-Learning.
Read the full abstract ▾
All authors Zhanwang Liu, Yuting Li, Haoyuan Gao, Yexin Li, Linghe Kong, Lichao Sun, Weiran Huang
Computer Vision
Exploring Spatiotemporal Feature Propagation for Video-Level Compressive Spectral Reconstruction: Dataset, Model and Benchmark
Lijing Cai, Zhan Shi, Chenglong Huang et al. (+6 more)
Recently, Spectral Compressive Imaging (SCI) has achieved remarkable success, unlocking significant potential for dynamic spectral
vision. However, existing reconstruction methods, primarily image-based, suffer from two limitations: (i) Encoding process masks spatial-spectral features, leading to uncertainty in reconstructing missing information from single compressed measurements, and (ii) The frame-by-frame reconstruction paradigm fails to ensure temporal consistency, which is crucial in the
video perception. To address these challenges, this paper seeks to advance spectral reconstruction from the image level to the
video level, leveraging the complementary features and temporal continuity across adjacent frames in dynamic scenes. Initially, we construct the first high-quality dynamic hyperspectral image
dataset (DynaSpec), comprising 30 sequences obtained through frame-scanning acquisition. Subsequently, we propose the Propagation-Guided Spectral
Video Reconstruction
Transformer (PG-SVRT), which employs a spatial-then-temporal
attention to effectively reconstruct spectral features from abundant
video information, while using a bridged
token to reduce computational complexity. Finally, we conduct simulation experiments to assess the performance of four SCI systems, and construct a DD-CASSI prototype for real-world data collection and benchmarking. Extensive experiments demonstrate that PG-SVRT achieves superior performance in reconstruction quality, spectral fidelity, and temporal consistency, while maintaining minimal FLOPs. Project page: https://github.com/nju-cite/DynaSpec
Read the full abstract ▾
All authors Lijing Cai, Zhan Shi, Chenglong Huang, Jinyao Wu, Qiping Li, Zikang Huo, Linsen Chen, Chongde Zi, Xun Cao
Computer Vision
Linking Modality Isolation in Heterogeneous Collaborative Perception
Changxing Liu, Zichen Chao and Siheng Chen
Collaborative perception leverages data exchange among multiple
agents to enhance overall perception capabilities. However, heterogeneity across
agents introduces domain gaps that hinder collaboration, and this is further exacerbated by an underexplored issue: modality isolation. It arises when multiple
agents with different modalities never co-occur in any training data frame, enlarging cross-modal domain gaps. Existing
alignment methods rely on supervision from spatially overlapping observations, thus fail to handle modality isolation. To address this challenge, we propose CodeAlign, the first efficient, co-occurrence-free
alignment framework that smoothly aligns modalities via cross-modal feature-code-feature(FCF) translation. The key idea is to explicitly identify the representation consistency through codebook, and directly learn mappings between modality-specific feature spaces, thereby eliminating the need for spatial correspondence. Codebooks regularize feature spaces into code spaces, providing compact yet expressive representations. With a prepared code space for each modality, CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space, enabling effective
alignment. Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior
alignment methods, reduces communication load by 1024x, and achieves state-of-the-art perception performance on both OPV2V and DAIR-V2X
dataset. Code will be released on https://github.com/cxliu0314/CodeAlign.
Read the full abstract ▾
Computer Vision
IdGlow: Dynamic Identity Modulation for Multi-Subject Generation
Honghao Cai, Xiangyuan Wang, Jing Li et al. (+15 more)
Multi-subject image generation requires seamlessly harmonizing multiple reference identities within a coherent scene. However, existing methods relying on rigid spatial masks or localized
attention often struggle with the "stability-plasticity dilemma," particularly failing in tasks that require complex structural deformations, such as identity-preserving age transformation. To address this, we present IdGlow, a mask-free, progressive two-stage framework built upon Flow Matching
diffusion models. In the supervised fine-tuning (SFT) stage, we introduce task-adaptive timestep scheduling aligned with
diffusion generative dynamics: a linear decay schedule that progressively relaxes constraints for natural group composition, and a temporal gating mechanism that concentrates identity injection within a critical semantic window, successfully preserving adult facial semantics without overriding child-like anatomical structures. To resolve attribute leakage and semantic ambiguity without explicit layout inputs, we further integrate a badcase-driven
Vision-
Language Model (VLM) for precise, context-aware
prompt synthesis. In the second stage, we design a Fine-Grained Group-Level Direct Preference Optimization (DPO) with a weighted margin formulation to simultaneously eliminate multi-subject artifacts, elevate texture harmony, and recalibrate identity fidelity towards real-world distributions. Extensive experiments on two challenging
benchmarks – direct multi-person fusion and age-transformed group generation – demonstrate that IdGlow fundamentally mitigates the stability-plasticity conflict, achieving a superior Pareto balance between state-of-the-art facial fidelity and commercial-grade aesthetic quality.
Read the full abstract ▾
All authors Honghao Cai, Xiangyuan Wang, Jing Li, Yunhao Bai, Tianze Zhou, Haohua Chen, Chao Hui, Changhao Qiao, Runqi Wang, Sijie Xu, Yuyang Hao, Zezhou Cui, Yuyuan Yang, Wei Zhu, Yibo Chen, Xu Tang, Yao Hu, Zhen Li
Computer Vision
Data-Centric Benchmark for Label Noise Estimation and Ranking in Remote Sensing Binary Building Segmentation
Keiller Nogueira, Codrut-Andrei Diaconu, Dávid Kerekes et al. (+9 more)
High-quality pixel-level annotations are essential for the semantic
segmentation of remote sensing imagery. However, such labels are expensive to obtain and often affected by noise due to the labor-intensive and time-consuming nature of pixel-wise annotation, which makes it challenging for human annotators to label every pixel accurately. Annotation errors can significantly degrade the performance and robustness of modern
segmentation models, motivating the need for reliable mechanisms to identify and quantify noisy training samples. This paper introduces a novel data-centric
benchmark, together with a new, publicly available binary building
segmentation dataset. This specific task serves as a representative and practically relevant testbed that enables controlled experimentation with different annotation perturbations. Furthermore, we also introduce two techniques for identifying, quantifying, and ranking training samples according to their level of label noise in remote sensing semantic
segmentation. Such proposed methods leverage complementary strategies based on model uncertainty, prediction consistency, and representation analysis, and consistently outperform established baselines across a range of experimental settings. The outcomes of this work are publicly available at https://github.com/keillernogueira/label_noise_
segmentation.
Read the full abstract ▾
All authors Keiller Nogueira, Codrut-Andrei Diaconu, Dávid Kerekes, Jakob Gawlikowski, Cédric Léonard, Nassim Ait Ali Braham, June Moh Goo, Zichao Zeng, Zhipeng Liu, Pallavi Jain, Andrea Nascetti, Ronny Hänsch
Computer Vision
UNICBench: UNIfied Counting Benchmark for MLLM
Chenggang Rong, Tao Han, Zhiyuan Zhao et al. (+5 more)
Counting is a core capability for
multimodal large
language models (MLLMs), yet there is no unified counting
dataset to rigorously evaluate this ability across image, text, and audio. We present UNICBench, a unified
multimodal, multi level counting
benchmark and evaluation toolkit with accurate ground truth, deterministic numeric parsing, and stratified reporting. The corpus comprises 5,300 images (5,508 QA), 872 documents (5,888 QA), and 2,069 audio clips (2,905 QA), annotated with a three level capability taxonomy and difficulty tags. Under a standardized protocol with fixed splits/
prompts/seeds and modality specific matching rules, we evaluate 45 state-of-the-art MLLMs across modalities. Results show strong performance on some basic counting tasks but significant gaps on
reasoning and the hardest partitions, highlighting long-tail errors and substantial headroom for improving general counting. UNICBench offers a rigorous and comparable basis for measurement and a public toolkit to accelerate progress.
Read the full abstract ▾
All authors Chenggang Rong, Tao Han, Zhiyuan Zhao, Yaowu Fan, Jia Wan, Song Guo, Yuan Yuan, Junyu Gao
Computer Vision · Cross-listed from cs.RO
LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models
Yuchen Hou and Lin Zhao
Vision-
Language-Action (VLA) models achieve over 95% success on standard
benchmarks. However, through systematic experiments, we find that current state-of-the-art VLA models largely ignore
language instructions. Prior work lacks: (1) systematic semantic perturbation diagnostics, (2) a
benchmark that forces
language understanding by design, and (3) linguistically diverse training data. This paper constructs the LangGap
benchmark, based on a four-dimensional semantic perturbation method – varying instruction semantics while keeping the tabletop layout fixed – revealing
language understanding deficits in {\pi}0.5. Existing
benchmarks like LIBERO assign only one task per layout, underutilizing available objects and target locations; LangGap fully diversifies pick-and-place tasks under identical layouts, forcing models to truly understand
language. Experiments show that targeted data augmentation can partially close the
language gap – success rate improves from 0% to 90% with single-task training, and 0% to 28% with multi-task training. However, as semantic diversity of extended tasks increases, model learning capacity proves severely insufficient; even trained tasks perform poorly. This reveals a fundamental challenge for VLA models in understanding diverse
language instructions – precisely the long-term value of LangGap.
Read the full abstract ▾
Computer Vision
AlignVAR: Towards Globally Consistent Visual Autoregression for Image Super-Resolution
Cencen Liu, Dongyang Zhang, Wen Yin et al. (+6 more)
Visual autoregressive (VAR) models have recently emerged as a promising alternative for image generation, offering stable training, non-iterative inference, and high-fidelity synthesis through next-scale prediction. This encourages the exploration of VAR for image super-resolution (ISR), yet its application remains underexplored and faces two critical challenges: locality-biased
attention, which fragments spatial structures, and residual-only supervision, which accumulates errors across scales, severely compromises global consistency of reconstructed images. To address these issues, we propose AlignVAR, a globally consistent visual autoregressive framework tailored for ISR, featuring two key components: (1) Spatial Consistency Autoregression (SCA), which applies an adaptive mask to reweight
attention toward structurally correlated regions, thereby mitigating excessive locality and enhancing long-range dependencies; and (2) Hierarchical Consistency Constraint (HCC), which augments residual learning with full reconstruction supervision at each scale, exposing accumulated deviations early and stabilizing the coarse-to-fine refinement process. Extensive experiments demonstrate that AlignVAR consistently enhances structural coherence and perceptual fidelity over existing generative methods, while delivering over 10x faster inference with nearly 50% fewer parameters than leading
diffusion-based approaches, establishing a new paradigm for efficient ISR.
Read the full abstract ▾
All authors Cencen Liu, Dongyang Zhang, Wen Yin, Jielei Wang, Tianyu Li, Ji Guo, Wenbo Jiang, Guoqing Wang, Guoming Lu
Computer Vision
WildActor: Unconstrained Identity-Preserving Video Generation
Qin Guo, Tianyu Yang, Xuanhua He et al. (+5 more)
Production-ready human
video generation requires digital actors to maintain strictly consistent full-body identities across dynamic shots, viewpoints and motions, a setting that remains challenging for existing methods. Prior methods often suffer from face-centric behavior that neglects body-level consistency, or produce copy-paste artifacts where subjects appear rigid due to pose locking. We present Actor-18M, a large-scale human
video dataset designed to capture identity consistency under unconstrained viewpoints and environments. Actor-18M comprises 1.6M
videos with 18M corresponding human images, covering both arbitrary views and canonical three-view representations. Leveraging Actor-18M, we propose WildActor, a framework for any-view conditioned human
video generation. We introduce an Asymmetric Identity-Preserving
Attention mechanism coupled with a Viewpoint-Adaptive Monte Carlo Sampling strategy that iteratively re-weights reference conditions by marginal utility for balanced
manifold coverage. Evaluated on the proposed Actor-Bench, WildActor consistently preserves body identity under diverse shot compositions, large viewpoint transitions, and substantial motions, surpassing existing methods in these challenging settings.
Read the full abstract ▾
All authors Qin Guo, Tianyu Yang, Xuanhua He, Fei Shen, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Dan Xu
Computer Vision · Cross-listed from cs.AI
MicroVerse: A Preliminary Exploration Toward a Micro-World Simulation
Rongsheng Wang, Minghao Wu, Hongru Zhou et al. (+4 more)
Recent advances in
video generation have opened new avenues for macroscopic simulation of complex dynamic systems, but their application to microscopic phenomena remains largely unexplored. Microscale simulation holds great promise for biomedical applications such as drug discovery, organ-on-chip systems, and disease mechanism studies, while also showing potential in education and interactive visualization. In this work, we introduce MicroWorldBench, a multi-level rubric-based
benchmark for microscale simulation tasks. MicroWorldBench enables systematic, rubric-based evaluation through 459 unique expert-annotated criteria spanning multiple microscale simulation task (e.g., organ-level processes, cellular dynamics, and subcellular molecular interactions) and evaluation dimensions (e.g., scientific fidelity, visual quality, instruction following). MicroWorldBench reveals that current SOTA
video generation models fail in microscale simulation, showing violations of physical laws, temporal inconsistency, and misalignment with expert criteria. To address these limitations, we construct MicroSim-10K, a high-quality, expert-verified simulation
dataset. Leveraging this
dataset, we train MicroVerse, a
video generation model tailored for microscale simulation. MicroVerse can accurately reproduce complex microscale mechanism. Our work first introduce the concept of Micro-World Simulation and present a proof of concept, paving the way for applications in biology, education, and scientific visualization. Our work demonstrates the potential of educational microscale simulations of biological mechanisms. Our data and code are publicly available at https://github.com/FreedomIntelligence/MicroVerse
Read the full abstract ▾
All authors Rongsheng Wang, Minghao Wu, Hongru Zhou, Zhihan Yu, Zhenyang Cai, Junying Chen, Benyou Wang
Computer Vision
Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptation
Yongbo He, Zirun Guo and Tao Jin
Adapting pretrained multi-modal models to evolving test-time distributions, known as multi-modal test-time adaptation, presents a significant challenge. Existing methods frequently encounter negative transfer in the unbiased modality and catastrophic forgetting in the biased modality. To address these challenges, we propose Decoupling Adaptation for Stability and Plasticity (DASP), a novel diagnose-then-mitigate framework. Our analysis reveals a critical discrepancy within the unified latent space: the biased modality exhibits substantially higher interdimensional redundancy (i.e., strong correlations across feature dimensions) compared to the unbiased modality. Leveraging this insight, DASP identifies the biased modality and implements an asymmetric adaptation strategy. This strategy employs a decoupled architecture where each modality-specific adapter is divided into stable and plastic components. The asymmetric mechanism works as follows: for the biased modality, which requires plasticity, the plastic component is activated and updated to capture domain-specific information, while the stable component remains fixed. Conversely, for the unbiased modality, which requires stability, the plastic component is bypassed, and the stable component is updated using KL regularization to prevent negative transfer. This asymmetric design enables the model to adapt flexibly to new domains while preserving generalizable knowledge. Comprehensive evaluations on diverse multi-modal
benchmarks demonstrate that DASP significantly outperforms state-of-the-art methods.
Read the full abstract ▾
Computer Vision
MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs
Yilian Liu, Xiaojun Jia, Guoshun Nan et al. (+6 more)
Multimodal Large
Language Models (MLLMs) have achieved remarkable performance but remain vulnerable to jailbreak attacks that can induce harmful content and undermine their secure deployment. Previous studies have shown that introducing additional inference steps, which disrupt security
attention, can make MLLMs more susceptible to being misled into generating malicious content. However, these methods rely on single-image masking or isolated visual cues, which only modestly extend
reasoning paths and thus achieve limited effectiveness, particularly against strongly aligned commercial closed-source models. To address this problem, in this paper, we propose Multi-Image Dispersion and Semantic Reconstruction (MIDAS), a
multimodal jailbreak framework that decomposes harmful semantics into risk-bearing subunits, disperses them across multiple visual clues, and leverages cross-image
reasoning to gradually reconstruct the malicious intent, thereby bypassing existing safety mechanisms. The proposed MIDAS enforces longer and more structured multi-image chained
reasoning, substantially increases the model's reliance on visual cues while delaying the exposure of malicious semantics and significantly reducing the model's security
attention, thereby improving the performance of jailbreak against advanced MLLMs. Extensive experiments across different
datasets and MLLMs demonstrate that the proposed MIDAS outperforms state-of-the-art jailbreak attacks for MLLMs and achieves an average attack success rate of 81.46% across 4 closed-source MLLMs. Our code is available at this [link](https://github.com/Winnie-Lian/MIDAS).
Read the full abstract ▾
All authors Yilian Liu, Xiaojun Jia, Guoshun Nan, Jiuyang Lyu, Zhican Chen, Tao Guan, Shuyuan Luo, Zhongyi Zhai, Yang Liu
Computer Vision
Geometry OR Tracker: Universal Geometric Operating Room Tracking
Yihua Shao, Kang Chen, Feng Xue et al. (+6 more)
In operating rooms (OR), world-scale multi-view 3D tracking supports downstream applications such as surgeon behavior recognition, where physically meaningful quantities such as distances and motion statistics must be measured in meters. However, real clinical deployments rarely satisfy the
geometric prerequisites for stable multi-view fusion and tracking: camera calibration and RGB-D registration are always unreliable, leading to cross-view
geometric inconsistency that produces "ghosting" during fusion and degrades 3D trajectories in a shared OR coordinate frame. To address this, we introduce Geometry OR Tracker, a two-stage pipeline that first rectifies imprecise calibration into a scaleconsistent and geometrically consistent camera setup with a single global scale via a Multi-view Metric Geometry Rectification module, and then performs Occlusion-Robust 3D Point Tracking directly in the unified OR world frame. On the MM-OR
benchmark, improved
geometric consistency translates into tracking gains: our rectification front-end reduces cross-view depth disagreement by more than 30$\times$ compared to raw calibration. Ablation studies further demonstrate the relationship between calibration quality and tracking accuracy, showing that improved
geometric consistency yields stronger world-frame tracking.
Read the full abstract ▾
All authors Yihua Shao, Kang Chen, Feng Xue, Siyu Chen, Long Bai, Hongyuan Yu, Hao Tang, Jinlin Wu, Nassir Navab
Computer Vision
Weakly Supervised Video Anomaly Detection with Anomaly-Connected Components and Intention Reasoning
Yu Wang and Shengjie Zhao
Weakly supervised
video anomaly detection (WS-VAD) involves identifying the temporal intervals that contain anomalous events in untrimmed
videos, where only
video-level annotations are provided as supervisory signals. However, a key limitation persists in WS-VAD, as dense frame-level annotations are absent, which often leaves existing methods struggling to learn anomaly semantics effectively. To address this issue, we propose a novel framework named LAS-VAD, short for Learning Anomaly Semantics for WS-VAD, which integrates anomaly-connected component mechanism and intention awareness mechanism. The former is designed to assign
video frames into distinct semantic groups within a
video, and frame segments within the same group are deemed to share identical semantic information. The latter leverages an intention-aware strategy to distinguish between similar normal and abnormal behaviors (e.g., taking items and stealing). To further model the semantic information of anomalies, as anomaly occurrence is accompanied by distinct characteristic attributes (i.e., explosions are characterized by flames and thick smoke), we additionally incorporate anomaly attribute information to guide accurate detection. Extensive experiments on two
benchmark datasets, XD-Violence and UCF-Crime, demonstrate that our LAS-VAD outperforms current state-of-the-art methods with remarkable gains.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
Zeyu Chen, Huanjin Yao, Ziwang Zhao et al. (+1 more)
Using
Multimodal Large
Language Models (MLLMs) as judges to achieve precise and consistent evaluations has gradually become an emerging paradigm across various domains. Evaluating the capability and reliability of MLLM-as-a-judge systems is therefore essential for ensuring trustworthy assessment. Existing judge
benchmarks categorize samples by task types but fail to capture the fundamental judgment capabilities required for reliable evaluation. In this work, we introduce M-JudgeBench, a ten-dimensional capability-oriented
benchmark designed to comprehensively assess the judgment abilities of MLLMs. Our
benchmark decomposes evaluation into pairwise Chain-of-Thought (CoT) comparison, length bias avoidance, and process error detection tasks, jointly covering ten fine-grained subtasks. This design enables diagnosis of model reliability across
reasoning styles, response lengths, and cross-model variations. Systematic evaluation uncovers the systematic weaknesses in existing MLLM-as-a-judge systems. To address this issue, we further propose Judge-MCTS, a data construction framework generating pairwise
reasoning trajectories with various correctness and length. Using Judge-MCTS, we construct an MCTS-augmented
dataset and train M-Judger, a series of strong judge models. Extensive experiments demonstrate the superiority of M-Judger on existing judge
benchmarks as well as M-JudgeBench. Overall, our work establishes a more principled foundation for evaluating MLLM-as-a-judge through M-JudgeBench and Judge-MCTS framework, paving the way for future research on judge model evaluation and capability-driven judge training.
Read the full abstract ▾
All authors Zeyu Chen, Huanjin Yao, Ziwang Zhao, Min Yang
Computer Vision
Multiple Inputs and Mixwd data for Alzheimer's Disease Classification Based on 3D Vision Transformer
Juan A. Castro-Silva, Maria N. Moreno Garcia and Diego H. Peluffo-Ordoñez
The current methods for diagnosing Alzheimer Disease using
Magnetic Resonance Imaging (MRI) have significant limitations. Many previous studies used 2D
Transformers to analyze individual
brain slices independently, potentially losing critical 3D contextual information. Region of interest-based models often focus on only a few
brain regions despite Alzheimer's affecting multiple areas. Additionally, most classification models rely on a single test, whereas diagnosing Alzheimer's requires a multifaceted approach integrating diverse data sources for a more accurate assessment. This study introduces a novel methodology called the Multiple Inputs and Mixed Data 3D
Vision Transformer (MIMD-3DVT). This method processes consecutive slices together to capture the feature dimensions and spatial information, fuses multiple 3D ROI imaging data inputs, and integrates mixed data from demographic factors, cognitive assessments, and
brain imaging. The proposed methodology was experimentally evaluated using a combined
dataset that included the Alzheimer's Disease Neuroimaging Initiative (ADNI), the Australian Imaging, Biomarker, and Lifestyle Flagship Study of Ageing (AIBL), and the Open Access Series of Imaging Studies (OASIS). Our MIMD-3DVT, utilizing single or multiple ROIs, achieved an accuracy of 97.14%, outperforming the state-of-the-art methods in distinguishing between Normal Cognition and Alzheimer's Disease.
Read the full abstract ▾
Computer Vision
Cross-Scale Pansharpening via ScaleFormer and the PanScale Benchmark
Ke Cao, Xuanhua He, Xueheng Li et al. (+7 more)
Pansharpening aims to generate high-resolution multi-spectral images by fusing the spatial detail of panchromatic images with the spectral richness of low-resolution MS data. However, most existing methods are evaluated under limited, low-resolution settings, limiting their generalization to real-world, high-resolution scenarios. To bridge this gap, we systematically investigate the data, algorithmic, and computational challenges of cross-scale pansharpening. We first introduce PanScale, the first large-scale, cross-scale pansharpening
dataset, accompanied by PanScale-Bench, a comprehensive
benchmark for evaluating generalization across varying resolutions and scales. To realize scale generalization, we propose ScaleFormer, a novel architecture designed for multi-scale pansharpening. ScaleFormer reframes generalization across image resolutions as generalization across sequence lengths: it tokenizes images into patch sequences of the same resolution but variable length proportional to image scale. A Scale-Aware Patchify module enables training for such variations from fixed-size crops. ScaleFormer then decouples intra-patch spatial feature learning from inter-patch sequential dependency modeling, incorporating Rotary Positional Encoding to enhance extrapolation to unseen scales. Extensive experiments show that our approach outperforms SOTA methods in fusion quality and cross-scale generalization. The
datasets and source code are available at https://github.com/caoke-963/ScaleFormer.
Read the full abstract ▾
All authors Ke Cao, Xuanhua He, Xueheng Li, Lingting Zhu, Yingying Wang, Ao Ma, Zhanjie Zhang, Man Zhou, Chengjun Xie, Jie Zhang
Computer Vision
Adaptive Dynamic Dehazing via Instruction-Driven and Task-Feedback Closed-Loop Optimization for Diverse Downstream Task Adaptation
Yafei Zhang, Shuaitian Song, Huafeng Li et al. (+2 more)
In real-world
vision systems,haze removal is required not only to enhance image visibility but also to meet the specific needs of diverse downstream tasks.To address this challenge,we propose a novel adaptive dynamic dehazing framework that incorporates a closed-loop optimization mechanism.It enables feedback-driven refinement based on downstream task performance and user instruction-guided adjustment during inference,allowing the model to satisfy the specific requirements of multiple downstream tasks without retraining.Technically,our framework integrates two complementary and innovative mechanisms: (1)a task feedback loop that dynamically modulates dehazing outputs based on performance across multiple downstream tasks,and (2) a text instruction interface that allows users to specify high-level task preferences.This dual-guidance strategy enables the model to adapt its dehazing behavior after training,tailoring outputs in real time to the evolving needs of multiple tasks.Extensive experiments across various
vision tasks demonstrate the strong effectiveness,robustness,and generalizability of our approach.These results establish a new paradigm for interactive,task-adaptive dehazing that actively collaborates with downstream applications.
Read the full abstract ▾
All authors Yafei Zhang, Shuaitian Song, Huafeng Li, Shujuan Wang, Yu Liu
Computer Vision
RAFM: Retrieval-Augmented Flow Matching for Unpaired CBCT-to-CT Translation
Xianhao Zhou, Jianghao Wu, Lanfeng Zhong et al. (+4 more)
Cone-beam CT (CBCT) is routinely acquired in radiotherapy but suffers from severe artifacts and unreliable Hounsfield Unit (HU) values, limiting its direct use for dose calculation. Synthetic CT (sCT) generation from CBCT is therefore an important task, yet paired CBCT–CT data are often unavailable or unreliable due to temporal gaps, anatomical variation, and registration errors. In this work, we introduce rectified flow (RF) into unpaired CBCT-to-CT translation in medical imaging. Although RF is theoretically compatible with unpaired learning through distribution-level coupling and deterministic transport, its practical effectiveness under small medical
datasets and limited batch sizes remains underexplored. Direct application with random or batch-local pseudo pairing can produce unstable supervision due to semantically mismatched endpoint samples. To address this challenge, we propose
Retrieval-Augmented Flow Matching (RAFM), which adapts RF to the medical setting by constructing
retrieval-guided pseudo pairs using a frozen DINOv3 encoder and a global CT
memory bank. This strategy improves empirical coupling quality and stabilizes unpaired flow-based training. Experiments on SynthRAD2023 under a strict subject-level true-unpaired protocol show that RAFM outperforms existing methods across FID, MAE, SSIM, PSNR, and SegScore. The code is available at https://github.com/HiLab-git/RAFM.git.
Read the full abstract ▾
All authors Xianhao Zhou, Jianghao Wu, Lanfeng Zhong, Ku Zhao, Jinlong He, Shaoting Zhang, Guotai Wang
Computer Vision
CaptionFool: Universal Image Captioning Model Attacks
Swapnil Parekh
Image captioning models are encoder-decoder architectures trained on large-scale image-text
datasets, making them susceptible to
adversarial attacks. We present CaptionFool, a novel universal (input-agnostic)
adversarial attack against state-of-the-art
transformer-based captioning models. By modifying only 7 out of 577 image patches (approximately 1.2% of the image), our attack achieves 94-96% success rate in generating arbitrary target captions, including offensive content. We further demonstrate that CaptionFool can generate "slang" terms specifically designed to evade existing content moderation filters. Our findings expose critical vulnerabilities in deployed
vision-
language models and underscore the urgent need for robust defenses against such attacks. Warning: This paper contains model outputs which are offensive in nature.
Read the full abstract ▾
Computer Vision
Wenjie Wei, Xiaolong Zhou, Malu Zhang et al. (+8 more)
Spiking neural networks (SNNs) offer an energy-efficient alternative to traditional neural networks due to their event-driven computing paradigm. However, recent advancements in spiking
transformers have focused on improving accuracy with large-scale architectures, which require significant computational resources and limit deployment on resource-constrained devices. In this paper, we propose a simple yet effective
token pruning method for spiking
transformers, termed TP-Spikformer, that reduces storage and computational overhead while maintaining competitive performance. Specifically, we first introduce a heuristic spatiotemporal information-retaining criterion that comprehensively evaluates
tokens' importance, assigning higher scores to informative
tokens for retention and lower scores to uninformative ones for
pruning. Based on this criterion, we propose an information-retaining
token pruning framework that employs a block-level early stopping strategy for uninformative
tokens, instead of removing them outright. This also helps preserve more information during
token pruning. We demonstrate the effectiveness, efficiency and scalability of TP-Spikformer through extensive experiments across diverse architectures, including Spikformer, QKFormer and Spike-driven
Transformer V1 and V3, and a range of tasks such as image classification, object detection, semantic
segmentation and event-based object tracking. Particularly, TP-Spikformer performs well in a training-free manner. These results reveal its potential as an efficient and practical solution for deploying SNNs in real-world applications with limited computational resources.
Read the full abstract ▾
All authors Wenjie Wei, Xiaolong Zhou, Malu Zhang, Ammar Belatreche, Qian Sun, Yimeng Shan, Dehao Zhang, Zijian Zhou, Zeyu Ma, Yang Yang, Haizhou Li
Computer Vision
Mesh-Pro: Asynchronous Advantage-guided Ranking Preference Optimization for Artist-style Quadrilateral Mesh Generation
Zhen Zhou, Jian Liu, Biwen Lei et al. (+10 more)
Reinforcement learning (RL) has demonstrated remarkable success in text and image generation, yet its potential in 3D generation remains largely unexplored. Existing attempts typically rely on offline direct preference optimization (DPO) method, which suffers from low training efficiency and limited generalization. In this work, we aim to enhance both the training efficiency and generation quality of RL in 3D mesh generation. Specifically, (1) we design the first asynchronous online RL framework tailored for 3D mesh generation post-training efficiency improvement, which is 3.75$\times$ faster than synchronous RL. (2) We propose Advantage-guided Ranking Preference Optimization (ARPO), a novel RL
algorithm that achieves a better trade-off between training efficiency and generalization than current RL
algorithms designed for 3D mesh generation, such as DPO and group relative policy optimization (GRPO). (3) Based on asynchronous ARPO, we propose Mesh-Pro, which additionally introduces a novel diagonal-aware mixed triangular-quadrilateral tokenization for mesh representation and a ray-based reward for
geometric integrity. Mesh-Pro achieves state-of-the-art performance on artistic and dense meshes.
Read the full abstract ▾
All authors Zhen Zhou, Jian Liu, Biwen Lei, Jing Xu, Haohan Weng, Yiling Zhu, Zhuo Chen, Junfeng Fan, Yunkai Ma, Dazhao Du, Song Guo, Fengshui Jing, Chunchao Guo
Computer Vision
Jano: Adaptive Diffusion Generation with Early-stage Convergence Awareness
Yuyang Chen, Linqian Zeng, Yijin ZHou et al. (+2 more)
Diffusion models have achieved remarkable success in generative AI, yet their computational efficiency remains a significant challenge, particularly for
Diffusion Transformers (DiTs) requiring intensive full-
attention computation. While existing acceleration approaches focus on content-agnostic uniform optimization strategies, we observe that different regions in generated content exhibit heterogeneous convergence patterns during the denoising process. We present Jano, a training-free framework that leverages this insight for efficient region-aware generation. Jano introduces an early-stage complexity recognition
algorithm that accurately identifies regional convergence requirements within initial denoising steps, coupled with an adaptive
token scheduling runtime that optimizes computational resource allocation. Through comprehensive evaluation on state-of-the-art models, Jano achieves substantial acceleration (average 2.0 times speedup, up to 2.4 times) while preserving generation quality. Our work challenges conventional uniform processing assumptions and provides a practical solution for accelerating large-scale content generation. The source code of our implementation is available at https://github.com/chen-yy20/Jano.
Read the full abstract ▾
All authors Yuyang Chen, Linqian Zeng, Yijin ZHou, Hengjie Li, Jidong Zhai
Computer Vision
Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training
Quan Kong, Yanru Xiao, Yuhao Shen et al. (+1 more)
Learning efficient and expressive visual representation has long been the pursuit of computer
vision research. While
Vision Transformers (ViTs) gradually replace traditional Convolutional Neural Networks (CNNs) as more scalable
vision learners, their applications are plagued by the quadratic complexity of the self-
attention mechanism. To address the challenge, we introduce a new linear-time sequence modeling method Test-Time Training (TTT) into
vision and propose
Vision-TTT, which treats visual sequences as
datasets and compresses the visual
token sequences in a novel self-supervised learning manner. By incorporating the dual-
dataset strategy and Conv2d-based
dataset preprocessing,
Vision-TTT effectively extends vanilla TTT to model 2D visual correlations with global receptive fields. Extensive experiments show that Vittt-T/S/B achieve $77.7\%,81.8\%,82.7\%$ Top-1 accuracy on ImageNet classification and also greatly outperform their counterparts on downstream tasks. At $1280\times1280$ resolution, Vittt-T reduces FLOPs by $79.4\%$ and runs $4.72\times$ faster with $88.9\%$ less
memory than DeiT-T. These results demonstrate the expressiveness and efficiency of
Vision-TTT as a strong candidate for the next-generation generic visual backbone.
Read the full abstract ▾
All authors Quan Kong, Yanru Xiao, Yuhao Shen, Cong Wang
Computer Vision
MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence
Xingyilang Yin, Chengzhengxu Li, Jiahao Chang et al. (+2 more)
Humans are born with
vision-based 4D spatial-temporal intelligence, which enables us to perceive and reason about the
evolution of 3D space over time from purely visual inputs. Despite its importance, this capability remains a significant bottleneck for current
multimodal large
language models (MLLMs). To tackle this challenge, we introduce MLLM-4D, a comprehensive framework designed to bridge the gaps in training data curation and model post-training for spatiotemporal understanding and
reasoning. On the data front, we develop a cost-efficient data curation pipeline that repurposes existing stereo
video datasets into high-quality 4D spatiotemporal instructional data. This results in the MLLM4D-2M and MLLM4D-R1-30k
datasets for Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT), alongside MLLM4D-Bench for comprehensive evaluation. Regarding model training, our post-training strategy establishes a foundational 4D understanding via SFT and further catalyzes 4D
reasoning capabilities by employing Group Relative Policy Optimization (GRPO) with specialized Spatiotemporal Chain of Thought (ST-CoT) prompting and Spatiotemporal reward functions (ST-reward) without involving the modification of architecture. Extensive experiments demonstrate that MLLM-4D achieves state-of-the-art spatial-temporal understanding and
reasoning capabilities from purely 2D RGB inputs. Project page: https://github.com/GVCLab/MLLM-4D.
Read the full abstract ▾
All authors Xingyilang Yin, Chengzhengxu Li, Jiahao Chang, Chi-Man Pun, Xiaodong Cun
Computer Vision
Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding
Wang Chen, Yuhui Zeng, Yongdong Luo et al. (+5 more)
Frame selection is crucial due to high frame redundancy and limited context windows when applying Large
Vision-
Language Models (LVLMs) to long
videos. Current methods typically select frames with high relevance to a given query, resulting in a disjointed set of frames that disregard the narrative structure of
video. In this paper, we introduce Wavelet-based Frame Selection by Detecting Semantic Boundary (WFS-SB), a training-free framework that presents a new perspective: effective
video understanding hinges not only on high relevance but, more importantly, on capturing semantic shifts - pivotal moments of narrative change that are essential to comprehending the holistic storyline of
video. However, direct detection of abrupt changes in the query-frame similarity signal is often unreliable due to high-frequency noise arising from model uncertainty and transient visual variations. To address this, we leverage the wavelet transform, which provides an ideal solution through its multi-resolution analysis in both time and frequency domains. By applying this transform, we decompose the noisy signal into multiple scales and extract a clean semantic change signal from the coarsest scale. We identify the local extrema of this signal as semantic boundaries, which segment the
video into coherent clips. Building on this, WFS-SB comprises a two-stage strategy: first, adaptively allocating a frame budget to each clip based on a composite importance score; and second, within each clip, employing the Maximal Marginal Relevance approach to select a diverse yet relevant set of frames. Extensive experiments show that WFS-SB significantly boosts LVLM performance, e.g., improving accuracy by 5.5% on VideoMME, 9.5% on MLVU, and 6.2% on LongVideoBench, consistently outperforming state-of-the-art methods. Our code is available at https://github.com/MAC-AutoML/WFS-SB.
Read the full abstract ▾
All authors Wang Chen, Yuhui Zeng, Yongdong Luo, Tianyu Xie, Luojun Lin, Jiayi Ji, Yan Zhang, Xiawu Zheng
Computer Vision
Multimodal Adaptive Retrieval Augmented Generation through Internal Representation Learning
Ruoshuang Du, Xin Sun, Qiang Liu et al. (+4 more)
Visual Question Answering systems face reliability issues due to
hallucinations, where models generate answers misaligned with visual input or factual knowledge. While
Retrieval Augmented Generation frameworks mitigate this issue by incorporating external knowledge, static
retrieval often introduces irrelevant or conflicting content, particularly in visual RAG settings where visually similar but semantically incorrect evidence may be retrieved. To address this, we propose
Multimodal Adaptive RAG (MMA-RAG), which dynamically assesses the confidence in the internal knowledge of the model to decide whether to incorporate the retrieved external information into the generation process. Central to MMA-RAG is a decision classifier trained through a layer-wise analysis, which leverages joint internal visual and textual representations to guide the use of reverse image
retrieval. Experiments demonstrated that the model achieves a significant improvement in response performance in three VQA
datasets. Meanwhile, ablation studies highlighted the importance of internal representations in adaptive
retrieval decisions. In general, the experimental results demonstrated that MMA-RAG effectively balances external knowledge utilization and inference robustness in diverse
multimodal scenarios.
Read the full abstract ▾
All authors Ruoshuang Du, Xin Sun, Qiang Liu, Bowen Song, Zhongqi Chen, Weiqiang Wang, Liang Wang
Computer Vision
What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
Yingqi Fan, Junlong Tong, Anhao Zhao et al. (+1 more)
Multimodal large
language models (MLLMs) project visual
tokens into the
embedding space of
language models, yet the internal structuring and processing of visual semantics remain poorly understood. In this work, we introduce a two-fold analytical framework featuring a novel probing tool, $\textbf{EmbedLens}$, to conduct a fine-grained analysis. We uncover a pronounced semantic sparsity at the input level: visual
tokens consistently partition into sink, dead, and alive categories. Remarkably, only the alive
tokens, comprising $\approx60\%$ of the total input, carry image-specific meaning. Furthermore, using a targeted patch-compression
benchmark, we demonstrate that these alive
tokens already encode rich, fine-grained cues (e.g., objects, colors, and OCR) prior to entering the LLM. Internal visual computations (such as visual
attention and feed-forward networks) are redundant for most standard tasks. For the small subset of highly
vision-centric tasks that actually benefit from internal processing, we reveal that alive
tokens naturally align with intermediate LLM layers rather than the initial
embedding space, indicating that shallow-layer processing is unnecessary and that direct mid-layer injection is both sufficient. Ultimately, our findings provide a unified mechanistic view of visual
token processing, paving the way for more efficient and interpretable MLLM architectures through selective
token pruning, minimized visual computation, and mid-layer injection. The code is released at: https://github.com/EIT-NLP/EmbedLens.
Read the full abstract ▾
All authors Yingqi Fan, Junlong Tong, Anhao Zhao, Xiaoyu Shen
Computer Vision
Hierarchical Classification for Improved Histopathology Image Analysis
Keunho Byeon, Jinsol Song, Seong Min Hong et al. (+2 more)
Whole-slide image analysis is essential for diagnostic tasks in pathology, yet existing deep learning methods primarily rely on flat classification, ignoring hierarchical relationships among class labels. In this study, we propose HiClass, a hierarchical classification framework for improved histopathology image analysis, that enhances both coarse-grained and fine-grained WSI classification. Built based upon a multiple instance learning approach, HiClass extends it by introducing bidirectional feature integration that facilitates information exchange between coarse-grained and fine-grained feature representations, effectively learning hierarchical features. Moreover, we introduce tailored loss functions, including hierarchical consistency loss, intra- and inter-class distance loss, and group-wise cross-
entropy loss, to further optimize hierarchical learning. We assess the performance of HiClass on a gastric biopsy
dataset with 4 coarse-grained and 14 fine-grained classes, achieving superior classification performance for both coarse-grained classification and fine-grained classification. These results demonstrate the effectiveness of HiClass in improving WSI classification by capturing coarse-grained and fine-grained histopathological characteristics.
Read the full abstract ▾
All authors Keunho Byeon, Jinsol Song, Seong Min Hong, Yosep Chong, Jin Tae Kwak
Computer Vision
M$^2$: Dual-Memory Augmentation for Long-Horizon Web Agents via Trajectory Summarization and Insight Retrieval
Dawei Yan, Haokui Zhang, Guangda Huzhang et al. (+8 more)
Multimodal Large
Language Models (MLLMs) based
agents have demonstrated remarkable potential in autonomous web
navigation. However, handling long-horizon tasks remains a critical bottleneck. Prevailing strategies often rely heavily on extensive data collection and model training, yet still struggle with high computational costs and insufficient
reasoning capabilities when facing complex, long-horizon scenarios. To address this, we propose M$^2$, a training-free,
memory-augmented framework designed to optimize context efficiency and decision-making robustness. Our approach incorporates a dual-tier
memory mechanism that synergizes Dynamic Trajectory Summarization (Internal
Memory) to compress verbose interaction history into concise state updates, and Insight
Retrieval Augmentation (External
Memory) to guide the
agent with actionable guidelines retrieved from an offline insight bank. Extensive evaluations across WebVoyager and OnlineMind2Web demonstrate that M$^2$ consistently surpasses baselines, yielding up to a 19.6% success rate increase and 58.7%
token reduction for Qwen3-VL-32B, while proprietary models like Claude achieve accuracy gains up to 12.5% alongside significantly lower computational overhead.
Read the full abstract ▾
All authors Dawei Yan, Haokui Zhang, Guangda Huzhang, Yang Li, Yibo Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Ying Li, Wei Dong, Chunhua Shen