Computer Vision
ASMIL: Attention-Stabilized Multiple Instance Learning for Whole Slide Imaging
Linfeng Ye, Shayan Mohajer Hamidi, Zhixiang Chi et al. (+5 more)
Attention-based multiple instance learning (MIL) has emerged as a powerful framework for whole slide image (WSI) diagnosis, leveraging
attention to aggregate instance-level features into bag-level predictions. Despite this success, we find that such methods exhibit a new failure mode: unstable
attention dynamics. Across four representative
attention-based MIL methods and two public WSI
datasets, we observe that
attention distributions oscillate across epochs rather than converging to a consistent pattern, degrading performance. This instability adds to two previously reported challenges: overfitting and over-concentrated
attention distribution. To simultaneously overcome these three limitations, we introduce
attention-stabilized multiple instance learning (ASMIL), a novel unified framework. ASMIL uses an anchor model to stabilize
attention, replaces softmax with a normalized sigmoid function in the anchor to prevent over-concentration, and applies
token random dropping to mitigate overfitting. Extensive experiments demonstrate that ASMIL achieves up to a 6.49% F1 score improvement over state-of-the-art methods. Moreover, integrating the anchor model and normalized sigmoid into existing
attention-based MIL methods consistently boosts their performance, with F1 score gains up to 10.73%. All code and data are publicly available at https://github.com/Linfeng-Ye/ASMIL.
Read the full abstract ▾
All authors Linfeng Ye, Shayan Mohajer Hamidi, Zhixiang Chi, Guang Li, Mert Pilanci, Takahiro Ogawa, Miki Haseyama, Konstantinos N. Plataniotis
Computer Vision
GameVerse: Can Vision-Language Models Learn from Video-based Reflection?
Kuan Zhang, Dongchen Liu, Qiyue Zhao et al. (+5 more)
Human gameplay is a visually grounded interaction loop in which players act, reflect on failures, and watch tutorials to refine strategies. Can
Vision-
Language Models (VLMs) also learn from
video-based reflection? We present GameVerse, a comprehensive
video game
benchmark that enables a reflective visual interaction loop. Moving beyond traditional fire-and-forget evaluations, it uses a novel reflect-and-retry paradigm to assess how VLMs internalize visual experience and improve policies. To facilitate systematic and scalable evaluation, we also introduce a cognitive hierarchical taxonomy spanning 15 globally popular games, dual action space for both semantic and GUI control, and milestone evaluation using advanced VLMs to quantify progress. Our experiments show that VLMs benefit from
video-based reflection in varied settings, and perform best by combining failure trajectories and expert tutorials-a training-free analogue to reinforcement learning (RL) plus supervised fine-tuning (SFT).Our project page is available at https://gameverse-bench.github.io/ . Our code is available at https://github.com/THUSI-Lab/GameVerse .
Read the full abstract ▾
All authors Kuan Zhang, Dongchen Liu, Qiyue Zhao, Jinkun Hou, Xinran Zhang, Qinlei Xie, Miao Liu, Yiming Li
Computer Vision · Cross-listed from cs.RO
Wei Xue, Mingcheng Li, Xuecheng Wu et al. (+3 more)
Vision-and-
Language Navigation (VLN) requires
agents to accurately perceive complex visual environments and reason over
navigation instructions and histories. However, existing methods passively process redundant visual inputs and treat all historical contexts indiscriminately, resulting in inefficient perception and unfocused
reasoning. To address these challenges, we propose ProFocus, a training-free progressive framework that unifies \underline{Pro}active Perception and \underline{Focus}ed
Reasoning through collaboration between large
language models (LLMs) and
vision-
language models (VLMs). For proactive perception, ProFocus transforms panoramic observations into structured ego-centric semantic maps, enabling the orchestration
agent to identify missing visual information needed for reliable decision-making, and to generate targeted visual queries with corresponding focus regions that guide the perception
agent to acquire the required observations. For focused
reasoning, we propose Branch-Diverse Monte Carlo Tree Search (BD-MCTS) to identify top-$k$ high-value waypoints from extensive historical candidates. The decision
agent focuses
reasoning on the historical contexts associated with these waypoints, rather than considering all historical waypoints equally. Extensive experiments validate the effectiveness of ProFocus, achieving state-of-the-art performance among zero-shot methods on R2R and REVERIE
benchmarks.
Read the full abstract ▾
All authors Wei Xue, Mingcheng Li, Xuecheng Wu, Jingqun Tang, Dingkang Yang, Lihua Zhang
Computer Vision
You Only Need One Stage: Novel-View Synthesis From A Single Blind Face Image
Taoyue Wang, Xiang Zhang, Xiaotian Li et al. (+2 more)
We propose a novel one-stage method, NVB-Face, for generating consistent Novel-View images directly from a single Blind Face image. Existing approaches to novel-view synthesis for objects or faces typically require a high-resolution RGB image as input. When dealing with degraded images, the conventional pipeline follows a two-stage process: first restoring the image to high resolution, then synthesizing novel views from the restored result. However, this approach is highly dependent on the quality of the restored image, often leading to inaccuracies and inconsistencies in the final output. To address this limitation, we extract single-view features directly from the blind face image and introduce a feature manipulator that transforms these features into 3D-aware, multi-view latent representations. Leveraging the powerful generative capacity of a
diffusion model, our framework synthesizes high-quality, consistent novel-view face images. Experimental results show that our method significantly outperforms traditional two-stage approaches in both consistency and fidelity.
Read the full abstract ▾
All authors Taoyue Wang, Xiang Zhang, Xiaotian Li, Huiyuan Yang, Lijun Yin
Computer Vision
Open-Vocabulary vs Supervised Learning Methods for Post-Disaster Visual Scene Understanding
Anna Michailidou, Georgios Angelidis, Vasileios Argyriou et al. (+2 more)
Aerial imagery is critical for large-scale post-disaster damage assessment. Automated interpretation remains challenging due to clutter, visual variability, and strong cross-event domain shift, while supervised approaches still rely on costly, task-specific annotations with limited coverage across disaster types and regions. Recent open-vocabulary and foundation
vision models offer an appealing alternative, by reducing dependence on fixed label sets and extensive task-specific annotations. Instead, they leverage large-scale pretraining and
vision-
language representations. These properties are particularly relevant for post-disaster domains, where visual concepts are ambiguous and data availability is constrained. In this work, we present a comparative evaluation of supervised learning and open-vocabulary
vision models for post-disaster scene understanding, focusing on semantic
segmentation and object detection across multiple
datasets, including FloodNet+, RescueNet, DFire, and LADD. We examine performance trends, failure modes, and practical trade-offs between different learning paradigms, providing insight into their applicability for real-world disaster response. The most notable remark across all evaluated
benchmarks is that supervised training remains the most reliable approach (i.e., when the label space is fixed and annotations are available), especially for small objects and fine boundary delineation in cluttered scenes.
Read the full abstract ▾
All authors Anna Michailidou, Georgios Angelidis, Vasileios Argyriou, Panagiotis Sarigiannidis, Georgios Th. Papadopoulos
Computer Vision
AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models
Zhen Qu, Xian Tao, Xiaoyi Bao et al. (+4 more)
Large
multimodal models (LMMs) exhibit strong task generalization capabilities, offering new opportunities for zero-shot visual anomaly
segmentation (ZSAS). However, existing LMM-based
segmentation approaches still face fundamental limitations: anomaly concepts are inherently abstract and context-dependent, lacking stable visual prototypes, and the weak
alignment between high-level semantic
embeddings and pixel-level spatial features hinders precise anomaly localization. To address these challenges, we present AG-VAS (Anchor-Guided Visual Anomaly
Segmentation), a new framework that expands the LMM vocabulary with three learnable semantic anchor
tokens-[SEG], [NOR], and [ANO], establishing a unified anchor-guided
segmentation paradigm. Specifically, [SEG] serves as an absolute semantic anchor that translates abstract anomaly semantics into explicit, spatially grounded visual entities (e.g., holes or scratches), while [NOR] and [ANO] act as relative anchors that model the contextual contrast between normal and abnormal patterns across categories. To further enhance cross-modal
alignment, we introduce a Semantic-Pixel
Alignment Module (SPAM) that aligns
language-level semantic
embeddings with high-resolution visual features, along with an Anchor-Guided Mask Decoder (AGMD) that performs anchor-conditioned mask prediction for precise anomaly localization. In addition, we curate Anomaly-Instruct20K, a large-scale instruction
dataset that organizes anomaly knowledge into structured descriptions of appearance, shape, and spatial attributes, facilitating effective learning and integration of the proposed semantic anchors. Extensive experiments on six industrial and medical
benchmarks demonstrate that AG-VAS achieves consistent state-of-the-art performance in the zero-shot setting.
Read the full abstract ▾
All authors Zhen Qu, Xian Tao, Xiaoyi Bao, Dingrong Wang, ShiChen Qu, Zhengtao Zhang, Xingang Wang
Computer Vision
When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains
Ahmadreza Jeddi, Kimia Shaban, Negin Baghbanzadeh et al. (+4 more)
Reinforcement learning (RL) is increasingly used to post-train medical
Vision-
Language Models (VLMs), yet it remains unclear whether RL improves medical visual
reasoning or mainly sharpens behaviors already induced by supervised fine-tuning (SFT). We present a controlled study that disentangles these effects along three axes:
vision, SFT, and RL. Using MedMNIST as a multi-modality testbed, we probe visual perception by benchmarking VLM
vision towers against
vision-only baselines, quantify
reasoning support and sampling efficiency via Accuracy@1 versus Pass@K, and evaluate when RL closes the support gap and how gains transfer across modalities. We find that RL is most effective when the model already has non-trivial support (high Pass@K): it primarily sharpens the output distribution, improving Acc@1 and sampling efficiency, while SFT expands support and makes RL effective. Based on these findings, we propose a boundary-aware recipe and instantiate it by RL post-training an OctoMed-initialized model on a small, balanced subset of PMC multiple-choice VQA, achieving strong average performance across six medical VQA
benchmarks.
Read the full abstract ▾
All authors Ahmadreza Jeddi, Kimia Shaban, Negin Baghbanzadeh, Natasha Sharan, Abhishek Moturu, Elham Dolatabadi, Babak Taati
Computer Vision
Adaptive Bidirectional Task Interaction for Joint Segmentation and Classification of Breast Ultrasound
Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed et al. (+2 more)
Lesion
segmentation and classification in breast ultrasound inform each other, yet many multi-task networks couple the two tasks only through shared features. This paper introduces a decoder in which a Task Interaction Module (TIM) exchanges latent features between the
segmentation and classification branches at each of four levels, and an Adaptive Interaction Weighting (AIW) unit uses activation dispersion to set, per image and level, one softmax budget that the two directions share. Multi-scale context fusion (MSCF) adds dilated context to the residual blocks. On BUSI and BUSI-WHU, split so that duplicate images and, for BUSI-WHU, images of one patient stay within one partition, the model exceeds the mean IoU of CNN,
transformer, multi-task and same-backbone baselines, reaching 72.9% and 85.4%, and flags lesion pixels in 19.0% of its predictions on normal BUSI test images against 33.3% for MTI-Net. Replacing the per-image budget by fixed or learned constant coefficients, or by independent sigmoid gates, costs 0.92 to 1.51 IoU points on BUSI. Code and data partitions: https://github.com/C-loud-Nine/Adaptive-Task-Interaction-BUS
Read the full abstract ▾
All authors Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo
Computer Vision
FoSS: Modeling Long Range Dependencies and Multimodal Uncertainty in Trajectory Prediction via Fourier State Space Integration
Yizhou Huang, Gengze Jiang, Yihua Cheng et al. (+1 more)
Accurate trajectory prediction is vital for safe autonomous driving, yet existing approaches struggle to balance modeling power and computational efficiency.
Attention-based architectures incur quadratic complexity with increasing
agents, while recurrent models struggle to capture long-range dependencies and fine-grained local dynamics. Building upon this, we present FoSS, a dual-branch framework that unifies frequency-domain
reasoning with linear-time sequence modeling. The frequency-domain branch performs a discrete
Fourier transform to decompose trajectories into amplitude components encoding global intent and phase components capturing local variations, followed by a progressive helix reordering module that preserves spectral order; two selective state-space submodules, Coarse2Fine-SSM and SpecEvolve-SSM, refine spectral features with O(N) complexity. In parallel, a time-domain dynamic selective SSM reconstructs self-
attention behavior in linear time to retain long-range temporal context. A cross-
attention layer fuses temporal and spectral representations, while learnable queries generate multiple candidate trajectories, and a weighted fusion head expresses motion uncertainty. Experiments on Argoverse 1 and Argoverse 2
benchmarks demonstrate that FoSS achieves state-of-the-art accuracy while reducing computation by 22.5% and parameters by over 40%. Comprehensive ablations confirm the necessity of each component.
Read the full abstract ▾
All authors Yizhou Huang, Gengze Jiang, Yihua Cheng, Kezhi Wang
Computer Vision · Cross-listed from cs.RO
Certifiable Factor Graph Optimization
Zhexin Xu, Nikolas R. Sanderson, Hanna Jiamei Zhang et al. (+1 more)
We show that the factor graph and certifiable estimation paradigms, which have thus far been treated as essentially independent in the literature, can be naturally synthesized into a unified framework for certifiable factor graph optimization that combines the ease of use of the former with the strong performance guarantees of the latter. The key insight enabling our synthesis is that the core mathematical constructions used to develop certifiable
estimators (Shor's relaxation and Burer-Monteiro factorization) inherit a factor graph structure from the original problem: applying these transformations to a QCQP-representable estimation task with an associated factor graph model yields a lifted problem with identical factor graph connectivity whose constituent variables and factors are simple one-to-one algebraic transformations (lifts) of those appearing in the original QCQP's factor graph. This correspondence enables the Riemannian Staircase methodology for certifiable estimation to be easily instantiated and deployed using the same mature, highly-performant factor graph libraries and workflows already ubiquitously employed throughout robotics and computer
vision. Experimental evaluation on a variety of pose graph optimization, landmark SLAM, and range-aided SLAM
benchmarks demonstrates that our certifiable factor graph optimization methodology enables the implementation of certifiable
estimators that are functionally equivalent to current state-of-the-art hand-designed, problem-specific methods, while dramatically reducing the required implementation effort from the order of months to hours.
Read the full abstract ▾
All authors Zhexin Xu, Nikolas R. Sanderson, Hanna Jiamei Zhang, David M. Rosen
Computer Vision
Cross-Modal Guidance for Fast Diffusion-Based Computed Tomography
Timofey Efimov, Singanallur Venkatakrishnan, Maliha Hossain et al. (+2 more)
Diffusion models have emerged as powerful priors for solving inverse problems in computed
tomography (CT). In certain applications, such as
neutron CT, it can be expensive to collect large amounts of measurements even for a single scan, leading to sparse data sets from which it is challenging to obtain high quality reconstructions even with
diffusion models. One strategy to mitigate this challenge is to leverage a complementary, easily available imaging modality; however, such approaches typically require retraining the
diffusion model with large
datasets. In this work, we propose incorporating an additional modality without retraining the
diffusion prior, enabling accelerated imaging of costly modalities. We further examine the impact of imperfect side modalities on cross-modal guidance. Our method is evaluated on sparse-view
neutron computed
tomography, where reconstruction quality is substantially improved by incorporating X-ray computed
tomography of the same samples.
Read the full abstract ▾
All authors Timofey Efimov, Singanallur Venkatakrishnan, Maliha Hossain, Haley Duba-Sullivan, Amirkoushyar Ziabari
Computer Vision
The MAMA-MIA Challenge: Advancing Generalizability and Fairness in Breast MRI Tumor Segmentation and Treatment Response Prediction
Lidia Garrucho, Smriti Joshi, Kaisar Kushibar et al. (+43 more)
Breast cancer is the most frequently diagnosed malignancy among women worldwide and a leading cause of cancer-related mortality. Dynamic contrast-enhanced
magnetic resonance imaging plays a central role in tumor characterization and treatment monitoring, particularly in patients receiving neoadjuvant chemotherapy. However, existing artificial intelligence models for breast
magnetic resonance imaging are typically developed and evaluated using heterogeneous
datasets, study populations, and assessment protocols, making direct comparison difficult and limiting understanding of model robustness across institutions and clinically relevant patient subgroups. The MAMA-MIA Challenge was designed to address these challenges by providing a standardized
benchmark for the joint evaluation of primary tumor
segmentation and prediction of pathologic complete response using pre-treatment
magnetic resonance imaging only. The training cohort comprised 1,506 patients from multiple institutions in the United States, while evaluation was conducted on an external test set of 574 patients from three independent European centers to assess cross-continental and cross-institutional generalization. A unified scoring framework combined predictive performance with subgroup consistency across age, menopausal status, and breast density. Twenty-six international teams participated in the final evaluation phase. Results demonstrate substantial performance variability under a common external evaluation framework and reveal trade-offs between overall accuracy and subgroup
fairness. The challenge provides standardized
datasets, evaluation protocols, and public resources to promote the development of robust and equitable artificial intelligence systems for breast cancer imaging.
Read the full abstract ▾
All authors Lidia Garrucho, Smriti Joshi, Kaisar Kushibar, Richard Osuala, Maciej Bobowicz, Xavier Bargalló, Paulius Jaruševičius, Kai Geissler, Raphael Schäfer, Muhammad Alberb, Tony Xu, Anne Martel, Daniel Sleiman, Navchetan Awasthi, Hadeel Awwad, Joan C. Vilanova, Robert Martí, Daan Schouten, Jeong Hoon Lee, Mirabela Rusu, Eleonora Poeta, Luisa Vargas, Eliana Pastor, Maria A. Zuluaga, Jessica Kächele, Dimitrios Bounias, Alexandra Ertl, Katarzyna Gwoździewicz, Maria-Laura Cosaka, Pasant M. Abo-Elhoda, Sara W. Tantawy, Shorouq S. Sakrana, Norhan O. Shawky-Abdelfatah, Amr Muhammad Abdo-Salem, Androniki Kozana, Eugen Divjak, Gordana Ivanac, Katerina Nikiforaki, Michail E. Klontzas, Rosa García-Dosdá, Meltem Gulsun-Akpinar, Oğuz Lafc{\i}, Carlos Martín-Isla, Oliver Díaz, Laura Igual, Karim Lekadir
Computer Vision
AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models
Changwoo Baek, Jouwon Song, Sohyeon Kim et al. (+1 more)
Large
Vision-
Language Models (LVLMs) have adopted visual
token pruning strategies to mitigate substantial computational overhead incurred by extensive visual
token sequences. While prior works primarily focus on either
attention-based or diversity-based
pruning methods, in-depth analysis of these approaches' characteristics and limitations remains largely unexplored. In this work, we conduct thorough empirical analysis using effective rank (erank) as a measure of feature diversity and
attention score
entropy to investigate visual
token processing mechanisms and analyze the strengths and weaknesses of each approach. Our analysis reveals two insights: (1) Our erank-based quantitative analysis shows that many diversity-oriented
pruning methods preserve substantially less feature diversity than intended; moreover, analysis using the CHAIR
dataset reveals that the diversity they do retain is closely tied to increased
hallucination frequency compared to
attention-based
pruning. (2) We further observe that
attention-based approaches are more effective on simple images where visual evidence is concentrated, while diversity-based methods better handle complex images with distributed features. Building on these empirical insights, we show that incorporating image-aware adjustments into existing hybrid
pruning strategies consistently improves their performance. We also provide a minimal instantiation of our empirical findings through a simple adaptive
pruning mechanism, which achieves strong and reliable performance across standard
benchmarks as well as
hallucination-specific evaluations. Our project page available at https://cvsp-lab.github.io/AgilePruner.
Read the full abstract ▾
All authors Changwoo Baek, Jouwon Song, Sohyeon Kim, Kyeongbo Kong
Computer Vision
Towards Policy-Adaptive Image Guardrail: Benchmark and Method
Caiyong Piao, Zhiyuan Yan, Haoming Xu et al. (+4 more)
Accurate rejection of sensitive or harmful visual content, i.e., harmful image guardrail, is critical in many application scenarios. This task must continuously adapt to the evolving safety policies and content across various domains and over time. However, traditional classifiers, confined to fixed categories, require frequent retraining when new policies are introduced.
Vision-
language models (VLMs) offer a more adaptable and generalizable foundation for dynamic safety guardrails. Despite this potential, existing VLM-based safeguarding methods are typically trained and evaluated under only a fixed safety policy. We find that these models are heavily overfitted to the seen policy, fail to generalize to unseen policies, and even lose the basic instruction-following ability and general knowledge. To address this issue, in this paper we make two key contributions. First, we
benchmark the cross-policy generalization performance of existing VLMs with SafeEditBench, a new evaluation suite. SafeEditBench leverages image-editing models to convert unsafe images into safe counterparts, producing policy-aligned
datasets where each safe-unsafe image pair remains visually similar except for localized regions violating specific safety rules. Human annotators then provide accurate safe/unsafe labels under five distinct policies, enabling fine-grained assessment of policy-aware generalization. Second, we introduce SafeGuard-VL, a reinforcement learning-based method with verifiable rewards (RLVR) for robust unsafe-image guardrails. Instead of relying solely on supervised fine-tuning (SFT) under fixed policies, SafeGuard-VL explicitly optimizes the model with policy-grounded rewards, promoting verifiable adaptation across evolving policies. Extensive experiments verify the effectiveness of our method for unsafe image guardrails across various policies.
Read the full abstract ▾
All authors Caiyong Piao, Zhiyuan Yan, Haoming Xu, Yunzhen Zhao, Kaiqing Lin, Feiyang Xu, Shuigeng Zhou
Computer Vision
Monocular 3D Object Position Estimation with VLMs for Human-Robot Interaction
Ari Wahl, Dorian Gawlinski, David Przewozny et al. (+3 more)
Pre-trained general-purpose
Vision-
Language Models (VLM) hold the potential to enhance intuitive human-machine interactions due to their rich world knowledge and 2D object detection capabilities. However, VLMs for 3D coordinates detection tasks are rare. In this work, we investigate interactive abilities of VLMs by returning 3D object positions given a
monocular RGB image from a wrist-mounted camera, natural
language input, and
robot states. We collected and curated a heterogeneous
dataset of more than 100,000 images and finetuned a VLM using QLoRA with a custom regression head. By implementing conditional routing, our model maintains its ability to process general visual queries while adding specialized 3D position estimation capabilities. Our results demonstrate robust predictive performance with a median MAE of 13 mm on the test set and a five-fold improvement over a simpler baseline without finetuning. In about 25% of the cases, predictions are within a range considered acceptable for the
robot to interact with objects.
Read the full abstract ▾
All authors Ari Wahl, Dorian Gawlinski, David Przewozny, Paul Chojecki, Felix Bießmann, Sebastian Bosse
Computer Vision
CoSMo3D: Open-World Promptable 3D Semantic Part Segmentation through LLM-Guided Canonical Spatial Modeling
Li Jin, Weikai Chen, Yujie Wang et al. (+7 more)
Open-world promptable 3D semantic
segmentation remains brittle as semantics are inferred in the input sensor coordinates. Yet, humans, in contrast, interpret parts via functional roles in a canonical space – wings extend laterally, handles protrude to the side, and legs support from below. Psychophysical evidence shows that we mentally rotate objects into canonical frames to reveal these roles. To fill this gap, we propose \methodName{}, which attains canonical space perception by inducing a latent canonical reference frame learned directly from data. By construction, we create a unified canonical
dataset through LLM-guided intra- and cross-category
alignment, exposing canonical spatial regularities across 200 categories. By induction, we realize canonicality inside the model through a dual-branch architecture with canonical map anchoring and canonical box calibration, collapsing pose variation and
symmetry into a stable canonical
embedding. This shift from input pose space to canonical
embedding yields far more stable and transferable part semantics. Experimental results show that \methodName{} establishes new state of the art in open-world promptable 3D
segmentation.
Read the full abstract ▾
All authors Li Jin, Weikai Chen, Yujie Wang, Yingda Yin, Zeyu Hu, Runze Zhang, Keyang Luo, Shengju Qian, Xin Wang, Xueying Qin
Computer Vision
VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning
Mingkang Dong, Hongyi Cai, Jie Li et al. (+4 more)
The effectiveness of
multimodal instruction tuning depends not only on
dataset scale, but critically on whether training samples genuinely require visual
reasoning. However, existing instruction
datasets often contain a substantial portion of visually redundant samples (solvable from text alone), as well as multimodally misaligned supervision that can degrade learning. To address this, we propose VisNec (Visual Necessity Score), a principled data selection framework that measures the marginal contribution of visual input during instruction tuning. By comparing predictive loss with and without visual context, VisNec identifies whether a training instance is
vision-critical, redundant, or misaligned. To preserve task diversity, we combine VisNec with semantic clustering and select high-necessity samples within each cluster. Across 10 downstream
benchmarks, training on only 15% of the LLaVA-665K
dataset selected by VisNec achieves 100.2% of full-data performance. On the smaller
Vision-Flan-186K
dataset, our selection not only further reduces data size but also surpasses full-data training by 15.8%. These results demonstrate that measuring and leveraging visual necessity provides an effective solution for both efficient and robust
multimodal instruction tuning. Codes and selected subsets will be released upon acceptance.
Read the full abstract ▾
All authors Mingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, Yuqian Fu
Computer Vision
RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations
Mochu Xiang, Zhelun Shen, Xuesong Li et al. (+7 more)
Human perceive the 3D world through 2D observations from limited viewpoints. While recent feed-forward generalizable 3D reconstruction models excel at recovering 3D structures from sparse images, their representations are often confined to observed regions, leaving unseen geometry un-modeled. This raises a key, fundamental challenge: Can we infer a complete 3D structure from partial 2D observations? We present RnG (Reconstruction and Generation), a novel feed-forward
Transformer that unifies these two tasks by predicting an implicit, complete 3D representation. At the core of RnG, we propose a reconstruction-guided causal
attention mechanism that separates reconstruction and generation at the
attention level, and treats the KV-cache as an implicit 3D representation. Then, arbitrary poses can efficiently query this cache to render high-fidelity, novel-view RGBD outputs. As a result, RnG not only accurately reconstructs visible geometry but also generates plausible, coherent unseen geometry and appearance. Our method achieves state-of-the-art performance in both generalizable 3D reconstruction and novel view generation, while operating efficiently enough for real-time interactive applications. Project page: https://npucvr.github.io/RnG
Read the full abstract ▾
All authors Mochu Xiang, Zhelun Shen, Xuesong Li, Jiahui Ren, Jing Zhang, Chen Zhao, Shanshan Liu, Haocheng Feng, Jingdong Wang, Yuchao Dai
Computer Vision
VP-Hype: A Hybrid Mamba-Transformer Framework with Visual-Textual Prompting for Hyperspectral Image Classification
Abdellah Zakaria Sellam, Fadi Abdeladhim Zidi, Salah Eddine Bekhouche et al. (+4 more)
Accurate classification of hyperspectral imagery (HSI) is often frustrated by the tension between high-dimensional spectral data and the extreme scarcity of labeled training samples. While hierarchical models like LoLA-SpecViT have demonstrated the power of local windowed
attention and parameter-efficient fine-tuning, the quadratic complexity of standard
Transformers remains a barrier to scaling. We introduce VP-Hype, a framework that rethinks HSI classification by unifying the linear-time efficiency of State-Space Models (SSMs) with the relational modeling of
Transformers in a novel hybrid architecture. Building on a robust 3D-CNN spectral front-end, VP-Hype replaces conventional
attention blocks with a Hybrid Mamba-
Transformer backbone to capture long-range dependencies with significantly reduced computational overhead. Furthermore, we address the label-scarcity problem by integrating dual-modal Visual and Textual
Prompts that provide context-aware guidance for the feature extraction process. Our experimental evaluation demonstrates that VP-Hype establishes a new state of the art in low-data regimes. Specifically, with a training sample distribution of only 2%, the model achieves Overall Accuracy (OA) of 99.69% on the Salinas
dataset and 99.45% on the Longkou
dataset. These results suggest that the convergence of hybrid sequence modeling and multi-modal prompting provides a robust path forward for high-performance, sample-efficient remote sensing.
Read the full abstract ▾
All authors Abdellah Zakaria Sellam, Fadi Abdeladhim Zidi, Salah Eddine Bekhouche, Ihssen Houhou, Marouane Tliba, Cosimo Distante, Abdenour Hadid
Computer Vision
TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization
Sumin Kim, Hyemin Jeong, Mingu Kang et al. (+3 more)
The exponential growth of
video content necessitates effective
video summarization to efficiently extract key information from long
videos. However, current approaches struggle to fully comprehend complex
videos, primarily because they employ static or modality-agnostic fusion strategies. These methods fail to account for the dynamic, frame-dependent variations in modality saliency inherent in
video data. To overcome these limitations, we propose TripleSumm, a novel architecture that adaptively weights and fuses the contributions of visual, text, and audio modalities at the frame level. Furthermore, a significant bottleneck for research into
multimodal video summarization has been the lack of comprehensive
benchmarks. Addressing this bottleneck, we introduce MoSu (Most Replayed
Multimodal Video Summarization), the first large-scale
benchmark that provides all three modalities. Extensive experiments demonstrate that TripleSumm achieves state-of-the-art performance, outperforming existing methods by a significant margin on four
benchmarks, including MoSu. Our code and
dataset are available at https://github.com/smkim37/TripleSumm.
Read the full abstract ▾
All authors Sumin Kim, Hyemin Jeong, Mingu Kang, Yejin Kim, Yoori Oh, Joonseok Lee
Computer Vision
FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editing
Maomao Li, Yunfei Liu and Yu Li
Image-driven
video editing aims to propagate edit contents from the modified first frame to the remaining frames. Existing methods usually invert the source
video to noise using a pre-trained image-to-
video (I2V) model and then guide the sampling process using the edited first frame. Generally, a popular choice for maintaining motion and layout from the source
video is intervening in the denoising process by injecting
attention during reconstruction. However, such injection often leads to unsatisfactory results, where excessive injection leads to conflicting semantics with the source
video while insufficient injection brings limited source representation. Recognizing this, we propose an Editing-awaRE (REE) injection method to modulate the injection intensity of each
token. Specifically, we first compute the pixel difference between the source and edited first frame to form a corresponding editing mask. Next, we track the editing area throughout the entire
video by using optical flow to warp the first-frame mask. Then, editing-aware feature injection intensity for each
token is generated accordingly, where injection is not conducted in editing areas. Building upon REE injection, we further propose a zero-shot image-driven
video editing framework with recent-emerging rectified-Flow models, dubbed FREE-Edit. Without fine-tuning or training, our FREE-Edit demonstrates effectiveness in various image-driven
video editing scenarios, showing its capability to produce higher-quality outputs compared with existing techniques. Project page: https://free-edit.github.io/page/.
Read the full abstract ▾
Computer Vision
BeautyGRPO: Aesthetic Alignment for Face Retouching via Dynamic Path Guidance and Fine-Grained Preference Modeling
Jiachen Yang, Xianhui Lin, Yi Dong et al. (+4 more)
Face retouching requires removing subtle imperfections while preserving unique facial identity features, in order to enhance overall aesthetic appeal. However, existing methods suffer from a fundamental trade-off. Supervised learning on labeled data is constrained to pixel-level label mimicry, failing to capture complex subjective human aesthetic preferences. Conversely, while online reinforcement learning (RL) excels at preference
alignment, its
stochastic exploration paradigm conflicts with the high-fidelity demands of face retouching and often introduces noticeable noise artifacts due to accumulated
stochastic drift. To address these limitations, we propose BeautyGRPO, a reinforcement learning framework that aligns face retouching with human aesthetic preferences. We construct FRPref-10K, a fine-grained preference
dataset covering five key retouching dimensions, and train a specialized reward model capable of evaluating subtle perceptual differences. To reconcile exploration and fidelity, we introduce Dynamic Path Guidance (DPG). DPG stabilizes the
stochastic sampling trajectory by dynamically computing an anchor-based ODE path and replanning a guided trajectory at each sampling timestep, effectively correcting
stochastic drift while maintaining controlled exploration. Extensive experiments show that BeautyGRPO outperforms both specialized face retouching methods and general image editing models, achieving superior texture quality, more accurate blemish removal, and overall results that better align with human aesthetic preferences.
Read the full abstract ▾
All authors Jiachen Yang, Xianhui Lin, Yi Dong, Zebiao Zheng, Xing Liu, Hong Gu, Yanmei Fang
Computer Vision
GRAD-Former: Gated Robust Attention-based Differential Transformer for Change Detection
Durgesh Ameta, Ujjwal Mishra, Praful Hambarde et al. (+1 more)
Change detection (CD) in remote sensing aims to identify semantic differences between satellite images captured at different times. While deep learning has significantly advanced this field, existing approaches based on convolutional neural networks (CNNs),
transformers and Selective State Space Models (SSMs) still struggle to precisely delineate change regions. In particular, traditional
transformer-based methods suffer from quadratic computational complexity when applied to very high-resolution (VHR) satellite images and often perform poorly with limited training data, leading to under-utilization of the rich spatial information available in VHR imagery. We present GRAD-Former, a novel framework that enhances contextual understanding while maintaining efficiency through reduced model size. The proposed framework consists of a novel encoder with Adaptive Feature Relevance and Refinement (AFRAR) module, fusion and decoder blocks. AFRAR integrates global-local contextual awareness through two proposed components: the Selective
Embedding Amplification (SEA) module and the Global-Local Feature Refinement (GLFR) module. SEA and GLFR leverage gating mechanisms and differential
attention, respectively, which generates multiple softmax heaps to capture important features while minimizing the captured irreverent features. Multiple experiments across three challenging CD
datasets (LEVIR-CD, CDD, DSIFN-CD) demonstrate GRAD-Former's superior performance compared to existing approaches. Notably, GRAD-Former outperforms the current state-of-the-art models across all the metrics and all the
datasets while using fewer parameters. Our framework establishes a new
benchmark for remote sensing change detection performance. Our code will be released at: https://github.com/Ujjwal238/GRAD-Former
Read the full abstract ▾
All authors Durgesh Ameta, Ujjwal Mishra, Praful Hambarde, Amit Shukla
Computer Vision · Cross-listed from cs.RO
D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping
Haozhe Lou, Mingtong Zhang, Haoran Geng et al. (+9 more)
Simulation provides a cost-effective and flexible platform for data generation and policy learning to develop robotic systems. However, bridging the gap between simulation and real-world dynamics remains a significant challenge, especially in physical parameter identification. In this work, we introduce a real-to-sim-to-real engine that leverages the Gaussian Splat representations to build a differentiable engine, enabling object mass identification from real-world visual observations and
robot control signals, while enabling grasping policy learning simultaneously. Through optimizing the mass of the manipulated object, our method automatically builds high-fidelity and physically plausible digital twins. Additionally, we propose a novel approach to train force-aware grasping policies from limited data by transferring feasible human demonstrations into simulated
robot demonstrations. Through comprehensive experiments, we demonstrate that our engine achieves accurate and robust performance in mass identification across various object geometries and mass values. Those optimized mass values facilitate force-aware policy learning, achieving superior and high performance in object grasping, effectively reducing the sim-to-real gap.
Read the full abstract ▾
All authors Haozhe Lou, Mingtong Zhang, Haoran Geng, Hanyang Zhou, Sicheng He, Zhiyuan Gao, Siheng Zhao, Jiageng Mao, Pieter Abbeel, Jitendra Malik, Daniel Seita, Yue Wang
Computer Vision
ConVibNet: Needle Detection during Continuous Insertion via Frequency-Inspired Features
Jiamei Guo, Zhehao Duan, Maria Neiiendam et al. (+3 more)
Purpose: Ultrasound-guided needle interventions are widely used in clinical practice, but their success critically depends on accurate needle placement, which is frequently hindered by the poor and intermittent visibility of needles in ultrasound images. Existing approaches remain limited by artifacts, occlusions, and low contrast, and often fail to support real-time continuous insertion. To overcome these challenges, this study introduces a robust real-time framework for continuous needle detection. Methods: We present ConVibNet, an extension of VibNet for detecting needles with significantly reduced visibility, addressing real-time, continuous needle tracking during insertion. ConVibNet leverages temporal dependencies across successive ultrasound frames to enable continuous estimation of both needle tip position and shaft angle in dynamic scenarios. To strengthen temporal awareness of needle-tip motion, we introduce a novel intersection-and-difference loss that explicitly leverages motion correlations across consecutive frames. In addition, we curated a dedicated
dataset for model development and evaluation. Results: The performance of the proposed ConVibNet model was evaluated on our
dataset, demonstrating superior accuracy compared to the baseline VibNet and UNet-LSTM models. Specifically, ConVibNet achieved a tip error of 2.80+-2.42 mm and an angle error of 1.69+-2.00 deg. These results represent a 0.75 mm improvement in tip localization accuracy over the best-performing baseline, while preserving real-time inference capability. Conclusion: ConVibNet advances real-time needle detection in ultrasound-guided interventions by integrating temporal correlation modeling with a novel intersection-and-difference loss, thereby improving accuracy and robustness and demonstrating high potential for integration into autonomous insertion systems.
Read the full abstract ▾
All authors Jiamei Guo, Zhehao Duan, Maria Neiiendam, Dianye Huang, Nassir Navab, Zhongliang Jiang