Computer Vision
HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation
Pingping Zhang, Tianyu Yan, Yuhao Wang et al. (+7 more)
Marine Animal
Segmentation (MAS) aims at identifying and segmenting marine animals from complex marine environments. Most of previous deep learning-based MAS methods struggle with the long-distance modeling issue. Recently, Segment Anything Model (SAM) has gained popularity in general image
segmentation. However, it lacks of perceiving fine-grained details and frequency information. To this end, we propose a novel learning framework, named Hierarchical Frequency Prompted SAM (HFP-SAM) for high-performance MAS. First, we design a Frequency Guided Adapter (FGA) to efficiently inject marine scene information into the frozen SAM backbone through frequency domain prior masks. Additionally, we introduce a Frequency-aware Point Selection (FPS) to generate highlighted regions through frequency analysis. These regions are combined with the coarse predictions of SAM to generate point
prompts and integrate into SAM's decoder for fine predictions. Finally, to obtain comprehensive
segmentation masks, we introduce a Full-View Mamba (FVM) to efficiently extract spatial and channel contextual information with linear computational complexity. Extensive experiments on four public
datasets demonstrate the superior performance of our approach. The source code is publicly available at https://github.com/Drchip61/TIP-HFP-SAM.
Read the full abstract ▾
All authors Pingping Zhang, Tianyu Yan, Yuhao Wang, Yang Liu, Tongdan Tang, Yili Ma, Long Lv, Feng Tian, Weibing Sun, and Huchuan Lu
Computer Vision
SVCBench: A Streaming Video Counting Benchmark for Spatial-Temporal State Maintenance
Pengyiang Liu, Zhongyue Shi, Hongye Hao et al. (+7 more)
Video understanding requires models to continuously track and update world state during playback. Although existing
benchmarks have advanced
video understanding evaluation across multiple dimensions, they provide limited visibility into how models maintain world state over time. We propose SVCBench, a Streaming
Video Counting
Benchmark that repositions counting as a minimal, controlled probe for diagnosing models' world-state maintenance capability. We decompose this capability into object counting and event counting, forming 8 fine-grained subcategories. Object counting covers tracking currently visible objects and cumulative unique identities, while event counting covers detecting instantaneous actions and tracking complete activity cycles. SVCBench contains 406
videos with frame-by-frame annotations of 10,071 event occurrences and object state changes, yielding 1,000 streaming QA pairs with 4,576 query points distributed along
video timelines. By observing state maintenance trajectories through streaming multi-point queries, we design three complementary metrics to diagnose numerical precision, trajectory consistency, and temporal awareness. Evaluations of mainstream
video-
language models show that current models still exhibit significant deficiencies in spatial-temporal state maintenance, with especially poor performance on periodic event counting. SVCBench provides a diagnostic framework for measuring and improving state maintenance in
video understanding systems. Our code and data are available at https://buaa-colalab.github.io/SVCBench.
Read the full abstract ▾
All authors Pengyiang Liu, Zhongyue Shi, Hongye Hao, Qi Fu, Xueting Bi, Siwei Zhang, Xiaoyang Hu, Zitian Wang, Linjiang Huang, Si Liu
Computer Vision · Cross-listed from cs.RO
HaltNav: Reactive Visual Halting over Lightweight Topological Priors for Robust Vision-Language Navigation
Zihui Yu, Pingcong Li, Bichi Zhang et al. (+1 more)
Vision-and-
Language Navigation (VLN) is shifting from rigid, step-by-step instruction following toward open-vocabulary, goal-oriented autonomy. Achieving this transition without exhaustive routing
prompts requires
agents to leverage structural priors. While prior work often assumes computationally heavy 2D/3D metric maps, we instead exploit a lightweight, text-based osmAG (OpenStreetMap Area Graph), a floorplan-level topological representation that is easy to obtain and maintain. However, global planning over a prior map alone is brittle in real-world deployments, where local connectivity can change (e.g., closed doors or crowded passages), leading to execution-time failures. To address this gap, we propose a hierarchical
navigation framework HaltNav that couples the robust global planning of osmAG with the local exploration and instruction-grounding capability of VLN. Our approach features an MLLM-based
brain module, which is capable of high-level task grounding and obstruction awareness. Conditioned on osmAG, the
brain converts the global route into a sequence of localized execution snippets, providing the VLN executor with prior-grounded, goal-centric sub-instructions. Meanwhile, it detects local anomalies via a mechanism we term Reactive Visual Halting (RVH), which interrupts the local control loop, updates osmAG by invalidating the corresponding
topology, and triggers replanning to orchestrate a viable detour. To train this halting capability efficiently, we introduce a data synthesis pipeline that leverages generative models to inject realistic obstacles into otherwise navigable scenes, substantially enriching hard negative samples. Extensive experiments demonstrate that our hierarchical framework outperforms several baseline methods without tedious
language instructions, and significantly improves robustness for long-horizon
vision-
language navigation under environmental changes.
Read the full abstract ▾
All authors Zihui Yu, Pingcong Li, Bichi Zhang, Sören Schwertfeger
Computer Vision
HSEmotion Team at ABAW-10 Competition: Facial Expression Recognition, Valence-Arousal Estimation, Action Unit Detection and Fine-Grained Violence Classification
Andrey V. Savchenko and Kseniia Tsypliakova
This article presents our results for the 10th Affective Behavior Analysis in-the-Wild (ABAW) competition. For frame-wise facial
emotion understanding tasks (frame-wise facial expression recognition, valence-arousal estimation, action unit detection), we propose a fast approach based on facial
embedding extraction with pre-trained EfficientNet-based
emotion recognition models. If the latter model's confidence exceeds a threshold, its prediction is used. Otherwise, we feed
embeddings into a simple multi-layered perceptron trained on the AffWild2
dataset. Estimated class-level scores are smoothed in a sliding window of fixed size to mitigate noise in frame-wise predictions. For the fine-grained violence detection task, we examine several pre-trained architectures for frame
embeddings and their aggregation for
video classification. Experimental results on four tasks from the ABAW challenge demonstrate that our approach significantly improves validation metrics over existing baselines.
Read the full abstract ▾
Computer Vision
CM-Bench: A Comprehensive Cross-Modal Feature Matching Benchmark Bridging Visible and Infrared Images
Liangzheng Sun, Mengfan He, Xingyu Shao et al. (+5 more)
Infrared-visible (IR-VIS) feature matching plays an essential role in cross-modality visual localization,
navigation and perception. Along with the rapid development of deep learning techniques, a number of representative image matching methods have been proposed. However, crossmodal feature matching is still a challenging task due to the significant appearance difference. A significant gap for cross-modal feature matching research lies in the absence of standardized
benchmarks and metrics for evaluations. In this paper, we introduce a comprehensive cross-modal feature matching
benchmark, CM-Bench, which encompasses 30 feature matching
algorithms across diverse cross-modal
datasets. Specifically, state-of-the-art traditional and deep learning-based methods are first summarized and categorized into sparse, semidense, and dense methods. These methods are evaluated by different tasks including homography estimation, relative pose estimation, and feature-matching-based geo-localization. In addition, we introduce a classification-network-based adaptive preprocessing front-end that automatically selects suitable enhancement strategies before matching. We also present a novel infrared-satellite cross-modal
dataset with manually annotated ground-truth correspondences for practical geo-localization evaluation. The
dataset and resource will be available at: https://github.com/SLZ98/CM-Bench.
Read the full abstract ▾
All authors Liangzheng Sun, Mengfan He, Xingyu Shao, Binbin Li, Zhiqiang Yan, Chunyu Li, Ziyang Meng, Fei Xing
Computer Vision
STRAP-ViT: Segregated Tokens with Randomized – Transformations for Defense against Adversarial Patches in ViTs
Nandish Chattopadhyay, Anadi Goyal, Chandan Karfa et al. (+1 more)
Adversarial patches are physically realizable localized noise, which are able to hijack
Vision Transformers (ViT) self-
attention, pulling focus toward a small, high-contrast region and corrupting the class
token to force confident misclassifications. In this paper, we claim that the
tokens which correspond to the areas of the image that contain the
adversarial noise, have different statistical properties when compared to the
tokens which do not overlap with the
adversarial perturbations. We use this insight to propose a mechanism, called STRAP-ViT, which uses Jensen-Shannon Divergence as a metric for segregating
tokens that behave as anomalies in the Detection Phase, and then apply randomized composite transformations on them during the Mitigation Phase to make the
adversarial noise ineffective. The minimum number of
tokens to transform is a hyper-parameter for the defense mechanism and is chosen such that at least 50% of the patch is covered by the transformed
tokens. STRAP-ViT fits as a non-trainable plug-and-play block within the ViT architectures, for inference purposes only, with a minimal computational cost and does not require any additional training cost/effort. STRAP-ViT has been tested on multiple pre-trained
vision transformer architectures (ViT-base-16 and DinoV2) and
datasets (ImageNet and CalTech-101), across multiple
adversarial attacks (
Adversarial Patch, LAVAN, GDPA and RP2), and found to provide excellent robust accuracies lying within a 2-3% range of the clean baselines, and outperform the state-of-the-art.
Read the full abstract ▾
All authors Nandish Chattopadhyay, Anadi Goyal, Chandan Karfa, Anupam Chattopadhyay
Computer Vision
RSONet: Region-guided Selective Optimization Network for RGB-T Salient Object Detection
Bin Wan, Runmin Cong, Xiaofei Zhou et al. (+3 more)
This paper focuses on the inconsistency in salient regions between RGB and thermal images. To address this issue, we propose the Region-guided Selective Optimization Network for RGB-T Salient Object Detection, which consists of the region guidance stage and saliency generation stage. In the region guidance stage, three parallel branches with same encoder-decoder structure equipped with the context interaction (CI) module and spatial-aware fusion (SF) module are designed to generate the guidance maps which are leveraged to calculate similarity scores. Then, in the saliency generation stage, the selective optimization (SO) module fuses RGB and thermal features based on the previously obtained similarity values to mitigate the impact of inconsistent distribution of salient targets between the two modalities. After that, to generate high-quality detection result, the dense detail enhancement (DDE) module which adopts the multiple dense connections and visual state space blocks is applied to low-level features for optimizing the detail information. In addition, the mutual interaction semantic (MIS) module is placed in the high-level features to dig the location cues by the mutual fusion strategy. We conduct extensive experiments on the RGB-T
dataset, and the results demonstrate that the proposed RSONet achieves competitive performance against 27 state-of-the-art SOD methods.
Read the full abstract ▾
All authors Bin Wan, Runmin Cong, Xiaofei Zhou, Hao Fang, Chengtao Lv, Sam Kwong
Computer Vision
G2HFNet: GeoGran-Aware Hierarchical Feature Fusion Network for Salient Object Detection in Optical Remote Sensing Images
Bin Wan, Runmin Cong, Xiaofei Zhou et al. (+3 more)
Remote sensing images captured from aerial perspectives often exhibit significant scale variations and complex backgrounds, posing challenges for salient object detection (SOD). Existing methods typically extract multi-level features at a single scale using uniform
attention mechanisms, leading to suboptimal representations and incomplete detection results. To address these issues, we propose a GeoGran-Aware Hierarchical Feature Fusion Network (G2HFNet) that fully exploits
geometric and granular cues in optical remote sensing images. Specifically, G2HFNet adopts Swin
Transformer as the backbone to extract multi-level features and integrates three key modules: the multi-scale detail enhancement (MDE) module to handle object scale variations and enrich fine details, the dual-branch geo-gran complementary (DGC) module to jointly capture fine-grained details and positional information in mid-level features, and the deep semantic perception (DSP) module to refine high-level positional cues via self-
attention. Additionally, a local-global guidance fusion (LGF) module is introduced to replace traditional convolutions for effective multi-level feature integration. Extensive experiments demonstrate that G2HFNet achieves high-quality saliency maps and significantly improves detection performance in challenging remote sensing scenarios.
Read the full abstract ▾
All authors Bin Wan, Runmin Cong, Xiaofei Zhou, Hao Fang, Chengtao Lv, Sam Kwong
Computer Vision
Vision Verification Enhanced Fusion of VLMs for Efficient Visual Reasoning
Selim Furkan Tekin, Yichang Xu, Gaowen Liu et al. (+3 more)
With the growing number and diversity of
Vision-
Language Models (VLMs), many works explore
language-based ensemble, collaboration, and routing techniques across multiple VLMs to improve multi-model
reasoning. In contrast, we address the diverse model selection using both
vision and
language modalities. We introduce focal error diversity to capture complementary
reasoning across VLMs and a CKA-based focal diversity metric (CKA-focal) to measure disagreement in their visual
embeddings. On the constructed ensemble surface from a pool of candidate VLMs, we applied a Genetic
Algorithm to effectively prune out those component VLMs that do not add value to the fusion performance. We identify the best combination for each task as well as fuse the outputs of each VLMs in the model pool, and show that heterogeneous models can capture epistemic uncertainty dynamically and mitigate
hallucinations. Our V3Fusion approach is capable of producing dual focal-diversity fused predictions with high performance for
vision-
language reasoning, even when there is no majority consensus or the majority of VLMs make incorrect predictions. Extensive experiments validate V3Fusion on four popular VLM
benchmarks (A-OKVQA, MMMU, MMMU-Pro, and OCR-VQA). The results show that V3Fusion outperforms the best-performing VLM on MMMU by 8.09% and MMMU-Pro by 4.87% gain in accuracy. For generative tasks, V3Fusion outperforms Intern-VL2-8b and Qwen2.5-VL-7b, the top-2 VLM performers on both A-OKVQA and OCR-VQA. Our code and
datasets are available at https://github.com/sftekin/v3fusion.
Read the full abstract ▾
All authors Selim Furkan Tekin, Yichang Xu, Gaowen Liu, Ramana Rao Kompella, Margaret L. Loper, Ling Liu
Computer Vision
Marker-Based 3D Reconstruction of Aggregates with a Comparative Analysis of 2D and 3D Morphologies
Haohang Huang, Jiayi Luo, Issam Qamhia et al. (+3 more)
Aggregates, serving as the main skeleton in assemblies of construction materials, are important functional components in various building and transportation infrastructures. They can be used in unbound layer applications, e.g. pavement base and railroad ballast, bound applications of cement concrete and asphalt concrete, and as riprap and large-sized primary crushed rocks. Information on the size and shape or morphology of aggregates can greatly facilitate the Quality Assurance/Quality Control (QA/QC) process by providing insights of aggregate behavior during composition and packing. A full 3D characterization of aggregate particle morphology is difficult both during production in a quarry and at a construction site. Many aggregate imaging approaches have been developed to quantify the particle morphology by computer
vision, including 2D image-based approaches that analyze particle silhouettes and 3D scanning-based methods that require expensive devices such as 3D
laser scanners or X-Ray Computed
Tomography (CT) equipment. This paper presents a flexible and cost-effective photogrammetry-based approach for the 3D reconstruction of aggregate particles. The proposed approach follows a marker-based design that enables background suppression, point cloud stitching, and scale referencing to obtain high-quality aggregate models. The accuracy of the reconstruction results was validated against ground-truth for selected aggregate samples. Comparative analyses were conducted on 2D and 3D morphological properties of the selected samples. Significant differences were found between the 2D and 3D statistics. Based on the presented approach, 3D shape information of aggregates can be obtained easily and at a low cost, thus allowing convenient aggregate inspection, data collection, and 3D morphological analysis.
Read the full abstract ▾
All authors Haohang Huang, Jiayi Luo, Issam Qamhia, Erol Tutumluer, John M. Hart, Andrew J. Stolba
Computer Vision
Learning Geometric and Photometric Features from Panoramic LiDAR Scans for Outdoor Place Categorization
Kazuto Nakashima, Hojung Jung, Yuki Oto et al. (+3 more)
Semantic place categorization, which is one of the essential tasks for autonomous
robots and vehicles, allows them to have capabilities of self-decision and
navigation in unfamiliar environments. In particular, outdoor places are more difficult targets than indoor ones due to perceptual variations, such as dynamic illuminance over twenty-four hours and occlusions by cars and pedestrians. This paper presents a novel method of categorizing outdoor places using convolutional neural networks (CNNs), which take omnidirectional depth/reflectance images obtained by 3D
LiDARs as the inputs. First, we construct a large-scale outdoor place
dataset named Multi-modal Panoramic 3D Outdoor (MPO) comprising two types of point clouds captured by two different
LiDARs. They are labeled with six outdoor place categories: coast, forest, indoor/outdoor parking, residential area, and urban area. Second, we provide CNNs for
LiDAR-based outdoor place categorization and evaluate our approach with the MPO
dataset. Our results on the MPO
dataset outperform traditional approaches and show the effectiveness in which we use both depth and reflectance modalities. To analyze our trained deep networks we visualize the learned features.
Read the full abstract ▾
All authors Kazuto Nakashima, Hojung Jung, Yuki Oto, Yumi Iwashita, Ryo Kurazume, Oscar Martinez Mozos
Computer Vision
AVION: Aerial Vision-Language Instruction from Offline Teacher to Prompt-Tuned Network
Yu Hu, Jianyang Gu, Hao Liu et al. (+4 more)
Adapting
vision-
language models to remote sensing imagery remains challenging due to two key factors: limited semantic coverage in textual representations and insufficient adaptability of visual features. These issues are particularly significant in aerial scenes, which involve various visual appearances and fine-grained object distinctions. We propose AVION, a knowledge
distillation framework tailored for remote sensing adaptation of
vision-
language models. The teacher module constructs semantically rich textual prototypes by collecting descriptions from a large
language model and verifying validity using remote sensing image features. The student module integrates lightweight and learnable
prompts into both
vision and
language encoders, guided by the teacher to align
embeddings and their cross-modal relationships. Once trained, the student operates independently during inference. Experiments on six optical remote sensing
benchmarks show that AVION improves few-shot classification and base-class accuracy without degrading generalization to novel categories. It also enhances mean recall for cross-modal
retrieval, with minimal additional trainable parameters.
Read the full abstract ▾
All authors Yu Hu, Jianyang Gu, Hao Liu, Yue Cao, Jozsef Hamari, Zheng Liu, Mohsen Zardadi
Computer Vision
VFM-Recon: Unlocking Cross-Domain Scene-Level Neural Reconstruction with Scale-Aligned Foundation Priors
Yuhang Ming, Tingkang Xi, Xingrui Yang et al. (+4 more)
Scene-level neural volumetric reconstruction from
monocular videos remains challenging, especially under severe domain shifts. Although recent advances in
vision foundation models (VFMs) provide transferable generalized priors learned from large-scale data, their scale-ambiguous predictions are incompatible with the scale consistency required by volumetric fusion. To address this gap, we present VFM-Recon, the first attempt to bridge transferable VFM priors with scale-consistent requirements in scene-level neural reconstruction. Specifically, we first introduce a lightweight scale
alignment stage that restores multi-view scale coherence. We then integrate pretrained VFM features into the neural volumetric reconstruction pipeline via lightweight task-specific adapters, which are trained for reconstruction while preserving the cross-domain robustness of pretrained representations. We train our model on ScanNet train split and evaluate on both in-distribution ScanNet test split and out-of-distribution TUM RGBD and Tanks and Temples
datasets. The results demonstrate that our model achieves state-of-the-art performance across all
dataset domains. In particular, on the challenging outdoor Tanks and Temples
dataset, our model achieves an F1 score of 73.5 in reconstructed mesh evaluation, substantially outperforming the closest neural volumetric reconstruction competitor, SimpleRecon, with an F1 score of 35.7, as well as the closest VFM competitor, $\pi^3$, with an F1 score of 67.0.
Read the full abstract ▾
All authors Yuhang Ming, Tingkang Xi, Xingrui Yang, Lixin Yang, Yong Peng, Cewu Lu, Wanzeng Kong
Computer Vision
VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model
Xiangyu Sun, Shijie Wang, Fengyi Zhang et al. (+5 more)
World models that forecast scene
evolution by generating future
video frames devote the bulk of their capacity to photometric details, yet the resulting predictions often remain geometrically inconsistent. We present VGGT-World, a geometry world model that side-steps
video generation entirely and instead forecasts the temporal
evolution of frozen geometry-foundation-model (GFM) features. Concretely, we repurpose the latent
tokens of a frozen VGGT as the world state and train a lightweight temporal flow
transformer to autoregressively predict their future trajectory. Two technical challenges arise in this high-dimensional (d=1024) feature space: (i) standard velocity-prediction flow matching collapses, and (ii) autoregressive rollout suffers from compounding exposure bias. We address the first with a clean-target (z-prediction) parameterization that yields a substantially higher signal-to-noise ratio, and the second with a two-stage latent flow-forcing curriculum that progressively conditions the model on its own partially denoised rollouts. Experiments on KITTI, Cityscapes, and TartanAir demonstrate that VGGT-World significantly outperforms the strongest baselines in depth
forecasting while running 3.6-5 times faster with only 0.43B trainable parameters, establishing frozen GFM features as an effective and efficient predictive state for 3D world modeling.
Read the full abstract ▾
All authors Xiangyu Sun, Shijie Wang, Fengyi Zhang, Lin Liu, Caiyan Jia, Ziying Song, Zi Huang, Yadan Luo
Computer Vision
From Sparse to Dense: Multi-View GRPO for Flow Models via Augmented Condition Space
Jiazi Bu, Pengyang Ling, Yujie Zhou et al. (+8 more)
Group Relative Policy Optimization (GRPO) has emerged as a powerful framework for preference
alignment in text-to-image (T2I) flow models. However, we observe that the standard paradigm where evaluating a group of generated samples against a single condition suffers from insufficient exploration of inter-sample relationships, constraining both
alignment efficacy and performance ceilings. To address this sparse single-view evaluation scheme, we propose Multi-View GRPO (MV-GRPO), a novel approach that enhances relationship exploration by augmenting the condition space to create a dense multi-view reward mapping. Specifically, for a group of samples generated from one
prompt, MV-GRPO leverages a flexible Condition Enhancer to generate semantically adjacent yet diverse captions. These captions enable multi-view advantage re-estimation, capturing diverse semantic attributes and providing richer optimization signals. By deriving the probability distribution of the original samples conditioned on these new captions, we can incorporate them into the training process without costly sample regeneration. Extensive experiments demonstrate that MV-GRPO achieves superior
alignment performance over state-of-the-art methods.
Read the full abstract ▾
All authors Jiazi Bu, Pengyang Ling, Yujie Zhou, Yibin Wang, Yuhang Zang, Tianyi Wei, Xiaohang Zhan, Jiaqi Wang, Tong Wu, Xingang Pan, Dahua Lin
Computer Vision
LR-SGS: Robust LiDAR-Reflectance-Guided Salient Gaussian Splatting for Self-Driving Scene Reconstruction
ZY Chen, F Zhu, H Zhu et al. (+4 more)
Recent 3D Gaussian Splatting (3DGS) methods have demonstrated the feasibility of self-driving scene reconstruction and novel view synthesis. However, most existing methods either rely solely on cameras or use
LiDAR only for Gaussian initialization or depth supervision, while the rich scene information contained in point clouds, such as reflectance, and the complementarity between
LiDAR and RGB have not been fully exploited, leading to degradation in challenging self-driving scenes, such as those with high ego-motion and complex lighting. To address these issues, we propose a robust and efficient
LiDAR-reflectance-guided Salient Gaussian Splatting method (LR-SGS) for self-driving scenes, which introduces a structure-aware Salient Gaussian representation, initialized from
geometric and reflectance feature points extracted from
LiDAR and refined through a salient transform and improved density control to capture edge and planar structures. Furthermore, we calibrate
LiDAR intensity into reflectance and attach it to each Gaussian as a lighting-invariant material channel, jointly aligned with RGB to enforce boundary consistency. Extensive experiments on the Waymo Open
Dataset demonstrate that LR-SGS achieves superior reconstruction performance with fewer Gaussians and shorter training time. In particular, on Complex Lighting scenes, our method surpasses OmniRe by 1.18 dB PSNR.
Read the full abstract ▾
All authors ZY Chen, F Zhu, H Zhu, DY Kong, XK Kuang, YJ Zhang, CM Jiang
Computer Vision
RoboStereo: Dual-Tower 4D Embodied World Models for Unified Policy Optimization
Ruicheng Zhang, Guangyu Chen, Zunnan Xu et al. (+5 more)
Scalable
Embodied AI faces fundamental constraints due to prohibitive costs and safety risks of real-world interaction. While
Embodied World Models (EWMs) offer promise through imagined rollouts, existing approaches suffer from
geometric hallucinations and lack unified optimization frameworks for practical policy improvement. We introduce RoboStereo, a symmetric dual-tower 4D world model that employs bidirectional cross-modal enhancement to ensure spatiotemporal
geometric consistency and alleviate physics
hallucinations. Building upon this high-fidelity 4D simulator, we present the first unified framework for world-model-based policy optimization: (1) Test-Time Policy Augmentation (TTPA) for pre-execution verification, (2) Imitative-Evolutionary Policy Learning (IEPL) leveraging visual perceptual rewards to learn from expert demonstrations, and (3) Open-Exploration Policy Learning (OEPL) enabling autonomous skill discovery and self-correction. Comprehensive experiments demonstrate RoboStereo achieves state-of-the-art generation quality, with our unified framework delivering >97% average relative improvement on fine-grained manipulation tasks.
Read the full abstract ▾
All authors Ruicheng Zhang, Guangyu Chen, Zunnan Xu, Zihao Liu, Zhizhou Zhong, Mingyang Zhang, Jun Zhou, Xiu Li
Computer Vision · Cross-listed from cs.IR
VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models
Ty Valencia, Burak Barlas, Varun Singhal et al. (+2 more)
Multimodal recommendation is commonly framed as a feature fusion problem, where textual and visual signals are combined to better model user preference. However, the effectiveness of
multimodal recommendation may depend not only on how modalities are fused, but also on whether item content is represented in a semantic space aligned with preference matching. This issue is particularly important because raw visual features often preserve appearance similarity, while user decisions are typically driven by higher-level semantic factors such as style, material, and usage context. Motivated by this observation, we propose LVLM-grounded
Multimodal Semantic Representation for Recommendation (VLM4Rec), a lightweight framework that organizes
multimodal item content through semantic
alignment rather than direct feature fusion. VLM4Rec first uses a large
vision-
language model to ground each item image into an explicit natural-
language description, and then encodes the grounded semantics into dense item representations for preference-oriented
retrieval. Recommendation is subsequently performed through a simple profile-based semantic matching mechanism over historical item
embeddings, yielding a practical offline-online decomposition. Extensive experiments on multiple
multimodal recommendation
datasets show that VLM4Rec consistently improves performance over raw visual features and several fusion-based alternatives, suggesting that representation quality may matter more than fusion complexity in this setting. The code is released at https://github.com/tyvalencia/enhancing-mm-rec-sys.
Read the full abstract ▾
All authors Ty Valencia, Burak Barlas, Varun Singhal, Ruchir Bhatia, Wei Yang
Computer Vision
Prompt-Driven Lightweight Foundation Model for Instance Segmentation-Based Fault Detection in Freight Trains
Guodong Sun, Qihang Liang, Xingyu Pan et al. (+2 more)
Accurate visual fault detection in freight trains remains a critical challenge for intelligent transportation system maintenance, due to complex operational environments, structurally repetitive components, and frequent occlusions or contaminations in safety-critical regions. Conventional instance
segmentation methods based on convolutional neural networks and
Transformers often suffer from poor generalization and limited boundary accuracy under such conditions. To address these challenges, we propose a lightweight self-prompted instance
segmentation framework tailored for freight train fault detection. Our method leverages the Segment Anything Model by introducing a self-
prompt generation module that automatically produces task-specific
prompts, enabling effective knowledge transfer from foundation models to domain-specific inspection tasks. In addition, we adopt a Tiny
Vision Transformer backbone to reduce computational cost, making the framework suitable for real-time deployment on edge devices in railway monitoring systems. We construct a domain-specific
dataset collected from real-world freight inspection stations and conduct extensive evaluations. Experimental results show that our method achieves 74.6 $AP^{\text{box}}$ and 74.2 $AP^{\text{mask}}$ on the
dataset, outperforming existing state-of-the-art methods in both accuracy and robustness while maintaining low computational overhead. This work offers a deployable and efficient
vision solution for automated freight train inspection, demonstrating the potential of foundation model adaptation in industrial-scale fault diagnosis scenarios. Project page: https://github.com/MVME-HBUT/SAM_FTI-FDet.git
Read the full abstract ▾
All authors Guodong Sun, Qihang Liang, Xingyu Pan, Moyun Liu, Yang Zhang
Computer Vision
Mastering Negation: Boosting Grounding Models via Grouped Opposition-Based Learning
Zesheng Yang, Xi Jiang, Bingzhang Hu et al. (+4 more)
Current
vision-
language detection and grounding models predominantly focus on
prompts with positive semantics and often struggle to accurately interpret and ground complex expressions containing negative semantics. A key reason for this limitation is the lack of high-quality training data that explicitly captures discriminative negative samples and negation-aware
language descriptions. To address this challenge, we introduce D-Negation, a new
dataset that provides objects annotated with both positive and negative semantic descriptions. Building upon the observation that negation
reasoning frequently appears in natural
language, we further propose a grouped opposition-based learning framework that learns negation-aware representations from limited samples. Specifically, our method organizes opposing semantic descriptions from D-Negation into structured groups and formulates two complementary loss functions that encourage the model to reason about negation and semantic qualifiers. We integrate the proposed
dataset and learning strategy into a state-of-the-art
language-based grounding model. By fine-tuning fewer than 10 percent of the model parameters, our approach achieves improvements of up to 4.4 mAP and 5.7 mAP on positive and negative semantic evaluations, respectively. These results demonstrate that explicitly modeling negation semantics can substantially enhance the robustness and localization accuracy of
vision-
language grounding models.
Read the full abstract ▾
All authors Zesheng Yang, Xi Jiang, Bingzhang Hu, Weili Guan, Runmin Cong, Guo-Jun Qi, Feng Zheng
Computer Vision
A2Z-10M+: Geometric Deep Learning with A-to-Z BRep Annotations for AI-Assisted CAD Modeling and Reverse Engineering
Pritham Kumar Jena, Bhavika Baburaj, Tushar Anand et al. (+3 more)
Reverse engineering and rapid prototyping of computer-aided design (CAD) models from 3D scans, sketches, or simple text
prompts are vital in industrial product design. However, recent advances in
geometric deep learning techniques lack a multi-modal understanding of parametric CAD features stored in their boundary representation (BRep). This study presents the largest compilation of 10 million multi-modal annotations and metadata for 1 million ABC CAD models, namely A2Z, to unlock an unprecedented level of BRep learning. A2Z comprises (i) high-resolution meshes with salient 3D scanning features, (ii) 3D hand-drawn sketches equipped with (iii)
geometric and topological information about BRep co-edges, corners, and surfaces, and (iv) textual captions and tags describing the product in the mechanical world. Creating such carefully structured, large-scale data, which requires nearly 5 terabytes of storage to leverage unparalleled CAD learning/
retrieval tasks, is very challenging. The scale, quality, and diversity of our multi-modal annotations are assessed using novel metrics, GPT-5, Gemini, and extensive human feedback mechanisms. To this end, we also merge an additional 25,000 CAD models of electronic enclosures (e.g., tablets, ports) designed by skilled professionals with our A2Z
dataset. Subsequently, we train and
benchmark a foundation model on a subset of 150K CAD models to detect BRep co-edges and corner vertices from 3D scans, a key downstream task in CAD reverse engineering. The annotated
dataset, metrics, and checkpoints will be publicly released to support numerous research directions.
Read the full abstract ▾
All authors Pritham Kumar Jena, Bhavika Baburaj, Tushar Anand, Vedant Dutta, Vineeth Ulavala, Sk Aziz Ali
Computer Vision
A Prediction-as-Perception Framework for 3D Object Detection
Song Zhang, Haoyu Chen and Ruibo Wang
Humans combine prediction and perception to observe the world. When faced with rapidly moving birds or insects, we can only perceive them clearly by predicting their next position and focusing our gaze there. Inspired by this, this paper proposes the Prediction-As-Perception (PAP) framework, integrating a prediction-perception architecture into 3D object perception tasks to enhance the model's perceptual accuracy. The PAP framework consists of two main modules: prediction and perception, primarily utilizing continuous frame information as input. Firstly, the prediction module forecasts the potential future positions of ego vehicles and surrounding traffic participants based on the perception results of the current frame. These predicted positions are then passed as queries to the perception module of the subsequent frame. The perceived results are iteratively fed back into the prediction module. We evaluated the PAP structure using the end-to-end model UniAD on the nuScenes
dataset. The results demonstrate that the PAP structure improves UniAD's target tracking accuracy by 10% and increases the inference speed by 15%. This indicates that such a biomimetic design significantly enhances the efficiency and accuracy of perception models while reducing computational resource consumption.
Read the full abstract ▾
Computer Vision
Neural Gate: Mitigating Privacy Risks in LVLMs via Neuron-Level Gradient Gating
Xiangkui Cao, Jie Zhang, Meina Kan et al. (+2 more)
Large
Vision-
Language Models (LVLMs) have shown remarkable potential across a wide array of
vision-
language tasks, leading to their adoption in critical domains such as finance and healthcare. However, their growing deployment also introduces significant security and
privacy risks. Malicious actors could potentially exploit these models to extract sensitive information, highlighting a critical vulnerability. Recent studies show that LVLMs often fail to consistently refuse instructions designed to compromise user
privacy. While existing work on
privacy protection has made meaningful progress in preventing the leakage of sensitive data, they are constrained by limitations in both generalization and non-destructiveness. They often struggle to robustly handle unseen
privacy-related queries and may inadvertently degrade a model's performance on standard tasks. To address these challenges, we introduce Neural Gate, a novel method for mitigating
privacy risks through
neuron-level model editing. Our method improves a model's
privacy safeguards by increasing its rate of refusal for
privacy-related questions, crucially extending this protective behavior to novel sensitive queries not encountered during the editing process. Neural Gate operates by learning a feature vector to identify
neurons associated with
privacy-related concepts within the model's representation of a subject. This localization then precisely guides the update of model parameters. Through comprehensive experiments on MiniGPT and LLaVA, we demonstrate that our method significantly boosts the model's
privacy protection while preserving its original utility. The code is available at https://github.com/Xiangkui-Cao/Neural-Gate.
Read the full abstract ▾
All authors Xiangkui Cao, Jie Zhang, Meina Kan, Shiguang Shan, Xilin Chen
Computer Vision
SDF-Net: Structure-Aware Disentangled Feature Learning for Opticall-SAR Ship Re-identification
Furui Chen, Han Wang, Yuhan Sun et al. (+5 more)
Cross-modal ship re-identification (ReID) between optical and synthetic aperture radar (SAR) imagery is fundamentally challenged by the severe radiometric discrepancy between passive optical imaging and coherent active radar sensing. While existing approaches primarily rely on statistical distribution
alignment or semantic matching, they often overlook a critical physical prior: ships are rigid objects whose
geometric structures remain stable across sensing modalities, whereas texture appearance is highly modality-dependent. In this work, we propose SDF-Net, a Structure-Aware Disentangled Feature Learning Network that systematically incorporates
geometric consistency into optical–SAR ship ReID. Built upon a ViT backbone, SDF-Net introduces a structure consistency constraint that extracts scale-invariant gradient energy statistics from intermediate layers to robustly anchor representations against radiometric variations. At the terminal stage, SDF-Net disentangles the learned representations into modality-invariant identity features and modality-specific characteristics. These decoupled cues are then integrated through a parameter-free additive residual fusion, effectively enhancing discriminative power. Extensive experiments on the HOSS-ReID
dataset demonstrate that SDF-Net consistently outperforms existing state-of-the-art methods. The code and trained models are publicly available at https://github.com/cfrfree/SDF-Net.
Read the full abstract ▾
All authors Furui Chen, Han Wang, Yuhan Sun, Jianing You, Yixuan Lv, Zhuang Zhou, Hong Tan, Shengyang Li
Computer Vision
MRGeo: Robust Cross-View Geo-Localization of Corrupted Images via Spatial and Channel Feature Enhancement
Le Wu, Lv Bo, Songsong Ouyang et al. (+1 more)
Cross-view geo-localization (CVGL) aims to accurately localize street-view images through
retrieval of corresponding geo-tagged satellite images. While prior works have achieved nearly perfect performance on certain standard
datasets, their robustness in real-world corrupted environments remains under-explored. This oversight causes severe performance degradation or failure when images are affected by corruption such as blur or weather, significantly limiting practical deployment. To address this critical gap, we introduce MRGeo, the first systematic method designed for robust CVGL under corruption. MRGeo employs a hierarchical defense strategy that enhances the intrinsic quality of features and then enforces a robust
geometric prior. Its core is the Spatial-Channel Enhancement Block, which contains: (1) a Spatial Adaptive Representation Module that models global and local features in parallel and uses a dynamic gating mechanism to arbitrate their fusion based on feature reliability; and (2) a Channel Calibration Module that performs compensatory adjustments by modeling multi-granularity channel dependencies to counteract information loss. To prevent spatial misalignment under severe corruption, a Region-level
Geometric Alignment Module imposes a
geometric structure on the final descriptors, ensuring coarse-grained consistency. Comprehensive experiments on both robustness
benchmark and standard
datasets demonstrate that MRGeo not only achieves an average R@1 improvement of 2.92% across three comprehensive robustness
benchmarks (CVUSA-C-ALL, CVACT\_val-C-ALL, and CVACT\_test-C-ALL) but also establishes superior performance in cross-area evaluation, thereby demonstrating its robustness and generalization capability.
Read the full abstract ▾
All authors Le Wu, Lv Bo, Songsong Ouyang, Yingying Zhu
Computer Vision · Cross-listed from eess.IV
Multiscale Structure-Guided Latent Diffusion for Multimodal MRI Translation
Jianqiang Lin, Zhiqiang Shen, Peng Cao et al. (+3 more)
Although
diffusion models have achieved remarkable progress in multi-modal
magnetic resonance imaging (MRI) translation tasks, existing methods still tend to suffer from anatomical inconsistencies or degraded texture details when handling arbitrary missing-modality scenarios. To address these issues, we propose a latent
diffusion-based multi-modal MRI translation framework, termed MSG-LDM. By leveraging the available modalities, the proposed method infers complete structural information, which preserves reliable boundary details. Specifically, we introduce a style–structure disentanglement mechanism in the latent space, which explicitly separates modality-specific style features from shared structural representations, and jointly models low-frequency anatomical layouts and high-frequency boundary details in a multi-scale feature space. During the structure disentanglement stage, high-frequency structural information is explicitly incorporated to enhance feature representations, guiding the model to focus on fine-grained structural cues while learning modality-invariant low-frequency anatomical representations. Furthermore, to reduce interference from modality-specific styles and improve the stability of structure representations, we design a style consistency loss and a structure-aware loss. Extensive experiments on the BraTS2020 and WMH
datasets demonstrate that the proposed method outperforms existing MRI synthesis approaches, particularly in reconstructing complete structures. The source code is publicly available at https://github.com/ziyi-start/MSG-LDM.
Read the full abstract ▾
All authors Jianqiang Lin, Zhiqiang Shen, Peng Cao, Jinzhu Yang, Osmar R. Zaiane, Xiaoli Liu
Computer Vision
DINOLight: Robust Ambient Light Normalization with Self-supervised Visual Prior Integration
Youngjin Oh, Junhyeong Kwon and Nam Ik Cho
This paper presents a new ambient light normalization framework, DINOLight, that integrates the self-supervised model DINOv2's image understanding capability into the restoration process as a visual prior. Ambient light normalization aims to restore images degraded by non-uniform shadows and lighting caused by multiple light sources and complex scene geometries. We observe that DINOv2 can reliably extract both semantic and
geometric information from a degraded image. Based on this observation, we develop a novel framework to utilize DINOv2 features for lighting normalization. First, we propose an adaptive feature fusion module that combines features from different DINOv2 layers using a point-wise softmax mask. Next, the fused features are integrated into our proposed restoration network in both spatial and frequency domains through an auxiliary cross-
attention mechanism. Experiments show that DINOLight achieves superior performance on the Ambient6K
dataset, and that DINOv2 features are effective for enhancing ambient light normalization. We also apply our method to shadow-removal
benchmark datasets, achieving competitive results compared to methods that use mask priors. Codes will be released upon acceptance.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation
Jia-Chen Zhang, Zhen-Wei Yan, Yu-Jie Xiong et al. (+1 more)
Parameter-Efficient Fine-Tuning (PEFT) has become a dominant paradigm for deploying LLMs in multi-task scenarios due to its extreme parameter efficiency. While Mixture-of-Experts (MoE) based LoRA variants have achieved promising results by dynamically routing
tokens to different low-rank experts, they largely overlook the hierarchical nature of task complexity. Existing methods typically employ experts with uniform architectures, limiting their ability to capture diverse feature granularities required by distinct tasks–where some tasks demand high-level semantic abstraction while others require fine-grained syntactic manipulation. To bridge this gap, we propose Expert Pyramid Tuning (EPT), a novel architecture that integrates the multi-scale feature pyramid concept from computer
vision into the realm of PEFT. Unlike standard LoRA, EPT decomposes task adaptation into two stages: (1) A shared meta-knowledge Subspace that encodes universal linguistic patterns in low dimensions; (2) A Pyramid Projection Mechanism that utilizes learnable up-projection operators to reconstruct high-dimensional features at varying scales. A task-aware router then dynamically selects the optimal combination of these multi-scale features. Extensive experiments across multiple multi-task
benchmarks demonstrate that EPT significantly outperforms SOTA MoE-LoRA variants. Crucially, thanks to the re-parameterization capability of our design, EPT achieves this performance improvement while simultaneously reducing the number of training parameters.
Read the full abstract ▾
All authors Jia-Chen Zhang, Zhen-Wei Yan, Yu-Jie Xiong, Chun-Ming Xia
Computer Vision
AccelAes: Accelerating Diffusion Transformers for Training-Free Aesthetic-Enhanced Image Generation
Xuanhua Yin, Chuanzhi Xu, Haoxian Zhou et al. (+2 more)
Diffusion Transformers (DiTs) are a dominant backbone for high-fidelity text-to-image generation due to strong scalability and
alignment at high resolutions. However, quadratic self-
attention over dense spatial
tokens leads to high inference latency and limits deployment. We observe that denoising is spatially non-uniform with respect to aesthetic descriptors in the
prompt. Regions associated with aesthetic
tokens receive concentrated cross-
attention and show larger temporal variation, while low-affinity regions evolve smoothly with redundant computation. Based on this insight, we propose AccelAes, a training-free framework that accelerates DiTs through aesthetics-aware spatio-temporal reduction while improving perceptual aesthetics. AccelAes builds AesMask, a one-shot aesthetic focus mask derived from
prompt semantics and cross-
attention signals. When localized computation is feasible, SkipSparse reallocates computation and guidance to masked regions. We further reduce temporal redundancy using a lightweight step-level prediction cache that periodically replaces full
Transformer evaluations. Experiments on representative DiT families show consistent acceleration and improved aesthetics-oriented quality. On Lumina-Next, AccelAes achieves a 2.11$\times$ speedup and improves ImageReward by +11.9% over the dense baseline. Code is available at https://github.com/xuanhuayin/AccelAes.
Read the full abstract ▾
All authors Xuanhua Yin, Chuanzhi Xu, Haoxian Zhou, Boyu Wei, Weidong Cai
Computer Vision · Cross-listed from stat.ML
Variational Garrote for Sparse Inverse Problems
Kanghun Lee, Hyungjoon Soh and Junghyo Jo
Sparse regularization plays a central role in solving inverse problems arising from incomplete or corrupted measurements. Different regularizers correspond to different prior assumptions about the structure of the unknown signal, and reconstruction performance depends on how well these priors match the intrinsic sparsity of the data. This work investigates the effect of sparsity priors in inverse problems by comparing conventional L1 regularization with the Variational Garrote (VG), a probabilistic method that approximates L0 sparsity through variational binary gating variables. A unified experimental framework is constructed across multiple reconstruction tasks including signal resampling, signal denoising, and sparse-view computed
tomography. To enable consistent comparison across models with different parameterizations, regularization strength is swept across wide ranges and reconstruction behavior is analyzed through train-generalization error curves. Experiments reveal characteristic bias-variance tradeoff patterns across tasks and demonstrate that VG frequently achieves lower minimum generalization error and improved stability in strongly underdetermined regimes where accurate support recovery is critical. These results suggest that sparsity priors closer to spike-and-slab structure can provide advantages when the underlying coefficient distribution is strongly sparse. The study highlights the importance of prior-data
alignment in sparse inverse problems and provides empirical insights into the behavior of variational L0-type methods across different information bottlenecks.
Read the full abstract ▾