Computer Vision
Shenghao Fu, Yukun Su, Fengyun Rao et al. (+3 more)
Aligning objects with corresponding textual descriptions is a fundamental challenge and a realistic requirement in
vision-
language understanding. While recent
multimodal embedding models excel at global image-text
alignment, they often struggle with fine-grained
alignment between image regions and specific phrases. In this work, we present ObjEmbed, a novel MLLM
embedding model that decomposes the input image into multiple regional
embeddings, each corresponding to an individual object, along with global
embeddings. It supports a wide range of visual understanding tasks like visual grounding, local image
retrieval, and global image
retrieval. ObjEmbed enjoys three key properties: (1) Object-Oriented Representation: It captures both semantic and spatial aspects of objects by generating two complementary
embeddings for each region: an object
embedding for semantic matching and an IoU
embedding that predicts localization quality. The final object matching score combines semantic similarity with the predicted IoU, enabling more accurate
retrieval. (2) Versatility: It seamlessly handles both region-level and image-level tasks. (3) Efficient Encoding: All objects in an image, along with the full image, are encoded in a single forward pass for high efficiency. Superior performance on 18 diverse
benchmarks demonstrates its strong semantic discrimination.
Read the full abstract ▾
All authors Shenghao Fu, Yukun Su, Fengyun Rao, Jing Lyu, Xiaohua Xie, Wei-Shi Zheng
Computer Vision
Tail-Aware Post-Training Quantization for 3D Geometry Models
Sicheng Pan, Chen Tang, Shuzhao Xie et al. (+6 more)
The burgeoning complexity and scale of 3D geometry models pose significant challenges for deployment on resource-constrained platforms. While Post-Training Quantization (PTQ) enables efficient inference without retraining, conventional methods, primarily optimized for 2D
Vision Transformers, fail to transfer effectively to 3D models due to intricate feature distributions and prohibitive calibration overhead. To address these challenges, we propose TAPTQ, a Tail-Aware Post-Training Quantization pipeline specifically engineered for 3D
geometric learning. Our contribution is threefold: (1) To overcome the data-scale bottleneck in 3D
datasets, we develop a progressive coarse-to-fine calibration construction strategy that constructs a highly compact subset to achieve both statistical purity and
geometric representativeness. (2) We reformulate the quantization interval search as an optimization problem and introduce a ternary-search-based solver, reducing the computational complexity from $\mathcal{O}(N)$ to $\mathcal{O}(\log N)$ for accelerated deployment. (3) To mitigate quantization error accumulation, we propose TRE-Guided Module-wise Compensation, which utilizes a Tail Relative Error (TRE) metric to adaptively identify and rectify distortions in modules sensitive to long-tailed activation outliers. Extensive experiments on the VGGT and Pi3
benchmarks demonstrate that TAPTQ consistently outperforms state-of-the-art PTQ methods in accuracy while significantly reducing calibration time. The code will be released soon.
Read the full abstract ▾
All authors Sicheng Pan, Chen Tang, Shuzhao Xie, Ke Yang, Weixiang Zhang, Jiawei Li, Bin Chen, Shu-Tao Xia, Zhi Wang
Computer Vision · Cross-listed from cs.AI
MACD: Model-Aware Contrastive Decoding via Counterfactual Data
Qixin Xiao and Kun Zhou
Video language models (
Video-LLMs) are prone to
hallucinations, generating plausible but ungrounded content when visual evidence is weak, ambiguous, or biased. Existing methods, such as contrastive decoding (CD), rely on random perturbations to construct contrastive data for
hallucination mitigation, but often fail to target the visual cues that drive
hallucination or align with model weaknesses. We propose Model-Aware Counterfactual Data based Contrastive Decoding (MACD), an inference strategy that combines model-guided counterfactual construction with contrastive decoding. MACD uses the
Video-LLM's own feedback to identify object regions most responsible for
hallucination, generating targeted object-level counterfactual inputs rather than arbitrary frame or temporal modifications. These counterfactual inputs are integrated into CD to enforce evidence-grounded
token selection during decoding. Experiments on EventHallusion, MVBench, Perception-test, and
Video-MME show that MACD consistently reduces
hallucination while maintaining or improving task accuracy across diverse
Video-LLMs, including Qwen and InternVL, with especially strong gains in scenarios involving small, occluded, or co-occurring objects.
Read the full abstract ▾
Computer Vision
Simplicity Prevails: The Emergence of Generalizable AIGI Detection in Visual Foundation Models
Yue Zhou, Xinan He, Kaiqing Lin et al. (+3 more)
While specialized detectors for AI-Generated Images (AIGI) achieve near-perfect accuracy on curated
benchmarks, they suffer from a dramatic performance collapse in realistic, in-the-wild scenarios. In this work, we demonstrate that simplicity prevails over complex architectural designs. A simple linear classifier trained on the frozen features of modern
Vision Foundation Models , including Perception Encoder, MetaCLIP 2, and DINOv3, establishes a new state-of-the-art. Through a comprehensive evaluation spanning traditional
benchmarks, unseen generators, and challenging in-the-wild distributions, we show that this baseline not only matches specialized detectors on standard
benchmarks but also decisively outperforms them on in-the-wild
datasets, boosting accuracy by striking margins of over 30%. We posit that this superior capability is an emergent property driven by the massive scale of pre-training data containing synthetic content. We trace the source of this capability to two distinct manifestations of data exposure:
Vision-
Language Models internalize an explicit semantic concept of forgery, while Self-Supervised Learning models implicitly acquire discriminative forensic features from the pretraining data. However, we also reveal persistent limitations: these models suffer from performance degradation under recapture and transmission, remain blind to VAE reconstruction and localized editing. We conclude by advocating for a paradigm shift in AI forensics, moving from overfitting on static
benchmarks to harnessing the evolving world knowledge of foundation models for real-world reliability.
Read the full abstract ▾
All authors Yue Zhou, Xinan He, Kaiqing Lin, Bing Fan, Feng Ding, Bin Li
Computer Vision
DenVisCoM: Dense Vision Correspondence Mamba for Efficient and Real-time Optical Flow and Stereo Estimation
Tushar Anand, Maheswar Bora, Antitza Dantcheva et al. (+1 more)
In this work, we propose a novel Mamba block DenVisCoM, as well as a novel hybrid architecture specifically tailored for accurate and real-time estimation of optical flow and disparity estimation. Given that such multi-view geometry and motion tasks are fundamentally related, we propose a unified architecture to tackle them jointly. Specifically, the proposed hybrid architecture is based on DenVisCoM and a
Transformer-based
attention block that efficiently addresses real-time inference,
memory footprint, and accuracy at the same time for joint estimation of motion and 3D dense perception tasks. We extensively analyze the
benchmark trade-off of accuracy and real-time processing on a large number of
datasets. Our experimental results and related analysis suggest that our proposed model can accurately estimate optical flow and disparity estimation in real time. All models and associated code are available at https://github.com/vimstereo/DenVisCoM.
Read the full abstract ▾
All authors Tushar Anand, Maheswar Bora, Antitza Dantcheva, Abhijit Das
Computer Vision
FastPhysGS: Accelerating Physics-based Dynamic 3DGS Simulation via Interior Completion and Adaptive Optimization
Yikun Ma, Yiqing Li, Jingwen Ye et al. (+4 more)
Extending 3D Gaussian Splatting (3DGS) to 4D physical simulation remains challenging. Based on the Material Point Method (MPM), existing methods either rely on manual parameter tuning or distill dynamics from
video diffusion models, limiting the generalization and optimization efficiency. Recent attempts using LLMs/VLMs suffer from a text/image-to-3D perceptual gap, yielding unstable physics behavior. In addition, they often ignore the surface structure of 3DGS, leading to implausible motion. We propose FastPhysGS, a fast and robust framework for physics-based dynamic 3DGS simulation:(1) Instance-aware Particle Filling (IPF) with Monte Carlo Importance Sampling (MCIS) to efficiently populate interior particles while preserving
geometric fidelity; (2) Bidirectional Graph Decoupling Optimization (BGDO), an adaptive strategy that rapidly optimizes material parameters predicted from a VLM. Experiments show FastPhysGS achieves high-fidelity physical simulation in 1 minute using only 7 GB runtime
memory, outperforming prior works with broad potential applications.
Read the full abstract ▾
All authors Yikun Ma, Yiqing Li, Jingwen Ye, Zhongkai Wu, Weidong Zhang, Lin Gao, Zhi Jin
Computer Vision
Physics Informed Generative AI Enabling Labour Free Segmentation For Microscopy Analysis
Salma Zahran, Zhou Ao, Zhengyang Zhang et al. (+3 more)
Semantic
segmentation of microscopy images is a critical task for high-throughput materials characterisation, yet its automation is severely constrained by the prohibitive cost, subjectivity, and scarcity of expert-annotated data. While physics-based simulations offer a scalable alternative to manual labelling, models trained on such data historically fail to generalise due to a significant domain gap, lacking the complex textures, noise patterns, and imaging artefacts inherent to experimental data. This paper introduces a novel framework for labour-free
segmentation that successfully bridges this simulation-to-reality gap. Our pipeline leverages phase-field simulations to generate an abundant source of microstructural morphologies with perfect, intrinsically-derived ground-truth masks. We then employ a Cycle-Consistent Generative
Adversarial Network (CycleGAN) for unpaired image-to-image translation, transforming the clean simulations into a large-scale
dataset of high-fidelity, realistic SEM images. A U-Net model, trained exclusively on this synthetic data, demonstrated remarkable generalisation when deployed on unseen experimental images, achieving a mean Boundary F1-Score of 0.90 and an Intersection over Union (IOU) of 0.88. Comprehensive validation using t-SNE feature-space projection and Shannon
entropy analysis confirms that our synthetic images are statistically and featurally indistinguishable from the real data
manifold. By completely decoupling model training from manual annotation, our generative framework transforms a data-scarce problem into one of data abundance, providing a robust and fully automated solution to accelerate materials discovery and analysis.
Read the full abstract ▾
All authors Salma Zahran, Zhou Ao, Zhengyang Zhang, Chen Chi, Chenchen Yuan, Yanming Wang
Computer Vision
Cross-Modal Purification and Fusion for Small-Object RGB-D Transmission-Line Defect Detection
Jiaming Cui, Wenqiang Li, Shuai Zhou et al. (+2 more)
Transmission line defect detection remains challenging for automated
UAV inspection due to the dominance of small-scale defects, complex backgrounds, and illumination variations. Existing RGB-based detectors, despite recent progress, struggle to distinguish geometrically subtle defects from visually similar background structures under limited chromatic contrast. This paper proposes CMAFNet, a Cross-Modal
Alignment and Fusion Network that integrates RGB appearance and depth geometry through a principled purify-then-fuse paradigm. CMAFNet consists of a Semantic Recomposition Module that performs dictionary-based feature purification via a learned codebook to suppress modality-specific noise while preserving defect-discriminative information, and a Contextual Semantic Integration Framework that captures global spatial dependencies using partial-channel
attention to enhance structural semantic
reasoning. Position-wise normalization within the purification stage enforces explicit reconstruction-driven cross-modal
alignment, ensuring statistical compatibility between heterogeneous features prior to fusion. Extensive experiments on the TLRGBD
benchmark, where 94.5% of instances are small objects, demonstrate that CMAFNet achieves 32.2% mAP@50 and 12.5% APs, outperforming the strongest baseline by 9.8 and 4.0 percentage points, respectively. A lightweight variant reaches 24.8% mAP50 at 228 FPS with only 4.9M parameters, surpassing all YOLO-based detectors while matching
transformer-based methods at substantially lower computational cost.
Read the full abstract ▾
All authors Jiaming Cui, Wenqiang Li, Shuai Zhou, Ruifeng Qin, Feng Shen
Computer Vision
FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding
Kangcong Li, Peng Ye, Lin Zhang et al. (+4 more)
Transitioning
Multimodal Large
Language Models (MLLMs) from offline to online streaming
video understanding is essential for continuous perception. However, existing methods lack flexible adaptivity, leading to irreversible detail loss and context fragmentation. To resolve this, we propose FreshMem, a Frequency-Space Hybrid
Memory network inspired by the
brain's logarithmic perception and
memory consolidation. FreshMem reconciles short-term fidelity with long-term coherence through two synergistic modules: Multi-scale Frequency
Memory (MFM), which projects overflowing frames into representative frequency coefficients, complemented by residual details to reconstruct a global historical "gist"; and Space Thumbnail
Memory (STM), which discretizes the continuous stream into episodic clusters by employing an adaptive compression strategy to distill them into high-density space thumbnails. Extensive experiments show that FreshMem significantly boosts the Qwen2-VL baseline, yielding gains of 5.20%, 4.52%, and 2.34% on StreamingBench, OV-Bench, and OVO-Bench, respectively. As a training-free solution, FreshMem outperforms several fully fine-tuned methods, offering a highly efficient paradigm for long-horizon streaming
video understanding.
Read the full abstract ▾
All authors Kangcong Li, Peng Ye, Lin Zhang, Chao Wang, Huafeng Qin, Jiayuan Fan, Tao Chen
Computer Vision · Cross-listed from eess.IV
Hyperspectral Image Fusion with Spectral-Band and Fusion-Scale Agnosticism
Yu-Jie Liang, Zihan Cao, Liang-Jian Deng et al. (+2 more)
Current deep learning models for Multispectral and Hyperspectral Image Fusion (MS/HS fusion) are typically designed for fixed spectral bands and spatial scales, which limits their transferability across diverse sensors. To address this, we propose SSA, a universal framework for MS/HS fusion with spectral-band and fusion-scale agnosticism. Specifically, we introduce Matryoshka Kernel (MK), a novel operator that enables a single model to adapt to arbitrary numbers of spectral channels. Meanwhile, we build SSA upon an Implicit Neural Representation (INR) backbone that models the HS signal as a continuous function, enabling reconstruction at arbitrary spatial resolutions. Together, these two forms of agnosticism enable a single MS/HS fusion model that generalizes effectively to unseen sensors and spatial scales. Extensive experiments demonstrate that our single model achieves state-of-the-art performance while generalizing well to unseen sensors and scales, paving the way toward future HS foundation models.
Read the full abstract ▾
All authors Yu-Jie Liang, Zihan Cao, Liang-Jian Deng, Yang Yang, Malu Zhang
Computer Vision · Cross-listed from cs.RO
Towards Autonomous Instrument Tray Assembly for Sterile Processing Applications
Raghavasimhan Sankaranarayanan, Paul Stuart, Nicholas Ahn et al. (+2 more)
The Sterile Processing and Distribution (SPD) department is responsible for cleaning, disinfecting, inspecting, and assembling
surgical instruments between surgeries. Manual inspection and preparation of instrument trays is a time-consuming, error-prone task, often prone to contamination and instrument breakage. In this work, we present a fully automated robotic system that sorts and structurally packs
surgical instruments into sterile trays, focusing on automation of the SPD assembly stage. A custom
dataset comprising 31
surgical instruments and 6,975 annotated images was collected to train a hybrid perception pipeline using YOLO12 for detection and a cascaded ResNet-based model for fine-grained classification. The system integrates a calibrated
vision module, a 6-DOF Staubli TX2-60L robotic arm with a custom dual electromagnetic gripper, and a rule-based packing
algorithm that reduces instrument collisions during transport. The packing framework uses 3D printed dividers and holders to physically isolate instruments, reducing collision and friction during transport. Experimental evaluations show high perception accuracy and statistically significant reduction in tool-to-tool collisions compared to human-assembled trays. This work serves as the scalable first step toward automating SPD workflows, improving safety, and consistency of
surgical preparation while reducing SPD processing times.
Read the full abstract ▾
All authors Raghavasimhan Sankaranarayanan, Paul Stuart, Nicholas Ahn, Arno Sungarian, Yash Chitalia
Computer Vision
SMTrack: State-Aware Mamba for Efficient Temporal Modeling in Visual Tracking
Yinchao Ma, Dengqing Yang, Zhangyu He et al. (+2 more)
Visual tracking aims to automatically estimate the state of a target object in a
video sequence, which is challenging especially in dynamic scenarios. Thus, numerous methods are proposed to introduce temporal cues to enhance tracking robustness. However, conventional CNN and
Transformer architectures exhibit inherent limitations in modeling long-range temporal dependencies in visual tracking, often necessitating either complex customized modules or substantial computational costs to integrate temporal cues. Inspired by the success of the state space model, we propose a novel temporal modeling paradigm for visual tracking, termed State-aware Mamba Tracker (SMTrack), providing a neat pipeline for training and tracking without needing customized modules or substantial computational costs to build long-range temporal dependencies. It enjoys several merits. First, we propose a novel selective state-aware space model with state-wise parameters to capture more diverse temporal cues for robust tracking. Second, SMTrack facilitates long-range temporal interactions with linear computational complexity during training. Third, SMTrack enables each frame to interact with previously tracked frames via hidden state propagation and updating, which releases computational costs of handling temporal cues during tracking. Extensive experimental results demonstrate that SMTrack achieves promising performance with low computational costs.
Read the full abstract ▾
All authors Yinchao Ma, Dengqing Yang, Zhangyu He, Wenfei Yang, Tianzhu Zhang
Computer Vision
VRGaussianAvatar: Integrating 3D Gaussian Avatars into VR
Hail Song, Boram Yoon, Seokhwan Yang et al. (+4 more)
We present VRGaussianAvatar, an integrated system that enables real-time full-body 3D Gaussian Splatting (3DGS) avatars in virtual reality using only head-mounted display (HMD) tracking signals. The system adopts a parallel pipeline with a VR Frontend and a GA Backend. The VR Frontend uses inverse kinematics to estimate full-body pose and streams the resulting pose along with stereo camera parameters to the backend. The GA Backend stereoscopically renders a 3DGS avatar reconstructed from a single image. To improve stereo rendering efficiency, we introduce Binocular Batching, which jointly processes left and right eye views in a single batched pass to reduce redundant computation and support high-resolution VR displays. We evaluate VRGaussianAvatar with quantitative performance tests and a within-subject user study against image- and
video-based mesh avatar baselines. Results show that VRGaussianAvatar sustains interactive VR performance and yields higher perceived appearance similarity, embodiment, and plausibility. Project page and source code are available at https://vrgaussianavatar.github.io.
Read the full abstract ▾
All authors Hail Song, Boram Yoon, Seokhwan Yang, Seoyoung Kang, Hyunjeong Kim, Henning Metzmacher, Woontack Woo
Computer Vision
Real-Time Loop Closure Detection in Visual SLAM via NetVLAD and Faiss
Enguang Fan
Loop closure detection (LCD) is a core component of simultaneous localization and mapping (SLAM): it identifies revisited places and enables pose-graph constraints that correct accumulated drift. Classic bag-of-words approaches such as DBoW are efficient but often degrade under appearance change and perceptual aliasing. In parallel, deep learning-based visual place recognition (VPR) descriptors (e.g., NetVLAD and
Transformer-based models) offer stronger robustness, but their computational cost is often viewed as a barrier to real-time SLAM. In this paper, we empirically evaluate NetVLAD as an LCD module and compare it against DBoW on the KITTI
dataset. We introduce a Fine-Grained Top-K precision-recall curve that better reflects LCD settings where a query may have zero or multiple valid matches. With Faiss-accelerated nearestneighbor search, NetVLAD achieves real-time query speed while improving accuracy and robustness over DBoW, making it a practical drop-in alternative for LCD in SLAM.
Read the full abstract ▾
Computer Vision
Moonworks Lunara Aesthetic II: An Image Variation Dataset
Yan Wang, Partho Hassan, Samiha Sadeka et al. (+3 more)
We introduce Lunara Aesthetic II, a publicly released, ethically sourced image
dataset designed to support controlled evaluation and learning of contextual consistency in modern image generation and editing systems. The
dataset comprises 2,854 anchor-linked variation pairs derived from original art and photographs created by Moonworks. Each variation pair applies contextual transformations, such as illumination, weather, viewpoint, scene composition, color tone, or mood; while preserving a stable underlying identity. Lunara Aesthetic II operationalizes identity-preserving contextual variation as a supervision signal while also retaining Lunara's signature high aesthetic scores. Results show high identity stability, strong target attribute realization, and a robust aesthetic profile that exceeds large-scale web
datasets. Released under the Apache 2.0 license, Lunara Aesthetic II is intended for benchmarking, fine-tuning, and analysis of contextual generalization, identity preservation, and edit robustness in image generation and image-to-image systems with interpretable, relational supervision. The
dataset is publicly available at: https://huggingface.co/
datasets/moonworks/lunara-aesthetic-image-variations.
Read the full abstract ▾
All authors Yan Wang, Partho Hassan, Samiha Sadeka, Nada Soliman, Sayeef Abdullah, Sabit Hassan
Computer Vision
From Frames to Sequences: Temporally Consistent Human-Centric Dense Prediction
Xingyu Miao, Junting Dong, Qin Zhao et al. (+3 more)
In this work, we focus on the challenge of temporally consistent human-centric dense prediction across
video sequences. Existing models achieve strong per-frame accuracy but often flicker under motion, occlusion, and lighting changes, and they rarely have paired human
video supervision for multiple dense tasks. We address this gap with a scalable synthetic data pipeline that generates photorealistic human frames and motion-aligned sequences with pixel-accurate depth, normals, and masks. Unlike prior static data synthetic pipelines, our pipeline provides both frame-level labels for spatial learning and sequence-level supervision for temporal learning. Building on this, we train a unified ViT-based dense predictor that (i) injects an explicit human
geometric prior via CSE
embeddings and (ii) improves geometry-feature reliability with a lightweight channel reweighting module after feature fusion. Our two-stage training strategy, combining static pretraining with dynamic sequence supervision, enables the model first to acquire robust spatial representations and then refine temporal consistency across motion-aligned sequences. Extensive experiments show that we achieve state-of-the-art performance on THuman2.1 and Hi4D and generalize effectively to in-the-wild
videos.
Read the full abstract ▾
All authors Xingyu Miao, Junting Dong, Qin Zhao, Yuhang Yang, Junhao Chen, Yang Long
Computer Vision
Contribution-aware Token Compression for Efficient Video Understanding via Reinforcement Learning
Yinchao Ma, Qiang Zhou, Zhibin Wang et al. (+4 more)
Video large
language models have demonstrated remarkable capabilities in
video understanding tasks. However, the redundancy of
video tokens introduces significant computational overhead during inference, limiting their practical deployment. Many compression
algorithms are proposed to prioritize retaining features with the highest
attention scores to minimize perturbations in
attention computations. However, the correlation between
attention scores and their actual contribution to correct answers remains ambiguous. To address the above limitation, we propose a novel Contribution-aware
token Compression
algorithm for
VIDeo understanding (CaCoVID) that explicitly optimizes the
token selection policy based on the contribution of
tokens to correct predictions. First, we introduce a reinforcement learning-based framework that optimizes a policy network to select
video token combinations with the greatest contribution to correct predictions. This paradigm shifts the focus from passive
token preservation to active discovery of optimal compressed
token combinations. Secondly, we propose a combinatorial policy optimization
algorithm with online combination space sampling, which dramatically reduces the exploration space for
video token combinations and accelerates the convergence speed of policy optimization. Extensive experiments on diverse
video understanding
benchmarks demonstrate the effectiveness of CaCoVID. Codes are available at https://github.com/LivingFutureLab/CaCoVID.
Read the full abstract ▾
All authors Yinchao Ma, Qiang Zhou, Zhibin Wang, Xianing Chen, Hanqing Yang, Jun Song, Bo Zheng
Computer Vision · Cross-listed from cs.LG
From Perception to Action: Spatial AI Agents and World Models
Gloria Felicia, Nolan Bryant, Handi Putra et al. (+3 more)
While large
language models have become the prevailing approach for agentic
reasoning and planning, their success in symbolic domains does not readily translate to the physical world. Spatial intelligence, the ability to perceive 3D structure, reason about object relationships, and act under physical constraints, is an orthogonal capability that proves important for
embodied agents. Existing surveys address either agentic architectures or spatial domains in isolation. None provide a unified framework connecting these complementary capabilities. This paper bridges that gap. Through a thorough review of over 2,000 papers, citing 742 works from top-tier venues, we introduce a unified three-axis taxonomy connecting agentic capabilities with spatial tasks across scales. Crucially, we distinguish spatial grounding (metric understanding of geometry and physics) from symbolic grounding (associating images with text), arguing that perception alone does not confer agency. Our analysis reveals three key findings mapped to these axes: (1) hierarchical
memory systems (Capability axis) are important for long-horizon spatial tasks. (2) GNN-LLM integration (Task axis) is a promising approach for structured spatial
reasoning. (3) World models (Scale axis) are essential for safe deployment across micro-to-macro spatial scales. We conclude by identifying six grand challenges and outlining directions for future research, including the need for unified evaluation frameworks to standardize cross-domain assessment. This taxonomy provides a foundation for unifying fragmented research efforts and enabling the next generation of spatially-aware autonomous systems in robotics, autonomous vehicles, and geospatial intelligence.
Read the full abstract ▾
All authors Gloria Felicia, Nolan Bryant, Handi Putra, Ayaan Gazali, Eliel Lobo, Esteban Rojas
Computer Vision
ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval
Tianyu Yang, Chenwei He, Xiangzhao Hao et al. (+6 more)
Composed Image
Retrieval (CIR) aims to retrieve target images based on a hybrid query comprising a reference image and a modification text. Early dual-tower
Vision-
Language Models (VLMs) struggle with cross-modality compositional
reasoning required for this task. While adapting generative
Multimodal Large
Language Models (MLLMs) for
retrieval offers a promising direction, we identify that this strategy overlooks a fundamental issue: compressing a generative MLLM into a single-
embedding discriminative retriever triggers a paradigm conflict, which leads to Capability Degradation - the deterioration of native fine-grained
reasoning after
retrieval adaptation. To address this challenge, we propose ReCALL, a model-agnostic framework that follows a diagnose-generate-refine pipeline: First, we diagnose cognitive blind spots of the retriever via self-guided informative instance mining. Next, we generate corrective instructions and triplets by prompting the foundation MLLM and conduct quality control with VQA-based consistency filtering. Finally, we refine the retriever through continual training on these triplets with a grouped contrastive scheme, thereby internalizing fine-grained visual-semantic distinctions and realigning the discriminative
embedding space of retriever with intrinsic compositional
reasoning within the MLLM. Extensive experiments on CIRR and FashionIQ show that ReCALL consistently recalibrates degraded capabilities and achieves state-of-the-art performance. Code is available at https://github.com/RemRico/Recall.
Read the full abstract ▾
All authors Tianyu Yang, Chenwei He, Xiangzhao Hao, Tianyue Wang, Jiarui Guo, Haiyun Guo, Leigang Qu, Jinqiao Wang, Tat-Seng Chua
Computer Vision
Federated Vision Transformer with Adaptive Focal Loss for Medical Image Classification
Xinyuan Zhao, Yihang Wu, Ahmad Chaddad et al. (+2 more)
While deep learning models like
Vision Transformer (ViT) have achieved significant advances, they typically require large
datasets. With data
privacy regulations, access to many original
datasets is restricted, especially medical images.
Federated learning (FL) addresses this challenge by enabling global model aggregation without data exchange. However, the heterogeneity of the data and the class imbalance that exist in local clients pose challenges for the generalization of the model. This study proposes a FL framework leveraging a dynamic adaptive focal loss (DAFL) and a client-aware aggregation strategy for local training. Specifically, we design a dynamic class imbalance coefficient that adjusts based on each client's sample distribution and class data distribution, ensuring minority classes receive sufficient
attention and preventing sparse data from being ignored. To address client heterogeneity, a weighted aggregation strategy is adopted, which adapts to data size and characteristics to better capture inter-client variations. The classification results on three public
datasets (ISIC, Ocular Disease and RSNA-ICH) show that the proposed framework outperforms DenseNet121, ResNet50, ViT-S/16, ViT-L/32, FedCLIP, Swin
Transformer, CoAtNet, and MixNet in most cases, with accuracy improvements ranging from 0.98% to 41.69%. Ablation studies on the imbalanced ISIC
dataset validate the effectiveness of the proposed loss function and aggregation strategy compared to traditional loss functions and other FL approaches. The codes can be found at: https://github.com/AIPMLab/ViT-FLDAF.
Read the full abstract ▾
All authors Xinyuan Zhao, Yihang Wu, Ahmad Chaddad, Tareef Daqqaq, Reem Kateb
Computer Vision
Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks
Bohan Zeng, Kaixin Zhu, Daili Hua et al. (+24 more)
World models have emerged as a critical frontier in AI research, aiming to enhance large models by infusing them with physical dynamics and world knowledge. The core objective is to enable
agents to understand, predict, and interact with complex environments. However, current research landscape remains fragmented, with approaches predominantly focused on injecting world knowledge into isolated tasks, such as visual prediction, 3D estimation, or symbol grounding, rather than establishing a unified definition or framework. While these task-specific integrations yield performance gains, they often lack the systematic coherence required for holistic world understanding. In this paper, we analyze the limitations of such fragmented approaches and propose a unified design specification for world models. We suggest that a robust world model should not be a loose collection of capabilities but a normative framework that integrally incorporates interaction, perception, symbolic
reasoning, and spatial representation. This work aims to provide a structured perspective to guide future research toward more general, robust, and principled models of the world.
Read the full abstract ▾
All authors Bohan Zeng, Kaixin Zhu, Daili Hua, Bozhou Li, Chengzhuo Tong, Yuran Wang, Xinyi Huang, Yifan Dai, Zixiang Zhang, Yifan Yang, Zhou Liu, Hao Liang, Xiaochen Ma, Ruichuan An, Tianyi Bai, Hongcheng Gao, Junbo Niu, Yang Shi, Xinlong Chen, Yue Ding, Minglei Shi, Kai Zeng, Yiwen Tang, Yuanxing Zhang, Pengfei Wan, Xintao Wang, Wentao Zhang
Computer Vision
PISCES: Annotation-free Text-to-Video Post-Training via Optimal Transport-Aligned Rewards
Minh-Quan Le, Gaurav Mittal, Cheng Zhao et al. (+3 more)
Text-to-
video (T2V) generation aims to synthesize
videos with high visual quality and temporal consistency that are semantically aligned with input text. Reward-based post-training has emerged as a promising direction to improve the quality and semantic
alignment of generated
videos. However, recent methods either rely on large-scale human preference annotations or operate on misaligned
embeddings from pre-trained
vision-
language models, leading to limited scalability or suboptimal supervision. We present $\texttt{PISCES}$, an annotation-free post-training
algorithm that addresses these limitations via a novel Dual Optimal Transport (OT)-aligned Rewards module. To align reward signals with human judgment, $\texttt{PISCES}$ uses OT to bridge text and
video embeddings at both distributional and discrete
token levels, enabling reward supervision to fulfill two objectives: (i) a Distributional OT-aligned Quality Reward that captures overall visual quality and temporal coherence; and (ii) a Discrete
Token-level OT-aligned Semantic Reward that enforces semantic, spatio-temporal correspondence between text and
video tokens. To our knowledge, $\texttt{PISCES}$ is the first to improve annotation-free reward supervision in generative post-training through the lens of OT. Experiments on both short- and long-
video generation show that $\texttt{PISCES}$ outperforms both annotation-based and annotation-free methods on VBench across Quality and Semantic scores, with human preference studies further validating its effectiveness. We show that the Dual OT-aligned Rewards module is compatible with multiple optimization paradigms, including direct backpropagation and reinforcement learning fine-tuning. Project page: https://roar-ai.github.io/pisces
Read the full abstract ▾
All authors Minh-Quan Le, Gaurav Mittal, Cheng Zhao, David Gu, Dimitris Samaras, Mei Chen
Computer Vision
Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?
Susan Liang, Chao Huang, Filippos Bellos et al. (+7 more)
State-of-the-art text-to-
video generation models such as Sora 2 and Veo 3 can now produce high-fidelity
videos with synchronized audio directly from a textual
prompt, marking a new milestone in multi-modal generation. However, evaluating such tri-modal outputs remains an unsolved challenge. Human evaluation is reliable but costly and difficult to scale, while traditional automatic metrics, such as FVD, CLAP, and ViCLIP, focus on isolated modality pairs, struggle with complex
prompts, and provide limited interpretability. Omni-modal large
language models (omni-LLMs) present a promising alternative: they naturally process audio,
video, and text, support rich
reasoning, and offer interpretable chain-of-thought feedback. Driven by this, we introduce Omni-Judge, a study assessing whether omni-LLMs can serve as human-aligned judges for text-conditioned audio-
video generation. Across nine perceptual and
alignment metrics, Omni-Judge achieves correlation comparable to traditional metrics and excels on semantically demanding tasks such as audio-text
alignment,
video-text
alignment, and audio-
video-text coherence. It underperforms on high-FPS perceptual metrics, including
video quality and audio-
video synchronization, due to limited temporal resolution. Omni-Judge provides interpretable explanations that expose semantic or physical inconsistencies, enabling practical downstream uses such as feedback-based refinement. Our findings highlight both the potential and current limitations of omni-LLMs as unified evaluators for multi-modal generation.
Read the full abstract ▾
All authors Susan Liang, Chao Huang, Filippos Bellos, Yolo Yunlong Tang, Qianxiang Shen, Jing Bi, Luchuan Song, Zeliang Zhang, Jason Corso, Chenliang Xu
Computer Vision
Junqing Lin, Xingyu Zheng, Pei Cheng et al. (+3 more)
In-context generation significantly enhances
Diffusion Transformers (DiTs) by enabling controllable image-to-image generation through reference examples. However, the resulting input concatenation drastically increases sequence length, creating a substantial computational bottleneck. Existing
token reduction techniques, primarily tailored for text-to-image synthesis, fall short in this paradigm as they apply uniform reduction strategies, overlooking the inherent role asymmetry between reference contexts and target latents across spatial, temporal, and functional dimensions. To bridge this gap, we introduce ToPi, a training-free
token pruning framework tailored for in-context generation in DiTs. Specifically, ToPi utilizes offline calibration-driven sensitivity analysis to identify pivotal
attention layers, serving as a robust proxy for redundancy estimation. Leveraging these layers, we derive a novel influence metric to quantify the contribution of each context
token for selective
pruning, coupled with a temporal update strategy that adapts to the evolving
diffusion trajectory. Empirical evaluations demonstrate that ToPi can achieve over 30% speedup in inference while maintaining structural fidelity and visual consistency across complex image generation tasks.
Read the full abstract ▾
All authors Junqing Lin, Xingyu Zheng, Pei Cheng, Bin Fu, Jingwei Sun, Guangzhong Sun
Computer Vision
UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perception
Wenzhuo Liu, Qiannan Guo, Zhen Wang et al. (+9 more)
Advanced Driver Assistance Systems (ADAS) need to understand human driver behavior while perceiving their
navigation context, but jointly learning these heterogeneous tasks would cause inter-task negative transfer and impair system performance. Here, we propose a Unified and Versatile
Multimodal Multi-Task Learning (UV-M3TL) framework to simultaneously recognize driver behavior, driver
emotion, vehicle behavior, and traffic context, while mitigating inter-task negative transfer. Our framework incorporates two core components: dual-branch spatial channel
multimodal embedding (DB-SCME) and adaptive feature-decoupled multi-task loss (AFD-Loss). DB-SCME enhances cross-task knowledge transfer while mitigating task conflicts by employing a dual-branch structure to explicitly model salient task-shared and task-specific features. AFD-Loss improves the stability of joint optimization while guiding the model to learn diverse multi-task representations by introducing an adaptive weighting mechanism based on learning dynamics and feature decoupling constraints. We evaluate our method on the AIDE
dataset, and the experimental results demonstrate that UV-M3TL achieves state-of-the-art performance across all four tasks. To further prove the versatility, we evaluate UV-M3TL on additional public multi-task perception
benchmarks (BDD100K, CityScapes, NYUD-v2, and PASCAL-Context), where it consistently delivers strong performance across diverse task combinations, attaining state-of-the-art results on most tasks.
Read the full abstract ▾
All authors Wenzhuo Liu, Qiannan Guo, Zhen Wang, Wenshuo Wang, Lei Yang, Yicheng Qiao, Lening Wang, Zhiwei Li, Chen Lv, Shanghang Zhang, Junqiang Xi, Huaping Liu
Computer Vision
Samba+: General and Accurate Salient Object Detection via A More Unified Mamba-based Framework
Wenzhuo Zhao, Keren Fu, Jiahao He et al. (+3 more)
Existing salient object detection (SOD) models are generally constrained by the limited receptive fields of convolutional neural networks (CNNs) and quadratic computational complexity of
Transformers. Recently, the emerging state-space model, namely Mamba, has shown great potential in balancing global receptive fields and computational efficiency. As a solution, we propose Saliency Mamba (Samba), a pure Mamba-based architecture that flexibly handles various distinct SOD tasks, including RGB/RGB-D/RGB-T SOD,
video SOD (VSOD), RGB-D VSOD, and visible-depth-thermal SOD. Specifically, we rethink the scanning strategy of Mamba for SOD, and introduce a saliency-guided Mamba block (SGMB) that features a spatial neighborhood scanning (SNS)
algorithm to preserve the spatial continuity of salient regions. A context-aware upsampling (CAU) method is also proposed to promote hierarchical feature
alignment and aggregation by modeling contextual dependencies. As one step further, to avoid the "task-specific" problem as in previous SOD solutions, we develop Samba+, which is empowered by training Samba in a multi-task joint manner, leading to a more unified and versatile model. Two crucial components that collaboratively tackle challenges encountered in input of arbitrary modalities and continual adaptation are investigated. Specifically, a hub-and-spoke graph
attention (HGA) module facilitates adaptive cross-modal interactive fusion, and a modality-anchored continual learning (MACL) strategy alleviates inter-modal conflicts together with catastrophic forgetting. Extensive experiments demonstrate that Samba individually outperforms existing methods across six SOD tasks on 22
datasets with lower computational cost, whereas Samba+ achieves even superior results on these tasks and
datasets by using a single trained versatile model. Additional results further demonstrate the potential of our Samba framework.
Read the full abstract ▾
All authors Wenzhuo Zhao, Keren Fu, Jiahao He, Xiaohong Liu, Qijun Zhao, Guangtao Zhai
Computer Vision
Faster and Better Alignment for Flow Matching Models via Step-aware Advantages
Zhixiong Yue, Feiyang Ye, Zixuan Ni et al. (+2 more)
Recent advances in flow matching models, particularly with reinforcement learning (RL), have significantly enhanced human preference
alignment in few-step text-to-image generators. However, existing RL-based approaches for flow matching models typically rely on numerous denoising steps, while suffering from sparse and imprecise reward signals that often lead to suboptimal
alignment. To address these limitations, we propose Temperature-Annealed Few-step Sampling with Group Relative Policy Optimization (TAFS-GRPO), a novel framework for training flow matching text-to-image models into efficient few-step generators well aligned with human preferences. Our method iteratively injects adaptive time-dependent noise into one-step clean predictions. By repeatedly annealing the model's sampled outputs, it introduces stochasticity into the sampling process while preserving the semantic integrity of each generated image. Moreover, its step-aware advantage integration mechanism combines GRPO with temperature-annealed sampling to eliminate the need for a differentiable reward function and provide dense, step-specific rewards for stable policy optimization. Extensive experiments demonstrate that TAFS-GRPO achieves strong performance in few-step text-to-image generation and significantly improves the
alignment of generated images with human preferences. The code and models of this work will be available to facilitate further research.
Read the full abstract ▾
All authors Zhixiong Yue, Feiyang Ye, Zixuan Ni, Sheng Shen, Yu Zhang
Computer Vision · Cross-listed from cs.GR
Two-chart Beltrami Optimization for Distortion-Controlled Spherical Bijection with Application to Brain Surface Registration
Zhehao Xu and Lok Ming Lui
Many genus-0 surface mapping tasks such as landmark
alignment, feature matching, and image-driven registration, can be reduced (via an initial spherical conformal map) to optimizing a spherical self-homeomorphism with controlled distortion. However, existing works lack efficient mechanisms to control the
geometric distortion of the resulting mapping. To resolve this issue, we formulate this as a Beltrami-space optimization problem, where the angle distortion is encoded explicitly by the Beltrami differential and bijectivity can be enforced through the constraint $\|\mu\|_{\infty}<1$. To make this practical on the sphere, we introduce the Spherical Beltrami Differential (SBD), a two-chart representation of quasiconformal self-maps of the unit sphere $\mathbb{S}^2$, together with cross-chart consistency conditions that yield a globally bijective spherical deformation (up to conformal
automorphisms). Building on the Spectral Beltrami Network, we develop BOOST, a differentiable optimization framework that updates two Beltrami fields to minimize task-driven losses while regularizing distortion and enforcing consistency along the seam. Experiments on large-deformation landmark matching and intensity-based spherical registration demonstrate improved task performance meanwhile maintaining controlled distortion and robust bijective behavior. We also apply the method to cortical surface registration by aligning sulcal landmarks and matching cortical sulcal depth, achieving comparative or better registration performance without sacrificing
geometric validity.
Read the full abstract ▾
Computer Vision
HandMCM: Multi-modal Point Cloud-based Correspondence State Space Model for 3D Hand Pose Estimation
Wencan Cheng and Gim Hee Lee
3D hand pose estimation that involves accurate estimation of 3D human hand keypoint locations is crucial for many human-computer interaction applications such as augmented reality. However, this task poses significant challenges due to self-occlusion of the hands and occlusions caused by interactions with objects. In this paper, we propose HandMCM to address these challenges. Our HandMCM is a novel method based on the powerful state space model (Mamba). By incorporating modules for local information injection/filtering and correspondence modeling, the proposed correspondence Mamba effectively learns the highly dynamic kinematic
topology of keypoints across various occlusion scenarios. Moreover, by integrating multi-modal image features, we enhance the robustness and representational capacity of the input, leading to more accurate hand pose estimation. Empirical evaluations on three
benchmark datasets demonstrate that our model significantly outperforms current state-of-the-art methods, particularly in challenging scenarios involving severe occlusions. These results highlight the potential of our approach to advance the accuracy and reliability of 3D hand pose estimation in practical applications.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.SP
Visible Light Positioning With Lamé Curve LEDs: A Generic Approach for Camera Pose Estimation
Wenxuan Pan, Yang Yang, Dong Wei et al. (+4 more)
Camera-based visible light positioning (VLP) is a promising technique for accurate and low-cost indoor camera pose estimation (CPE). To reduce the number of required light-emitting diodes (LEDs), advanced methods commonly exploit LED shape features for positioning. Although interesting, they are typically restricted to a single LED geometry, leading to failure in heterogeneous LED-shape scenarios. To address this challenge, this paper investigates Lamé curves as a unified representation of common LED shapes and proposes a generic VLP
algorithm using Lamé curve-shaped LEDs, termed LC-VLP. In the considered system, multiple ceiling-mounted Lamé curve-shaped LEDs periodically broadcast their curve parameters via visible light communication, which are captured by a camera-equipped receiver. Based on the received LED images and curve parameters, the receiver can estimate the camera pose using LC-VLP. Specifically, an LED database is constructed offline to store the curve parameters, while online positioning is formulated as a nonlinear least-squares problem and solved iteratively. To provide a reliable initialization, a correspondence-free perspective-n-points (FreePnP)
algorithm is further developed, enabling approximate CPE without any pre-calibrated reference points. The performance of LC-VLP is verified by both simulations and experiments. Simulations show that LC-VLP outperforms state-of-the-art methods in both circular- and rectangular-LED scenarios. Compared to a perspective arcs
algorithm, LC-VLP can achieve reductions of both over 30% in average position and rotation errors. Experiments further show that LC-VLP can achieve an average position accuracy of less than 4 cm.
Read the full abstract ▾
All authors Wenxuan Pan, Yang Yang, Dong Wei, Zhiyu Zhu, Jintao Wang, Huan Wu, Yao Nie