Computer Vision
EgoPoseVR: Spatiotemporal Multi-Modal Reasoning for Egocentric Full-Body Pose in Virtual Reality
Haojie Cheng, Shaun Jing Heng Ong, Shaoyu Cai et al. (+3 more)
Immersive virtual reality (VR) applications demand accurate, temporally coherent full-body pose tracking. Recent head-mounted camera-based approaches show promise in
egocentric pose estimation, but encounter challenges when applied to VR head-mounted displays (HMDs), including temporal instability, inaccurate lower-body estimation, and the lack of real-time performance. To address these limitations, we present EgoPoseVR, an end-to-end framework for accurate
egocentric full-body pose estimation in VR that integrates headset motion cues with
egocentric RGB-D observations through a dual-modality fusion pipeline. A spatiotemporal encoder extracts frame- and joint-level representations, which are fused via cross-
attention to fully exploit complementary motion cues across modalities. A kinematic optimization module then imposes constraints from HMD signals, enhancing the accuracy and stability of pose estimation. To facilitate training and evaluation, we introduce a large-scale synthetic
dataset of over 1.8 million temporally aligned HMD and RGB-D frames across diverse VR scenarios. Experimental results show that EgoPoseVR outperforms state-of-the-art
egocentric pose estimation models. A user study in real-world scenes further shows that EgoPoseVR achieved significantly higher subjective ratings in accuracy, stability, embodiment, and intention for future use compared to baseline methods. These results show that EgoPoseVR enables robust full-body pose tracking, offering a practical solution for accurate VR embodiment without requiring additional body-worn sensors or room-scale tracking systems.
Read the full abstract ▾
All authors Haojie Cheng, Shaun Jing Heng Ong, Shaoyu Cai, Aiden Tat Yang Koh, Fuxi Ouyang, Eng Tat Khoo
Computer Vision
A Mixed Reality System for Robust Manikin Localization in Childbirth Training
Haojie Cheng, Chang Liu, Abhiram Kanneganti et al. (+3 more)
Opportunities for medical students to gain practical experience in vaginal births are increasingly constrained by shortened clinical rotations, patient reluctance, and the unpredictable nature of labour. To alleviate clinicians' instructional burden and enhance trainees' learning efficiency, we introduce a mixed reality (MR) system for childbirth training that combines virtual guidance with tactile manikin interaction, thereby preserving authentic haptic feedback while enabling independent practice without continuous on-site expert supervision. The system extends the passthrough capability of commercial head-mounted displays (HMDs) by spatially calibrating an external RGB-D camera, allowing real-time visual integration of physical training objects. Building on this capability, we implement a coarse-to-fine localization pipeline that first aligns the maternal manikin with fiducial markers to define a delivery region and then registers the pre-scanned neonatal head within this area. This process enables spatially accurate overlay of virtual guiding hands near the manikin, allowing trainees to follow expert trajectories reinforced by haptic interaction. Experimental evaluations demonstrate that the system achieves accurate and stable manikin localization on a standalone headset, ensuring practical deployment without external computing resources. A large-scale user study involving 83 fourth-year medical students was subsequently conducted to compare MR-based and virtual reality (VR)-based childbirth training. Four senior obstetricians independently assessed performance using standardized criteria. Results showed that MR training achieved significantly higher scores in delivery, post-delivery, and overall task performance, and was consistently preferred by trainees over VR training.
Read the full abstract ▾
All authors Haojie Cheng, Chang Liu, Abhiram Kanneganti, Mahesh Arjandas Choolani, Arundhati Tushar Gosavi, Eng Tat Khoo
Computer Vision
The Geometric Observability Index: Influence, Fisher Information, and Weak Observability in SE(3) Pose Estimation
Joe-Mei Feng, Hsin-Hsiung Kao and Sheng-Wei Yu
We introduce the
Geometric Observability Index (GOI), a per-feature sensitivity measure for pose estimation on SE(3): the metric norm of the pose perturbation that a single measurement induces through the (possibly rank-deficient) Gauss-Newton
curvature, restricted to the observable subspace. We prove that GOI equals the norm of the M-
estimator influence function, that the underlying
curvature operator coincides with the Fisher information, and that its smallest observable
eigenvalue governs both the worst-case amplification of a measurement's effect and a finite-sample stability radius O(sigma/sqrt(n*lambda_min)). Operationally the theory cuts both ways. GOI is the exact per-measurement attribution, predicting the true leave-one-out pose shift with log-correlation r = 1.00; yet the influence standardized by its inlier null covariance collapses exactly to the classical chi-square residual statistic. Residual gating is thus the leverage-corrected influence test – a first-principles explanation of its robustness – while raw-influence gating conflates a measurement's information with its harm and is predicted to over-reject high-leverage inliers in weakly observable geometry. Controlled synthetic experiments validate every quantitative claim, and studies on five TUM RGB-D sequences (four dynamic, one static control) and two KITTI odometry sequences confirm the prediction: parity of the two criteria under well-conditioned geometry, and significant degradation of raw-influence gating at cond(H) of order 10^4. All code is released for reproducibility.
Read the full abstract ▾
Computer Vision
LoGoSeg: Integrating Local and Global Features for Open-Vocabulary Semantic Segmentation
Junyang Chen, Xiangbo Lv, Zhiqiang Kou et al. (+3 more)
Open-vocabulary semantic
segmentation (OVSS) extends traditional closed-set
segmentation by enabling pixel-wise annotation for both seen and unseen categories using arbitrary textual descriptions. While existing methods leverage
vision-
language models (VLMs) like CLIP, their reliance on image-level pretraining often results in imprecise spatial
alignment, leading to mismatched
segmentations in ambiguous or cluttered scenes. However, most existing approaches lack strong object priors and region-level constraints, which can lead to object
hallucination or missed detections, further degrading performance. To address these challenges, we propose LoGoSeg, an efficient single-stage framework that integrates three key innovations: (i) an object existence prior that dynamically weights relevant categories through global image-text similarity, effectively reducing
hallucinations; (ii) a region-aware
alignment module that establishes precise region-level visual-textual correspondences; and (iii) a dual-stream fusion mechanism that optimally combines local structural information with global semantic context. Unlike prior works, LoGoSeg eliminates the need for external mask proposals, additional backbones, or extra
datasets, ensuring efficiency. Extensive experiments on six
benchmarks (A-847, PC-459, A-150, PC-59, PAS-20, and PAS-20b) demonstrate its competitive performance and strong generalization in open-vocabulary settings.
Read the full abstract ▾
All authors Junyang Chen, Xiangbo Lv, Zhiqiang Kou, Xingdong Sheng, Ning Xu, Yiguo Qiao
Computer Vision
LocateEdit-Bench: A Benchmark for Instruction-Based Editing Localization
Shiyu Wu, Shuyan Li, Jing Li et al. (+2 more)
Recent advancements in image editing have enabled highly controllable and semantically-aware alteration of visual content, posing unprecedented challenges to manipulation localization. However, existing AI-generated forgery localization methods primarily focus on inpainting-based manipulations, making them ineffective against the latest instruction-based editing paradigms. To bridge this critical gap, we propose LocateEdit-Bench, a large-scale
dataset comprising $231$K edited images, designed specifically to
benchmark localization methods against instruction-driven image editing. Our
dataset incorporates four cutting-edge editing models and covers three common edit types. We conduct a detailed analysis of the
dataset and develop two multi-metric evaluation protocols to assess existing localization methods. Our work establishes a foundation to keep pace with the evolving landscape of image editing, thereby facilitating the development of effective methods for future forgery localization.
Dataset will be open-sourced upon acceptance.
Read the full abstract ▾
All authors Shiyu Wu, Shuyan Li, Jing Li, Jing Liu, Yequan Wang
Computer Vision
A Hybrid CNN and ML Framework for Multi-modal Classification of Movement Disorders Using MRI and Brain Structural Features
Mengyu Li, Ingibjörg Kristjánsdóttir, Thilo van Eimeren et al. (+3 more)
Atypical Parkinsonian Disorders (APD), also known as Parkinson-plus syndrome, are a group of neurodegenerative diseases that include progressive supranuclear palsy (PSP) and multiple system atrophy (MSA). In the early stages, overlapping clinical features often lead to misdiagnosis as Parkinson's disease (PD). Identifying reliable imaging biomarkers for early differential diagnosis remains a critical challenge. In this study, we propose a hybrid framework combining convolutional neural networks (CNNs) with machine learning (ML) techniques to classify APD subtypes versus PD and distinguish between the subtypes themselves: PSP vs. PD, MSA vs. PD, and PSP vs. MSA. The model leverages multi-modal input data, including T1-weighted
magnetic resonance imaging (MRI),
segmentation masks of 12 deep
brain structures associated with APD, and their corresponding volumetric measurements. By integrating these complementary modalities, including image data, structural
segmentation masks, and quantitative volume features, the hybrid approach achieved promising classification performance with area under the curve (AUC) scores of 0.95 for PSP vs. PD, 0.86 for MSA vs. PD, and 0.92 for PSP vs. MSA. These results highlight the potential of combining spatial and structural information for robust subtype differentiation. In conclusion, this study demonstrates that fusing CNN-based image features with volume-based ML inputs improves classification accuracy for APD subtypes. The proposed approach may contribute to more reliable early-stage diagnosis, facilitating timely and targeted interventions in clinical practice.
Read the full abstract ▾
All authors Mengyu Li, Ingibjörg Kristjánsdóttir, Thilo van Eimeren, Kathrin Giehl, Lotta M. Ellingsen, the ASAP Neuroimaging Initiative
Computer Vision
Visual Implicit Geometry Transformer for Autonomous Driving
Arsenii Shirokov, Mikhail Kuznetsov, Danila Stepochkin et al. (+5 more)
We introduce the Visual Implicit Geometry
Transformer (ViGT), an autonomous driving
geometric model that estimates continuous 3D occupancy fields from surround-view camera rigs. ViGT represents a step towards foundational
geometric models for autonomous driving, prioritizing scalability, architectural simplicity, and generalization across diverse sensor configurations. Our approach achieves this through a calibration-free architecture, enabling a single model to adapt to different sensor setups. Unlike general-purpose
geometric foundational models that focus on pixel-aligned predictions, ViGT estimates a continuous 3D occupancy field in a birds-eye-view (BEV) addressing domain-specific requirements. ViGT naturally infers geometry from multiple camera views into a single metric coordinate frame, providing a common representation for multiple
geometric tasks. Unlike most existing occupancy models, we adopt a self-supervised training procedure that leverages synchronized image-
LiDAR pairs, eliminating the need for costly manual annotations. We validate the scalability and generalizability of our approach by training our model on a mixture of five large-scale autonomous driving
datasets (NuScenes, Waymo, NuPlan, ONCE, and Argoverse) and achieving state-of-the-art performance on the pointmap estimation task, with the best average rank across all evaluated baselines. We further evaluate ViGT on the Occ3D-nuScenes
benchmark, where ViGT achieves comparable performance with supervised methods. The source code is publicly available at \href{https://github.com/whesense/ViGT}{https://github.com/whesense/ViGT}.
Read the full abstract ▾
All authors Arsenii Shirokov, Mikhail Kuznetsov, Danila Stepochkin, Egor Evdokimov, Daniil Glazkov, Nikolay Patakin, Anton Konushin, Dmitry Senushkin
Computer Vision
ShapeGaussian: High-Fidelity 4D Human Reconstruction in Monocular Videos via Vision Priors
Zhenxiao Liang, Ning Zhang, Youbao Tang et al. (+4 more)
We introduce ShapeGaussian, a high-fidelity, template-free method for 4D human reconstruction from casual
monocular videos. Generic reconstruction methods lacking robust
vision priors, such as 4DGS, struggle to capture high-deformation human motion without multi-view cues. While template-based approaches, primarily relying on SMPL, such as HUGS, can produce photorealistic results, they are highly susceptible to errors in human pose estimation, often leading to unrealistic artifacts. In contrast, ShapeGaussian effectively integrates template-free
vision priors to achieve both high-fidelity and robust scene reconstructions. Our method follows a two-step pipeline: first, we learn a coarse, deformable geometry using pretrained models that estimate data-driven priors, providing a foundation for reconstruction. Then, we refine this geometry using a neural deformation model to capture fine-grained dynamic details. By leveraging 2D
vision priors, we mitigate artifacts from erroneous pose estimation in template-based methods and employ multiple reference frames to resolve the invisibility issue of 2D keypoints in a template-free manner. Extensive experiments demonstrate that ShapeGaussian surpasses template-based methods in reconstruction accuracy, achieving superior visual quality and robustness across diverse human motions in casual
monocular videos.
Read the full abstract ▾
All authors Zhenxiao Liang, Ning Zhang, Youbao Tang, Ruei-Sung Lin, Qixing Huang, Peng Chang, Jing Xiao
Computer Vision
PIRATR: Parametric Object Inference for Robotic Applications with Transformers in 3D Point Clouds
Michael Schwingshackl, Fabio F. Oberweger, Mario Niedermeyer et al. (+2 more)
We present PIRATR, an end-to-end 3D object detection framework for robotic use cases in point clouds. Extending PI3DETR, our method streamlines parametric 3D object detection by jointly estimating multi-class 6-DoF poses and class-specific parametric attributes directly from occlusion-affected point cloud data. This formulation enables not only
geometric localization but also the estimation of task-relevant properties for parametric objects, such as a gripper's opening, where the 3D model is adjusted according to simple, predefined rules. The architecture employs modular, class-specific heads, making it straightforward to extend to novel object types without re-designing the pipeline. We validate PIRATR on an automated forklift platform, focusing on three structurally and functionally diverse categories: crane grippers, loading platforms, and pallets. Trained entirely in a synthetic environment, PIRATR generalizes effectively to real outdoor
LiDAR scans, achieving a detection mAP of 0.919 without additional fine-tuning. PIRATR establishes a new paradigm of pose-aware, parameterized perception. This bridges the gap between low-level
geometric reasoning and actionable world models, paving the way for scalable, simulation-trained perception systems that can be deployed in dynamic robotic environments. Code available at https://github.com/swingaxe/piratr.
Read the full abstract ▾
All authors Michael Schwingshackl, Fabio F. Oberweger, Mario Niedermeyer, Huemer Johannes, Markus Murschitz
Computer Vision
IndustryShapes: An RGB-D Benchmark dataset for 6D object pose estimation of industrial assembly components and tools
Panagiotis Sapoutzoglou, Orestis Vaggelis, Athina Zacharia et al. (+2 more)
We introduce IndustryShapes, a new RGB-D
benchmark dataset of industrial tools and components, designed for both instance-level and novel object 6D pose estimation approaches. The
dataset provides a realistic and application-relevant testbed for benchmarking these methods in the context of industrial robotics bridging the gap between lab-based research and deployment in real-world manufacturing scenarios. Unlike many previous
datasets that focus on household or consumer products or use synthetic, clean tabletop
datasets, or objects captured solely in controlled lab environments, IndustryShapes introduces five new object types with challenging properties, also captured in realistic industrial assembly settings. The
dataset has diverse complexity, from simple to more challenging scenes, with single and multiple objects, including scenes with multiple instances of the same object and it is organized in two parts: the classic set and the extended set. The classic set includes a total of 4,6k images and 6k annotated poses. The extended set introduces additional data modalities to support the evaluation of model-free and sequence-based approaches. To the best of our knowledge, IndustryShapes is the first
dataset to offer RGB-D static onboarding sequences. We further evaluate the
dataset on a representative set of state-of-the art methods for instance-based and novel object 6D pose estimation, including also object detection,
segmentation, showing that there is room for improvement in this domain. The
dataset page can be found in https://pose-lab.github.io/IndustryShapes.
Read the full abstract ▾
All authors Panagiotis Sapoutzoglou, Orestis Vaggelis, Athina Zacharia, Evangelos Sartinas, Maria Pateraki
Computer Vision · Cross-listed from cs.RO
VLN-Pilot: Large Vision-Language Model as an Autonomous Indoor Drone Operator
Bessie Dominguez-Dager, Sergio Suescun-Ferrandiz, Felix Escalona et al. (+2 more)
This paper introduces VLN-Pilot, a novel framework in which a large
Vision-and-
Language Model (VLLM) assumes the role of a human pilot for indoor drone
navigation. By leveraging the
multimodal reasoning abilities of VLLMs, VLN-Pilot interprets free-form natural
language instructions and grounds them in visual observations to plan and execute drone trajectories in GPS-denied indoor environments. Unlike traditional rule-based or
geometric path-planning approaches, our framework integrates
language-driven semantic understanding with visual perception, enabling context-aware, high-level flight behaviors with minimal task-specific engineering. VLN-Pilot supports fully autonomous instruction-following for drones by
reasoning about spatial relationships, obstacle avoidance, and dynamic reactivity to unforeseen events. We validate our framework on a custom photorealistic indoor simulation
benchmark and demonstrate the ability of the VLLM-driven
agent to achieve high success rates on complex instruction-following tasks, including long-horizon
navigation with multiple semantic targets. Experimental results highlight the promise of replacing remote drone pilots with a
language-guided autonomous
agent, opening avenues for scalable, human-friendly control of indoor
UAVs in tasks such as inspection, search-and-rescue, and facility monitoring. Our results suggest that VLLM-based pilots may dramatically reduce operator workload while improving safety and mission flexibility in constrained indoor environments.
Read the full abstract ▾
All authors Bessie Dominguez-Dager, Sergio Suescun-Ferrandiz, Felix Escalona, Francisco Gomez-Donoso, Miguel Cazorla
Computer Vision
FastVMT: Eliminating Redundancy in Video Motion Transfer
Yue Ma, Zhikai Wang, Tianhao Ren et al. (+9 more)
Video motion transfer aims to synthesize
videos by generating visual content according to a text
prompt while transferring the motion pattern observed in a reference
video. Recent methods predominantly use the
Diffusion Transformer (DiT) architecture. To achieve satisfactory runtime, several methods attempt to accelerate the computations in the DiT, but fail to address structural sources of inefficiency. In this work, we identify and remove two types of computational redundancy in earlier work: motion redundancy arises because the generic DiT architecture does not reflect the fact that frame-to-frame motion is small and smooth; gradient redundancy occurs if one ignores that gradients change slowly along the
diffusion trajectory. To mitigate motion redundancy, we mask the corresponding
attention layers to a local neighborhood such that interaction weights are not computed unnecessarily distant image regions. To exploit gradient redundancy, we design an optimization scheme that reuses gradients from previous
diffusion steps and skips unwarranted gradient computations. On average, FastVMT achieves a 3.43x speedup without degrading the visual fidelity or the temporal consistency of the generated
videos.
Read the full abstract ▾
All authors Yue Ma, Zhikai Wang, Tianhao Ren, Mingzhe Zheng, Hongyu Liu, Jiayi Guo, Kunyu Feng, Yuxuan Xue, Zixiang Zhao, Konrad Schindler, Qifeng Chen, Linfeng Zhang
Computer Vision
A Comparative Study of 3D Person Detection: Sensor Modalities and Robustness in Diverse Indoor and Outdoor Environments
Malaz Tamim, Andrea Matic-Flierl and Karsten Roscher
Accurate 3D person detection is critical for safety in applications such as robotics, industrial monitoring, and surveillance. This work presents a systematic evaluation of 3D person detection using camera-only,
LiDAR-only, and camera-
LiDAR fusion. While most existing research focuses on autonomous driving, we explore detection performance and robustness in diverse indoor and outdoor scenes using the JRDB
dataset. We compare three representative models - BEVDepth (camera), PointPillars (
LiDAR), and DAL (camera-
LiDAR fusion) - and analyze their behavior under varying occlusion and distance levels. Our results show that the fusion-based approach consistently outperforms single-modality models, particularly in challenging scenarios. We further investigate robustness against sensor corruptions and misalignments, revealing that while DAL offers improved resilience, it remains sensitive to sensor misalignment and certain
LiDAR-based corruptions. In contrast, the camera-based BEVDepth model showed the lowest performance and was most affected by occlusion, distance, and noise. Our findings highlight the importance of utilizing sensor fusion for enhanced 3D person detection, while also underscoring the need for ongoing research to address the vulnerabilities inherent in these systems.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging
Yayuan Li, Ze Peng, Jian Zhang et al. (+3 more)
Model merging combines multiple fine-tuned models into a single model by adding their weight updates, providing a lightweight alternative to retraining. Existing methods primarily target resolving conflicts between task updates, leaving the failure mode of over-counting shared knowledge unaddressed. We show that when tasks share aligned spectral directions (i.e., overlapping singular vectors), a simple linear combination repeatedly accumulates these directions, inflating the singular values and biasing the merged model toward shared subspaces. To mitigate this issue, we propose Singular Value Calibration (SVC), a training-free and data-free post-processing method that quantifies subspace overlap and rescales inflated singular values to restore a balanced spectrum. Across
vision and
language benchmarks, SVC consistently improves strong merging baselines and achieves state-of-the-art performance. Furthermore, by modifying only the singular values, SVC improves the performance of Task Arithmetic by 13.0%. Code is available at https://github.com/lyymuwu/SVC.
Read the full abstract ▾
All authors Yayuan Li, Ze Peng, Jian Zhang, Jintao Guo, Yue Duan, Yinghuan Shi
Computer Vision
SSG: Scaled Spatial Guidance for Multi-Scale Visual Autoregressive Generation
Youngwoo Shin, Jiwan Hur and Junmo Kim
Visual autoregressive (VAR) models generate images through next-scale prediction, naturally achieving coarse-to-fine, fast, high-fidelity synthesis mirroring human perception. In practice, this hierarchy can drift at inference time, as limited capacity and accumulated error cause the model to deviate from its coarse-to-fine nature. We revisit this limitation from an information-theoretic perspective and deduce that ensuring each scale contributes high-frequency content not explained by earlier scales mitigates the train-inference discrepancy. With this insight, we propose Scaled Spatial Guidance (SSG), training-free, inference-time guidance that steers generation toward the intended hierarchy while maintaining global coherence. SSG emphasizes target high-frequency signals, defined as the semantic residual, isolated from a coarser prior. To obtain this prior, we leverage a principled frequency-domain procedure, Discrete Spatial Enhancement (DSE), which is devised to sharpen and better isolate the semantic residual through frequency-aware construction. SSG applies broadly across VAR models leveraging discrete visual
tokens, regardless of tokenization design or conditioning modality. Experiments demonstrate SSG yields consistent gains in fidelity and diversity while preserving low latency, revealing untapped efficiency in coarse-to-fine image generation. Code is available at https://github.com/Youngwoo-git/SSG.
Read the full abstract ▾
Computer Vision
Generalization of Self-Supervised Vision Transformers for Protein Localization Across Microscopy Domains
Ben Isselmann, Dilara Göksu and Andreas Weinmann
Task-specific microscopy
datasets are often too small to train deep learning models that learn robust feature representations. Self-supervised learning (SSL) can mitigate this by pretraining on large unlabeled
datasets, but it remains unclear how well such representations transfer across microscopy domains with different staining protocols and channel configurations. We investigate the cross-domain transferability of DINO-pretrained
Vision Transformers for
protein localization on the OpenCell
dataset. We generate image
embeddings using three DINO backbones pretrained on ImageNet-1k, the Human
Protein Atlas (HPA), and OpenCell, and evaluate them by training a supervised classification head on OpenCell labels. All pretrained models transfer well, with the microscopy-specific HPA-pretrained model achieving the best performance (mean macro $F_1$-score = 0.8221 $\pm$ 0.0062), slightly outperforming a DINO model trained directly on OpenCell (0.8057 $\pm$ 0.0090). These results highlight the value of large-scale pretraining and indicate that domain-relevant SSL representations can generalize effectively to related but distinct microscopy
datasets, enabling strong downstream performance even when task-specific labeled data are limited.
Read the full abstract ▾
Computer Vision
Structure-First Point Cloud Learning with Mapper Region Graphs
Jeongbin You, Donggun Kim, Sejun Park et al. (+1 more)
Robust 3D point cloud classification is often pursued by scaling up backbones or relying on specialized data augmentation. We instead ask whether structural abstraction alone can improve robustness, and study a simple
topology-inspired decomposition based on the Mapper
algorithm. We propose Mapper-GIN, a lightweight pipeline that partitions a point cloud into overlapping regions using Mapper (PCA lens, cubical cover, and followed by density-based clustering), constructs a region graph from their overlaps, and performs graph classification with a Graph Isomorphism Network. On the corruption
benchmark ModelNet40-C, Mapper-GIN achieves competitive and stable accuracy under Noise and Transformation corruptions with only 0.5M parameters. In contrast to prior approaches that require heavier architectures or additional mechanisms to gain robustness, Mapper-GIN attains strong corruption robustness through simple region-level graph abstraction and GIN message passing. Overall, our results suggest that region-graph structure offers an efficient and interpretable source of robustness for 3D visual recognition.
Read the full abstract ▾
All authors Jeongbin You, Donggun Kim, Sejun Park, Seungsang Oh
Computer Vision
VGGT-Motion: Motion-Aware Calibration-Free Monocular SLAM for Long-Range Consistency
Zhuang Xiong, Chen Zhang, Qingshan Xu et al. (+1 more)
Despite recent progress in calibration-free
monocular SLAM via 3D
vision foundation models, scale drift remains severe on long sequences. Motion-agnostic partitioning breaks contextual coherence and causes zero-motion drift, while conventional
geometric alignment is computationally expensive. To address these issues, we propose VGGT-Motion, a calibration-free SLAM system for efficient and robust global consistency over kilometer-scale trajectories. Specifically, we first propose a motion-aware submap construction mechanism that uses optical flow to guide adaptive partitioning, prune static redundancy, and encapsulate turns for stable local geometry. We then design an anchor-driven direct Sim(3) registration strategy. By exploiting context-balanced anchors, it achieves search-free, pixel-wise dense
alignment and efficient loop closure without costly feature matching. Finally, a lightweight submap-level pose graph optimization enforces global consistency with linear complexity, enabling scalable long-range operation. Experiments show that VGGT-Motion markedly improves trajectory accuracy and efficiency, achieving state-of-the-art performance in zero-shot, long-range calibration-free
monocular SLAM.
Read the full abstract ▾
All authors Zhuang Xiong, Chen Zhang, Qingshan Xu, Wenbing Tao
Computer Vision · Cross-listed from cs.MM
XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoning
Hanwen Zhang, Yao Liu, Peiyuan Jiang et al. (+6 more)
Explainable
Multimodal Emotion Recognition plays a crucial role in applications such as human-computer interaction and social media analytics. However, current approaches struggle with cue-level perception and
reasoning due to two main challenges: 1) general-purpose modality encoders are pretrained to capture global structures and general semantics rather than fine-grained emotional cues, resulting in limited sensitivity to emotional signals; and 2) available
datasets usually involve a trade-off between annotation quality and scale, which leads to insufficient supervision for emotional cues and ultimately limits cue-level
reasoning. Moreover, existing evaluation metrics are inadequate for assessing cue-level
reasoning performance. To address these challenges, we propose eXplainable
Emotion GPT (XEmoGPT), a novel EMER framework capable of both perceiving and
reasoning over emotional cues. It incorporates two specialized modules: the
Video Emotional Cue Bridge (VECB) and the Audio Emotional Cue Bridge (AECB), which enhance the
video and audio encoders through carefully designed tasks for fine-grained emotional cue perception. To further support cue-level
reasoning, we construct a large-scale
dataset, EmoCue, designed to teach XEmoGPT how to reason over
multimodal emotional cues. In addition, we introduce EmoCue-360, an automated metric that extracts and matches emotional cues using semantic similarity, and release EmoCue-Eval, a
benchmark of 400 expert-annotated samples covering diverse emotional scenarios. Experimental results show that XEmoGPT achieves strong performance in both emotional cue perception and
reasoning.
Read the full abstract ▾
All authors Hanwen Zhang, Yao Liu, Peiyuan Jiang, Lang Junjie, Xie Jun, Yihui He, Yajiao Deng, Siyu Du, Qiao Liu
Computer Vision
Feature points evaluation on omnidirectional vision with a photorealistic fisheye sequence – A report on experiments done in 2014
Julien Moreau, S. Ambellouis and Yassine Ruichek
What is this report: This is a scientific report, contributing with a detailed bibliography, a
dataset which we will call now PFSeq for ”Photorealistic Fisheye Sequence” and make available at https://doi.org/10. 57745/DYIVVU, and comprehensive experiments. This work should be considered as a draft, and has been done during my PhD thesis ”Construction of 3D models from fisheye
video data-Application to the localisation in urban area” in 2014 [Mor16]. These results have never been published. The aim was to find the best features detector and descriptor for fisheye images, in the context of selfcalibration, with cameras mounted on the top of a car and aiming at the zenith (to proceed then fisheye visual odometry and stereovision in urban scenes). We face a chicken and egg problem, because we can not take advantage of an accurate projection model for an optimal features detection and description, and we rightly need good features to perform the calibration (i.e. to compute the accurate projection model of the camera). What is not this report: It does not contribute with new features
algorithm. It does not compare standard features
algorithms to
algorithms designed for omnidirectional images (unfortunately). It has not been peer-reviewed. Discussions have been translated and enhanced but the experiments have not been run again and the report has not been updated accordingly to the
evolution of the state-of-the-art (read this as a 2014 report).
Read the full abstract ▾
Computer Vision
SOMA-1M: A Large-Scale SAR-Optical Multi-resolution Alignment Dataset for Multi-Task Remote Sensing
Peihao Wu, Yongxiang Yao, Yi Wan et al. (+4 more)
Synthetic Aperture Radar (SAR) and optical imagery provide complementary strengths that constitute the critical foundation for transcending single-modality constraints and facilitating cross-modal collaborative processing and intelligent interpretation. However, existing
benchmark datasets often suffer from limitations such as single spatial resolution, insufficient data scale, and low
alignment accuracy, making them inadequate for supporting the training and generalization of multi-scale foundation models. To address these challenges, we introduce SOMA-1M (SAR-Optical Multi-resolution
Alignment), a pixel-level precisely aligned
dataset containing over 1.3 million pairs of georeferenced images with a specification of 512 x 512 pixels. This
dataset integrates imagery from Sentinel-1, PIESAT-1, Capella Space, and Google Earth, achieving global multi-scale coverage from 0.5 m to 10 m. It encompasses 12 typical land cover categories, effectively ensuring scene diversity and complexity. To address
multimodal projection deformation and massive data registration, we designed a rigorous coarse-to-fine image matching framework ensuring pixel-level
alignment. Based on this
dataset, we established comprehensive evaluation
benchmarks for four hierarchical
vision tasks, including image matching, image fusion, SAR-assisted cloud removal, and cross-modal translation, involving over 30 mainstream
algorithms. Experimental results demonstrate that supervised training on SOMA-1M significantly enhances performance across all tasks. Notably,
multimodal remote sensing image (MRSI) matching performance achieves current state-of-the-art (SOTA) levels. SOMA-1M serves as a foundational resource for robust
multimodal algorithms and remote sensing foundation models. The
dataset will be released publicly at: https://github.com/PeihaoWu/SOMA-1M.
Read the full abstract ▾
All authors Peihao Wu, Yongxiang Yao, Yi Wan, Wenfei Zhang, Ruipeng Zhao, Jiayuan Li, Yongjun Zhang
Computer Vision
MerNav: A Highly Generalizable Memory-Execute-Review Framework for Zero-Shot Object Goal Navigation
Dekang Qi, Shuang Zeng, Xinyuan Chang et al. (+4 more)
Visual
Language Navigation (VLN) is one of the fundamental capabilities for
embodied intelligence and a critical challenge that urgently needs to be addressed. However, existing methods are still unsatisfactory in terms of both success rate (SR) and generalization: Supervised Fine-Tuning (SFT) approaches typically achieve higher SR, while Training-Free (TF) approaches often generalize better, but it is difficult to obtain both simultaneously. To this end, we propose a
Memory-Execute-Review framework. It consists of three parts: a hierarchical
memory module for providing information support, an execute module for routine decision-making and actions, and a review module for handling abnormal situations and correcting behavior. We validated the effectiveness of this framework on the Object Goal
Navigation task. Across 4
datasets, our average SR achieved absolute improvements of 7% and 5% compared to all baseline methods under TF and Zero-Shot (ZS) settings, respectively. On the most commonly used HM3D_v0.1 and the more challenging open vocabulary
dataset HM3D_OVON, the SR improved by 8% and 6%, under ZS settings. Furthermore, on the MP3D and HM3D_OVON
datasets, our method not only outperformed all TF methods but also surpassed all SFT methods, achieving comprehensive leadership in both SR (5% and 2%) and generalization. Additionally, we deployed the MerNav model on the
humanoid robot and conducted experiments in the real world. The project address is: https://qidekang.github.io/MerNav.github.io/
Read the full abstract ▾
All authors Dekang Qi, Shuang Zeng, Xinyuan Chang, Feng Xiong, Shichao Xie, Xiaolong Wu, Mu Xu
Computer Vision · Cross-listed from cs.AI
Semantic Purification for Conditional Representation Learning
Jiaquan Wang, Yan Lyu, Chen Li et al. (+1 more)
Conditional representation learning aims to extract criterion-specific features for customized tasks. Recent methods construct conditional subspaces spanned by criterion-specific text bases in the
embedding space of
vision-
language models (VLMs). Image
embeddings are then projected onto these subspaces to obtain conditional representations. However, since VLMs are not explicitly trained to disentangle semantics associated with different criteria, the corresponding conditional subspaces remain coupled. This coupling induces semantic leakage during projection, thereby degrading the semantic purity of conditional representations. To suppress semantic leakage, we propose Semantic Purification for Conditional Representation Learning (SP-CRL). Specifically, SP-CRL first decomposes the original text basis and performs
curvature-based adaptive truncation on the resulting basis vectors to construct a purer conditional subspace. It then identifies an appropriate noise subspace and projects image
embeddings onto its null space to remove irrelevant semantic components. Extensive experiments across customized clustering, customized few-shot classification, and customized
retrieval tasks demonstrate that SP-CRL achieves state-of-the-art performance with superior generalization.
Read the full abstract ▾
All authors Jiaquan Wang, Yan Lyu, Chen Li, Yuheng Jia
Computer Vision
Yue Lu, Xiangyu Zhou, Shizhou Zhang et al. (+3 more)
Continual learning (CL) empowers AI systems to progressively acquire knowledge from non-stationary data streams. However, catastrophic forgetting remains a critical challenge. In this work, we identify
attention drift in
Vision Transformers as a primary source of catastrophic forgetting, where the
attention to previously learned visual concepts shifts significantly after learning new tasks. Inspired by neuroscientific insights into the selective
attention in the human visual system, we propose a novel
attention-retaining framework to mitigate forgetting in CL. Our method constrains
attention drift by explicitly modifying gradients during backpropagation through a two-step process: 1) extracting
attention maps of the previous task using a layer-wise rollout mechanism and generating instance-adaptive binary masks, and 2) when learning a new task, applying these masks to zero out gradients associated with previous
attention regions, thereby preventing disruption of learned visual concepts. For compatibility with modern optimizers, the gradient masking process is further enhanced by scaling parameter updates proportionally to maintain their relative magnitudes. Experiments and visualizations demonstrate the effectiveness of our method in mitigating catastrophic forgetting and preserving visual concepts. It achieves state-of-the-art performance and exhibits robust generalizability across diverse CL scenarios.
Read the full abstract ▾
All authors Yue Lu, Xiangyu Zhou, Shizhou Zhang, Yinghui Xing, Guoqiang Liang, Wencong Zhang
Computer Vision · Cross-listed from eess.IV
Towards Segmenting the Invisible: An End-to-End Registration and Segmentation Framework for Weakly Supervised Tumour Analysis
Budhaditya Mukhopadhyay, Chirag Mandal, Pavan Tummala et al. (+3 more)
Liver tumour ablation presents a significant clinical challenge: whilst tumours are clearly visible on pre-operative MRI, they are often effectively invisible on intra-operative CT due to minimal contrast between pathological and healthy tissue. This work investigates the feasibility of cross-modality weak supervision for scenarios where pathology is visible in one modality (MRI) but absent in another (CT). We present a hybrid registration-
segmentation framework that combines MSCGUNet for inter-modal image registration with a UNet-based
segmentation module, enabling registration-assisted pseudo-label generation for CT images. Our evaluation on the CHAOS
dataset demonstrates that the pipeline can successfully register and segment healthy liver anatomy, achieving a Dice score of 0.72. However, when applied to clinical data containing tumours, performance degrades substantially (Dice score of 0.16), revealing the fundamental limitations of current registration methods when the target pathology lacks corresponding visual features in the target modality. We analyse the "domain gap" and "feature absence" problems, demonstrating that whilst spatial propagation of labels via registration is feasible for visible structures, segmenting truly invisible pathology remains an open challenge. Our findings highlight that registration-based label transfer cannot compensate for the absence of discriminative features in the target modality, providing important insights for future research in cross-modality medical image analysis. Code an weights are available at: https://github.com/BudhaTronix/Weakly-Supervised-Tumour-Detection
Read the full abstract ▾
All authors Budhaditya Mukhopadhyay, Chirag Mandal, Pavan Tummala, Naghmeh Mahmoodian, Andreas Nürnberger, Soumick Chatterjee
Computer Vision
Chang Zou, Changlin Li, Yang Li et al. (+7 more)
While
diffusion models have achieved great success in the field of
video generation, this progress is accompanied by a rapidly escalating computational burden. Among the existing acceleration methods, Feature Caching is popular due to its training-free property and considerable speedup performance, but it inevitably faces semantic and detail drop with further compression. Another widely adopted method, training-aware step-
distillation, though successful in image generation, also faces drastic degradation in
video generation with a few steps. Furthermore, the quality loss becomes more severe when simply applying training-free feature caching to the step-distilled models, due to the sparser sampling steps. This paper novelly introduces a
distillation-compatible learnable feature caching mechanism for the first time. We employ a lightweight learnable neural predictor instead of traditional training-free heuristics for
diffusion models, enabling a more accurate capture of the high-dimensional feature
evolution process. Furthermore, we explore the challenges of highly compressed
distillation on large-scale
video models and propose a conservative Restricted MeanFlow approach to achieve more stable and lossless
distillation. By undertaking these initiatives, we further push the acceleration boundaries to $11.8\times$ while preserving generation quality. Extensive experiments demonstrate the effectiveness of our method. Code has been made publicly available: https://github.com/Tencent-Hunyuan/DisCa
Read the full abstract ▾
All authors Chang Zou, Changlin Li, Yang Li, Patrol Li, Jianbing Wu, Xiao He, Songtao Liu, Zhao Zhong, Kailin Huang, Linfeng Zhang
Computer Vision
Synthetic Defect Geometries of Cast Metal Objects Modeled via 2d Voronoi Tessellations
Natascha Jeziorski, Petra Gospodnetić and Claudia Redenbach
In industry, defect detection is crucial for quality control. Non-destructive testing (NDT) methods are preferred as they do not influence the functionality of the object while inspecting. Automated data evaluation for automated defect detection is a growing field of research. In particular, machine learning approaches show promising results. To provide training data in sufficient amount and quality, synthetic data can be used. Rule-based approaches enable synthetic data generation in a controllable environment. Therefore, a digital twin of the inspected object including synthetic defects is needed. We present parametric methods to model 3d mesh objects of various defect types that can then be added to the object geometry to obtain synthetic defective objects. The models are motivated by common defects in metal casting but can be transferred to other machining procedures that produce similar defect shapes. Synthetic data resembling the real inspection data can then be created by using a physically based Monte Carlo simulation of the respective testing method. Using our defect models, a variable and arbitrarily large synthetic data set can be generated with the possibility to include rarely occurring defects in sufficient quantity. Pixel-perfect annotation can be created in parallel. As an example, we will use visual surface inspection, but the procedure can be applied in combination with simulations for any other NDT method.
Read the full abstract ▾
Computer Vision
Stable Velocity: A Variance Perspective on Flow Matching
Donglin Yang, Yongxing Zhang, Xin Yu et al. (+5 more)
While flow matching is elegant, its reliance on single-sample conditional velocities leads to high-variance training targets that destabilize optimization and slow convergence. By explicitly characterizing this variance, we identify 1) a high-variance regime near the prior, where optimization is challenging, and 2) a low-variance regime near the data distribution, where conditional and marginal velocities nearly coincide. Leveraging this insight, we propose Stable Velocity, a unified framework that improves both training and sampling. For training, we introduce Stable Velocity Matching (StableVM), an unbiased variance-reduction objective, along with Variance-Aware Representation
Alignment (VA-REPA), which adaptively strengthen auxiliary supervision in the low-variance regime. For inference, we show that dynamics in the low-variance regime admit closed-form simplifications, enabling Stable Velocity Sampling (StableVS), a finetuning-free acceleration. Extensive experiments on ImageNet $256\times256$ and large pretrained text-to-image and text-to-
video models, including SD3.5, Flux, Qwen-Image, and Wan2.2, demonstrate consistent improvements in training efficiency and more than $2\times$ faster sampling within the low-variance regime without degrading sample quality. Our code is available at https://github.com/linYDTHU/StableVelocity.
Read the full abstract ▾
All authors Donglin Yang, Yongxing Zhang, Xin Yu, Liang Hou, Xin Tao, Pengfei Wan, Xiaojuan Qi, Renjie Liao
Computer Vision
LD-SLRO: Latent Diffusion Structured Light for 3-D Reconstruction of Highly Reflective Objects
Sanghoon Jeon, Gihyun Jung, Suhyeon Ka et al. (+1 more)
Fringe projection profilometry-based 3-D reconstruction of objects with high reflectivity and low surface roughness remains a significant challenge. When measuring such glossy surfaces, specular reflection and indirect illumination often lead to severe distortion or loss of the projected fringe patterns. To address these issues, we propose a latent
diffusion-based structured light for reflective objects (LD-SLRO). Phase-shifted fringe images captured from highly reflective surfaces are first encoded to extract latent representations that capture surface reflectance characteristics. These latent features are then used as conditional inputs to a latent
diffusion model, which probabilistically suppresses reflection-induced artifacts and recover lost fringe information, yielding high-quality fringe images. The proposed components, including the specular reflection encoder, time-variant channel affine layer, and
attention modules, further improve fringe restoration quality. In addition, LD-SLRO provides high flexibility in configuring the input and output fringe sets. Experimental results demonstrate that the proposed method improves both fringe quality and 3-D reconstruction accuracy over state-of-the-art methods, reducing the average root-mean-squared error from 1.8176 mm to 0.9619 mm.
Read the full abstract ▾
All authors Sanghoon Jeon, Gihyun Jung, Suhyeon Ka, Jae-Sang Hyun
Computer Vision · Cross-listed from cs.AI
M$^2$-Miner: Multi-Agent Enhanced MCTS for Mobile GUI Agent Data Mining
Rui Lv, Juncheng Mo, Tianyi Chu et al. (+11 more)
Graphical User Interface (GUI)
agent is pivotal to advancing intelligent human-computer interaction paradigms. Constructing powerful GUI
agents necessitates the large-scale annotation of high-quality user-behavior trajectory data (i.e., intent-trajectory pairs) for training. However, manual annotation methods and current GUI
agent data mining approaches typically face three critical challenges: high construction cost, poor data quality, and low data richness. To address these issues, we propose M$^2$-Miner, the first low-cost and automated mobile GUI
agent data-mining framework based on Monte Carlo Tree Search (MCTS). For better data mining efficiency and quality, we present a collaborative multi-
agent framework, comprising InferAgent, OrchestraAgent, and JudgeAgent for guidance, acceleration, and evaluation. To further enhance the efficiency of mining and enrich intent diversity, we design an intent recycling strategy to extract extra valuable interaction trajectories. Additionally, a progressive model-in-the-loop training strategy is introduced to improve the success rate of data mining. Extensive experiments have demonstrated that the GUI
agent fine-tuned using our mined data achieves state-of-the-art performance on several commonly used mobile GUI
benchmarks. Our work will be released to facilitate the community research.
Read the full abstract ▾
All authors Rui Lv, Juncheng Mo, Tianyi Chu, Chen Rao, Hongyi Jing, Jiajie Teng, Jiafu Chen, Shiqi Zhang, Liangzi Ding, Shuo Fang, Huaizhong Lin, Ziqiang Dang, Chenguang Ma, Lei Zhao