Computer Vision
Envision: Embodied Visual Planning via Goal-Imagery Video Diffusion
Yuming Gu, Yizhi Wang, Yining Hong et al. (+9 more)
Embodied visual planning aims to enable manipulation tasks by imagining how a scene evolves toward a desired goal and using the imagined trajectories to guide actions.
Video diffusion models, through their image-to-
video generation capability, provide a promising foundation for such visual imagination. However, existing approaches are largely forward predictive, generating trajectories conditioned on the initial observation without explicit goal modeling, thus often leading to spatial drift and goal misalignment. To address these challenges, we propose Envision, a
diffusion-based framework that performs visual planning for
embodied agents. By explicitly constraining the generation with a goal image, our method enforces physical plausibility and goal consistency throughout the generated trajectory. Specifically, Envision operates in two stages. First, a Goal Imagery Model identifies task-relevant regions, performs region-aware cross
attention between the scene and the instruction, and synthesizes a coherent goal image that captures the desired outcome. Then, an Env-Goal
Video Model, built upon a first-and-last-frame-conditioned
video diffusion model (FL2V), interpolates between the initial observation and the goal image, producing smooth and physically plausible
video trajectories that connect the start and goal states. Experiments on object manipulation and image editing
benchmarks demonstrate that Envision achieves superior goal
alignment, spatial consistency, and object preservation compared to baselines. The resulting visual plans can directly support downstream robotic planning and control, providing reliable guidance for
embodied agents.
Read the full abstract ▾
All authors Yuming Gu, Yizhi Wang, Yining Hong, Yipeng Gao, Hao Jiang, Angtian Wang, Bo Liu, Nathaniel S. Dennler, Zhengfei Kuang, Hao Li, Gordon Wetzstein, Chongyang Ma
Computer Vision
Rethinking Memory Design in SAM-Based Visual Object Tracking
Mohamad Alansari, Muzammal Naseer, Hasan Al Marzouqi et al. (+2 more)
\noindent
Memory has become the central mechanism enabling robust visual object tracking in modern
segmentation-based frameworks. Recent methods built upon Segment Anything Model 2 (SAM2) have demonstrated strong performance by refining how past observations are stored and reused. However, existing approaches address
memory limitations in a method-specific manner, leaving the broader design principles of
memory in SAM-based tracking poorly understood. Moreover, it remains unclear how these
memory mechanisms transfer to stronger, next-generation foundation models such as Segment Anything Model 3 (SAM3). In this work, we present a systematic
memory-centric study of SAM-based visual object tracking. We first analyze representative SAM2-based trackers and show that most methods primarily differ in how short-term
memory frames are selected, while sharing a common object-centric representation. Building on this insight, we faithfully reimplement these
memory mechanisms within the SAM3 framework and conduct large-scale evaluations across ten diverse
benchmarks, enabling a controlled analysis of
memory design independent of backbone strength. Guided by our empirical findings, we propose a unified hybrid
memory framework that explicitly decomposes
memory into short-term appearance
memory and long-term distractor-resolving
memory. This decomposition enables the integration of existing
memory policies in a modular and principled manner. Extensive experiments demonstrate that the proposed framework consistently improves robustness under long-term occlusion, complex motion, and distractor-heavy scenarios on both SAM2 and SAM3 backbones. Code is available at: https://github.com/HamadYA/SAM3_Tracking_Zoo. This is a preprint. Some results are being finalized and may be updated in a future revision.
Read the full abstract ▾
All authors Mohamad Alansari, Muzammal Naseer, Hasan Al Marzouqi, Naoufel Werghi, Sajid Javed
Computer Vision
Jiacheng Ye, Shansan Gong, Jiahui Gao et al. (+6 more)
While autoregressive Large
Vision-
Language Models (VLMs) have achieved remarkable success, their sequential generation often limits their efficacy in complex visual planning and dynamic robotic control. In this work, we investigate the potential of constructing
Vision-
Language Models upon
diffusion-based large
language models (dLLMs) to overcome these limitations. We introduce Dream-VL, an open
diffusion-based VLM (dVLM) that achieves state-of-the-art performance among previous dVLMs. Dream-VL is comparable to top-tier AR-based VLMs trained on open data on various
benchmarks but exhibits superior potential when applied to visual planning tasks. Building upon Dream-VL, we introduce Dream-VLA, a dLLM-based
Vision-
Language-Action model (dVLA) developed through continuous pre-training on open robotic
datasets. We demonstrate that the natively bidirectional nature of this
diffusion backbone serves as a superior foundation for VLA tasks, inherently suited for action chunking and parallel generation, leading to significantly faster convergence in downstream fine-tuning. Dream-VLA achieves top-tier performance of 97.2% average success rate on LIBERO, 71.4% overall average on SimplerEnv-Bridge, and 60.5% overall average on SimplerEnv-Fractal, surpassing leading models such as $\pi_0$ and GR00T-N1. We also validate that dVLMs surpass AR baselines on downstream tasks across different training objectives. We release both Dream-VL and Dream-VLA to facilitate further research in the community.
Read the full abstract ▾
All authors Jiacheng Ye, Shansan Gong, Jiahui Gao, Junming Fan, Shuang Wu, Wei Bi, Haoli Bai, Lifeng Shang, Lingpeng Kong
Computer Vision
Enhancing Noise Resilience in Face Clustering via Sparse Differential Transformer
Dafeng Zhang, Yongqi Song and Shizhuo Liu
The method used to measure relationships between face
embeddings plays a crucial role in determining the performance of face clustering. Existing methods employ the Jaccard similarity coefficient instead of the cosine distance to enhance the measurement accuracy. However, these methods introduce too many irrelevant nodes, producing Jaccard coefficients with limited discriminative power and adversely affecting clustering performance. To address this issue, we propose a prediction-driven Top-K Jaccard similarity coefficient that enhances the purity of neighboring nodes, thereby improving the reliability of similarity measurements. Nevertheless, accurately predicting the optimal number of neighbors (Top-K) remains challenging, leading to suboptimal clustering results. To overcome this limitation, we develop a
Transformer-based prediction model that examines the relationships between the central node and its neighboring nodes near the Top-K to further enhance the reliability of similarity estimation. However, vanilla
Transformer, when applied to predict relationships between nodes, often introduces noise due to their overemphasis on irrelevant feature relationships. To address these challenges, we propose a Sparse Differential
Transformer (SDT), instead of the vanilla
Transformer, to eliminate noise and enhance the model's anti-noise capabilities. Extensive experiments on multiple
datasets, such as MS-Celeb-1M, demonstrate that our approach achieves state-of-the-art (SOTA) performance, outperforming existing methods and providing a more robust solution for face clustering.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
Learning Multi-Modal Mobility Dynamics for Generalized Next Location Recommendation
Junshu Dai, Yu Wang, Tongya Zheng et al. (+6 more)
The precise prediction of human mobility has produced significant socioeconomic impacts, such as location recommendations and evacuation suggestions. However, existing methods suffer from limited generalization capability: unimodal approaches are constrained by data sparsity and inherent biases, while multi-modal methods struggle to effectively capture mobility dynamics caused by the semantic gap between static multi-modal representation and spatial-temporal dynamics. Therefore, we leverage multi-modal spatial-temporal knowledge to characterize mobility dynamics for the location recommendation task, dubbed as Multi-Modal Mobility (M$^3$ob). First, we construct a unified spatial-temporal relational graph (STRG) for multi-modal representation, by leveraging the functional semantics and spatial-temporal knowledge captured by the large
language models (LLMs)-enhanced spatial-temporal knowledge graph (STKG). Second, we design a gating mechanism to fuse spatial-temporal graph representations of different modalities, and propose an STKG-guided cross-modal
alignment to inject spatial-temporal dynamic knowledge into the static image modality. Extensive experiments on six public
datasets show that our proposed method not only achieves consistent improvements in normal scenarios but also exhibits significant generalization ability in abnormal scenarios.
Read the full abstract ▾
All authors Junshu Dai, Yu Wang, Tongya Zheng, Wei Ji, Qinghong Guo, Ji Cao, Jie Song, Canghong Jin, Mingli Song
Computer Vision
PTalker: Personalized Speech-Driven 3D Talking Head Animation via Style Disentanglement and Modality Alignment
Bin Wang, Yang Xu, Huan Zhao et al. (+2 more)
Speech-driven 3D talking head generation aims to produce lifelike facial animations precisely synchronized with
speech. While considerable progress has been made in achieving high lip-synchronization accuracy, existing methods largely overlook the intricate nuances of individual speaking styles, which limits personalization and realism. In this work, we present a novel framework for personalized 3D talking head animation, namely "PTalker". This framework preserves speaking style through style disentanglement from audio and facial motion sequences and enhances lip-synchronization accuracy through a three-level
alignment mechanism between audio and mesh modalities. Specifically, to effectively disentangle style and content, we design disentanglement constraints that encode driven audio and motion sequences into distinct style and content spaces to enhance speaking style representation. To improve lip-synchronization accuracy, we adopt a modality
alignment mechanism incorporating three aspects: spatial
alignment using Graph
Attention Networks to capture vertex connectivity in the 3D mesh structure, temporal
alignment using cross-
attention to capture and synchronize temporal dependencies, and feature
alignment by top-k bidirectional contrastive losses and KL divergence constraints to ensure consistency between
speech and mesh modalities. Extensive qualitative and quantitative experiments on public
datasets demonstrate that PTalker effectively generates realistic, stylized 3D talking heads that accurately match identity-specific speaking styles, outperforming state-of-the-art methods. The source code and supplementary
videos are available at: PTalker.
Read the full abstract ▾
All authors Bin Wang, Yang Xu, Huan Zhao, Hao Zhang, Zixing Zhang
Computer Vision
KV-Tracker: Real-Time Pose Tracking with Transformers
Marwan Taher, Ignacio Alzugaray, Kirill Mazur et al. (+2 more)
Multi-view 3D geometry networks offer a powerful prior but are prohibitively slow for real-time applications. We propose a novel way to adapt them for online use, enabling real-time 6-DoF pose tracking and online reconstruction of objects and scenes from
monocular RGB
videos. Our method rapidly selects and manages a set of images as keyframes to map a scene or object via $\pi^3$ with full bidirectional
attention. We then cache the global self-
attention block's key-value (KV) pairs and use them as the sole scene representation for online tracking. This allows for up to $15\times$ speedup during inference without the fear of drift or catastrophic forgetting. Our caching strategy is model-agnostic and can be applied to other off-the-shelf multi-view networks without retraining. We demonstrate KV-Tracker on both scene-level tracking and the more challenging task of on-the-fly object tracking and reconstruction without depth measurements or object priors. Experiments on the TUM RGB-D, 7-Scenes, Arctic and OnePose
datasets show the strong performance of our system while maintaining high frame-rates up to ${\sim}27$ FPS.
Read the full abstract ▾
All authors Marwan Taher, Ignacio Alzugaray, Kirill Mazur, Xin Kong, Andrew J. Davison
Computer Vision
ReFRM3D: A Radiomics-enhanced Fused Residual Multiparametric 3D Network with Multi-Scale Feature Fusion for Glioma Characterization
Md. Abdur Rahman, Md Noman Hossain, Arefin Ittesafun Abian et al. (+4 more)
Gliomas are among the most aggressive cancers, with complex diagnostic processes. Existing glioma
segmentation methods often struggle with high variability in imaging data and inadequate optimization. Furthermore, radiomic analysis is typically applied only after
segmentation is finished, limiting its ability to inform the
segmentation process itself. To address these challenges, we propose a novel radiomics-enhanced fused residual multiparametric 3D network (ReFRM3D) for
brain tumor characterization. The framework is based on a 3D U-Net architecture and features multi-scale feature fusion, hybrid upsampling, and an extended residual skip mechanism. Additionally, we introduce a radiomic conditioning mechanism that extracts texture and intensity descriptors from an intermediate coarse
segmentation and re-injects them into the decoder to refine the final output. Experimental results on BraTS2019, BraTS2020, and BraTS2021 show strong performance, with mean DSC values of 93.45%, 93.61%, and 92.06%, respectively, across whole tumor, enhancing tumor, and tumor core regions. Compared with recent models, ReFRM3D improves average DSC by 5.79%, 7.25%, and 0.96% on these
datasets. Our model also generalized well on the BraTS-Africa
dataset with an average DSC of 87.8%, despite differences in scanner field strength and patient demographics.
Read the full abstract ▾
All authors Md. Abdur Rahman, Md Noman Hossain, Arefin Ittesafun Abian, Mohaimenul Azam Khan Raiaan, Yan Zhang, Mirjam Jonkman, Sami Azam
Computer Vision
Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains
Jesen Zhang, Ningyuan Liu, Kaitong Cai et al. (+5 more)
Multimodal LLMs often produce fluent yet unreliable
reasoning, exhibiting weak step-to-step coherence and insufficient visual grounding, largely because existing
alignment approaches supervise only the final answer while ignoring the reliability of the intermediate
reasoning process. We introduce SR-MCR, a lightweight and label-free framework that aligns
reasoning by exploiting intrinsic process signals derived directly from model outputs. Five self-referential cues – semantic
alignment, lexical fidelity, non-redundancy, visual grounding, and step consistency – are integrated into a normalized, reliability-weighted reward that provides fine-grained process-level guidance. A critic-free GRPO objective, enhanced with a confidence-aware cooling mechanism, further stabilizes training and suppresses trivial or overly confident generations. Built on Qwen2.5-VL, SR-MCR improves both answer accuracy and
reasoning coherence across a broad set of visual
benchmarks; among open-source models of comparable size, SR-MCR-7B achieves state-of-the-art performance with an average accuracy of 81.4%. Ablation studies confirm the independent contributions of each reward term and the cooling module.
Read the full abstract ▾
All authors Jesen Zhang, Ningyuan Liu, Kaitong Cai, Sidi Liu, Jing Yang, Ziliang Chen, Xiaofei Sun, Keze Wang
Computer Vision · Cross-listed from cs.RO
VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
Borong Zhang, Jiahao Li, Jiachen Shen et al. (+8 more)
While
Vision-
Language-Action models (VLAs) are rapidly advancing toward generalist
robot policies, quantitatively characterizing their capability boundaries and failure modes remains challenging. To address this, we introduce VLA-Arena, a comprehensive
benchmark. It features a novel structured task design framework to quantify difficulty across three orthogonal axes: (1) Task Structure, (2)
Language Command, and (3) Visual Observation. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For task structure, VLA-Arena comprises 11 task suites organized into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon, totaling 170 tasks. Each suite spans three difficulty levels (L0-L2), with fine-tuning restricted to L0 to rigorously assess generalization. Orthogonal to this,
language (W0-W4) and visual (V0-V4) perturbations can be applied to any task as diagnostic probes to distinguish robust grounding from superficial pattern matching. Our extensive evaluation of state-of-the-art VLAs reveals critical limitations: memorization over generalization, superficial visual perception, and a neglect of safety constraints. Additionally, model rank reversals across L0-L2 validate that each level provides non-redundant insights. To foster research addressing these model limitations and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L
datasets for fine-tuning. Our
benchmark,
datasets, models, and leaderboard are publicly available at https://vla-arena.github.io.
Read the full abstract ▾
All authors Borong Zhang, Jiahao Li, Jiachen Shen, Yuhao Zhang, Yishuai Cai, Lu Liu, Hailu Ji, Yuanpei Chen, Juntao Dai, Jiaming Ji, Yaodong Yang
Computer Vision
CoAgent: Collaborative Planning and Consistency Agent for Coherent Video Generation
Qinglin Zeng, Kaitong Cai, Ruiqi Chen et al. (+2 more)
Maintaining narrative coherence and visual consistency remains a central challenge in open-domain
video generation. Existing text-to-
video models often treat each shot independently, resulting in identity drift, scene inconsistency, and unstable temporal structure. We propose CoAgent, a collaborative and closed-loop framework for coherent
video generation that formulates the process as a plan-synthesize-verify pipeline. Given a user
prompt, style reference, and pacing constraints, a Storyboard Planner decomposes the input into structured shot-level plans with explicit entities, spatial relations, and temporal cues. A Global Context Manager maintains entity-level
memory to preserve appearance and identity consistency across shots. Each shot is then generated by a Synthesis Module under the guidance of a Visual Consistency Controller, while a Verifier
Agent evaluates intermediate results using
vision-
language reasoning and triggers selective regeneration when inconsistencies are detected. Finally, a pacing-aware editor refines temporal rhythm and transitions to match the desired narrative flow. Extensive experiments demonstrate that CoAgent significantly improves coherence, visual consistency, and narrative quality in long-form
video generation.
Read the full abstract ▾
All authors Qinglin Zeng, Kaitong Cai, Ruiqi Chen, Qinhan Lv, Keze Wang
Computer Vision
DreamOmni3: Scribble-based Editing and Generation
Bin Xia, Bohao Peng, Jiyang Liu et al. (+8 more)
Recently unified generation and editing models have achieved remarkable success with their impressive performance. These models rely mainly on text
prompts for instruction-based editing and generation, but
language often fails to capture users intended edit locations and fine-grained visual details. To this end, we propose two tasks: scribble-based editing and generation, that enables more flexible creation on graphical user interface (GUI) combining user textual, images, and freehand sketches. We introduce DreamOmni3, tackling two challenges: data creation and framework design. Our data synthesis pipeline includes two parts: scribble-based editing and generation. For scribble-based editing, we define four tasks: scribble and instruction-based editing, scribble and
multimodal instruction-based editing, image fusion, and doodle editing. Based on DreamOmni2
dataset, we extract editable regions and overlay hand-drawn boxes, circles, doodles or cropped image to construct training data. For scribble-based generation, we define three tasks: scribble and instruction-based generation, scribble and
multimodal instruction-based generation, and doodle generation, following similar data creation pipelines. For the framework, instead of using binary masks, which struggle with complex edits involving multiple scribbles, images, and instructions, we propose a joint input scheme that feeds both the original and scribbled source images into the model, using different colors to distinguish regions and simplify processing. By applying the same index and position encodings to both images, the model can precisely localize scribbled regions while maintaining accurate editing. Finally, we establish comprehensive
benchmarks for these tasks to promote further research. Experimental results demonstrate that DreamOmni3 achieves outstanding performance, and models and code will be publicly released.
Read the full abstract ▾
All authors Bin Xia, Bohao Peng, Jiyang Liu, Sitong Wu, Jingyao Li, Junjia Huang, Xu Zhao, Yitong Wang, Ruihang Chu, Bei Yu, Jiaya Jia
Computer Vision
SCAFusion: A Multimodal 3D Detection Framework for Small Object Detection in Lunar Surface Exploration
Xin Chen, Kang Luo, Yangyi Xiao et al. (+1 more)
Reliable and precise detection of small and irregular objects, such as meteor fragments and rocks, is critical for autonomous
navigation and operation in lunar surface exploration. Existing
multimodal 3D perception methods designed for terrestrial autonomous driving often underperform in off world environments due to poor feature
alignment, limited
multimodal synergy, and weak small object detection. This paper presents SCAFusion, a
multimodal 3D object detection model tailored for lunar robotic missions. Built upon the BEVFusion framework, SCAFusion integrates a Cognitive Adapter for efficient camera backbone tuning, a Contrastive
Alignment Module to enhance camera
LiDAR feature consistency, a Camera Auxiliary Training Branch to strengthen visual representation, and most importantly, a Section aware Coordinate
Attention mechanism explicitly designed to boost the detection performance of small, irregular targets. With negligible increase in parameters and computation, our model achieves 69.7% mAP and 72.1% NDS on the nuScenes validation set, improving the baseline by 5.0% and 2.7%, respectively. In simulated lunar environments built on Isaac Sim, SCAFusion achieves 90.93% mAP, outperforming the baseline by 11.5%, with notable gains in detecting small meteor like obstacles.
Read the full abstract ▾
All authors Xin Chen, Kang Luo, Yangyi Xiao, Hesheng Wang
Computer Vision
Tracking by Predicting 3-D Gaussians Over Time
Tanish Baranwal, Himanshu Gaurav Singh, Jathushan Rajasegaran et al. (+1 more)
We propose
Video Gaussian Masked Autoencoders (
Video-GMAE), a self-supervised approach for representation learning that encodes a sequence of images into a set of Gaussian splats moving over time. Representing a
video as a set of Gaussians enforces a reasonable inductive bias: that 2-D
videos are often consistent projections of a dynamic 3-D scene. We find that tracking emerges when pretraining a network with this architecture. Mapping the trajectory of the learnt Gaussians onto the image plane gives zero-shot tracking performance comparable to state-of-the-art. With small-scale finetuning, our models achieve 34.6% improvement on Kinetics, and 13.1% on Kubric
datasets, surpassing existing self-supervised
video approaches. The project page and code are publicly available at https://videogmae.org/ and https://github.com/tekotan/
video-gmae.
Read the full abstract ▾
All authors Tanish Baranwal, Himanshu Gaurav Singh, Jathushan Rajasegaran, Jitendra Malik
Computer Vision · Cross-listed from cs.LG
Toward Real-World IoT Security: Concept Drift-Resilient IoT Botnet Detection via Latent Space Representation Learning and Alignment
Hassan Wasswa and Timothy Lynar
Although AI-based models have achieved high accuracy in IoT threat detection, their deployment in enterprise environments is constrained by reliance on stationary
datasets that fail to reflect the dynamic nature of real-world IoT NetFlow traffic, which is frequently affected by concept drift. Existing solutions typically rely on periodic classifier retraining, resulting in high computational overhead and the risk of catastrophic forgetting. To address these challenges, this paper proposes a scalable framework for adaptive IoT threat detection that eliminates the need for continuous classifier retraining. The proposed approach trains a classifier once on latent-space representations of historical traffic, while an
alignment model maps incoming traffic to the learned historical latent space prior to classification, thereby preserving knowledge of previously observed attacks. To capture inter-instance relationships among attack samples, the low-dimensional latent representations are further transformed into a graph-structured format and classified using a graph neural network. Experimental evaluations on real-world heterogeneous IoT traffic
datasets demonstrate that the proposed framework maintains robust detection performance under concept drift. These results highlight the framework's potential for practical deployment in dynamic and large-scale IoT environments.
Read the full abstract ▾
Computer Vision · Cross-listed from q-bio.NC
JParc: Joint cortical surface parcellation with registration
Jian Li, Karthik Gopinath, Brian L. Edlow et al. (+2 more)
Cortical surface parcellation is a fundamental task in both basic neuroscience research and clinical applications, enabling more accurate mapping of
brain regions. Model-based and learning-based approaches for automated parcellation alleviate the need for manual labeling. Despite the advancement in parcellation performance, learning-based methods shift away from registration and atlas propagation without exploring the reason for the improvement compared to traditional methods. In this study, we present JParc, a joint cortical registration and parcellation framework, that outperforms existing state-of-the-art parcellation methods. In rigorous experiments, we demonstrate that the enhanced performance of JParc is primarily attributable to accurate cortical registration and a learned parcellation atlas. By leveraging a shallow subnetwork to fine-tune the propagated atlas labels, JParc achieves a Dice score greater than 90% on the Mindboggle
dataset, using only basic
geometric features (sulcal depth,
curvature) that describe cortical folding patterns. The superior accuracy of JParc can significantly increase the statistical power in
brain mapping studies as well as support applications in
surgical planning and many other downstream neuroscientific and clinical tasks.
Read the full abstract ▾
All authors Jian Li, Karthik Gopinath, Brian L. Edlow, Adrian V. Dalca, Bruce Fischl
Computer Vision
Scalpel-SAM: A Semi-Supervised Paradigm for Adapting SAM to Infrared Small Object Detection
Zihan Liu, Xiangning Ren, Dezhang Kong et al. (+2 more)
Infrared small object detection urgently requires semi-supervised paradigms due to the high cost of annotation. However, existing methods like SAM face significant challenges of domain gaps, inability of encoding physical priors, and inherent architectural complexity. To address this, we designed a Hierarchical MoE Adapter consisting of four white-box neural operators. Building upon this core component, we propose a two-stage paradigm for knowledge
distillation and transfer: (1) Prior-Guided Knowledge
Distillation, where we use our MoE adapter and 10% of available fully supervised data to distill SAM into an expert teacher (Scalpel-SAM); and (2) Deployment-Oriented Knowledge Transfer, where we use Scalpel-SAM to generate pseudo labels for training lightweight and efficient downstream models. Experiments demonstrate that with minimal annotations, our paradigm enables downstream models to achieve performance comparable to, or even surpassing, their fully supervised counterparts. To our knowledge, this is the first semi-supervised paradigm that systematically addresses the data scarcity issue in IR-SOT using SAM as the teacher model.
Read the full abstract ▾
All authors Zihan Liu, Xiangning Ren, Dezhang Kong, Yipeng Zhang, Meng Han
Computer Vision
Event-based high temporal resolution measurement of shock wave motion field
Taihang Lei, Banglei Guan, Minzu Liang et al. (+4 more)
Accurate measurement of shock wave motion parameters with high spatiotemporal resolution is essential for applications such as power field testing and damage assessment. However, significant challenges are posed by the fast, uneven propagation of shock waves and unstable testing conditions. To address these challenges, a novel framework is proposed that utilizes multiple event cameras to estimate the asymmetry of shock waves, leveraging its high-speed and high-dynamic range capabilities. Initially, a polar coordinate system is established, which encodes events to reveal shock wave propagation patterns, with adaptive region-of-interest (ROI) extraction through event offset calculations. Subsequently, shock wave front events are extracted using iterative slope analysis, exploiting the continuity of velocity changes. Finally, the
geometric model of events and shock wave motion parameters is derived according to event-based optical imaging model, along with the 3D reconstruction model. Through the above process, multi-angle shock wave measurement, motion field reconstruction, and explosive equivalence inversion are achieved. The results of the speed measurement are compared with those of the pressure sensors and the empirical formula, revealing a maximum error of 5.20% and a minimum error of 0.06%. The experimental results demonstrate that our method achieves high-precision measurement of the shock wave motion field with both high spatial and temporal resolution, representing significant progress.
Read the full abstract ▾
All authors Taihang Lei, Banglei Guan, Minzu Liang, Pengju Sun, Jing Tao, Yang Shang, Qifeng Yu
Computer Vision
Pose-Guided Residual Refinement for Interpretable Text-to-Motion Generation and Editing
Sukhyun Jeong and Yong-Hoon Choi
Text-based 3D motion generation aims to automatically synthesize diverse motions from natural-
language descriptions to extend user creativity, whereas motion editing modifies an existing motion sequence in response to text while preserving its overall structure. Pose-code-based frameworks such as CoMo map quantifiable pose attributes into discrete pose codes that support interpretable motion control, but their frame-wise representation struggles to capture subtle temporal dynamics and high-frequency details, often degrading reconstruction fidelity and local controllability. To address this limitation, we introduce pose-guided residual refinement for motion (PGR$^2$M), a hybrid representation that augments interpretable pose codes with residual codes learned via residual vector quantization (RVQ). A pose-guided RVQ tokenizer decomposes motion into pose latents that encode coarse global structure and residual latents that model fine-grained temporal variations. Residual dropout further discourages over-reliance on residuals, preserving the semantic
alignment and editability of the pose codes. On top of this tokenizer, a base
Transformer autoregressively predicts pose codes from text, and a refine
Transformer predicts residual codes conditioned on text, pose codes, and quantization stage. Experiments on HumanML3D and KIT-ML show that PGR$^2$M improves Fréchet inception distance and reconstruction metrics for both generation and editing compared with CoMo and recent
diffusion- and tokenization-based baselines, while user studies confirm that it enables intuitive, structure-preserving motion edits.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
MEGA-PCC: A Mamba-based Efficient Approach for Joint Geometry and Attribute Point Cloud Compression
Kai-Hsiang Hsieh, Monyneath Yim, Wen-Hsiao Peng et al. (+1 more)
Joint compression of point cloud geometry and attributes is essential for efficient 3D data representation. Existing methods often rely on post-hoc recoloring procedures and manually tuned bitrate allocation between geometry and attribute bitstreams in inference, which hinders end-to-end optimization and increases system complexity. To overcome these limitations, we propose MEGA-PCC, a fully end-to-end, learning-based framework featuring two specialized models for joint compression. The main compression model employs a shared encoder that encodes both geometry and attribute information into a unified latent representation, followed by dual decoders that sequentially reconstruct geometry and then attributes. Complementing this, the Mamba-based
Entropy Model (MEM) enhances
entropy coding by capturing spatial and channel-wise correlations to improve probability estimation. Both models are built on the Mamba architecture to effectively model long-range dependencies and rich contextual features. By eliminating the need for recoloring and heuristic bitrate tuning, MEGA-PCC enables data-driven bitrate allocation during training and simplifies the overall pipeline. Extensive experiments demonstrate that MEGA-PCC achieves superior rate-distortion performance and runtime efficiency compared to both traditional and learning-based baselines, offering a powerful solution for AI-driven point cloud compression.
Read the full abstract ▾
All authors Kai-Hsiang Hsieh, Monyneath Yim, Wen-Hsiao Peng, Jui-Chiu Chiang
Computer Vision
Comparing Object Detection Models for Electrical Substation Component Mapping
Namish Bansal, Haley Mody, Dennies Kiprono Bor et al. (+2 more)
Electrical substations are a significant component of an electrical grid. Indeed, the assets at these substations (e.g.,
transformers) are vulnerable to hazards such as hurricanes, flooding, earthquakes, and geomagnetically induced currents (GICs). Because failures can have significant economic and public safety implications, identifying key substation components is essential for quantifying vulnerability. Unfortunately, traditional manual mapping of substation infrastructure is time-consuming and labor-intensive. Therefore, an autonomous solution utilizing computer
vision models is preferable, as it offers greater convenience and efficiency. In this study, we train and compare 16 models on a manually labeled
dataset of US substation images. These models include 12 You Only Look Once (YOLO) models, 2 Roboflow Detection
Transformer (RF-DETR) models, and 2 Cascade R-CNN models. RF-DETR-large achieved the highest overall detection performance with mAP@50 and mAP@50:95 scores of 0.881 and 0.632, respectively. Across all models, alternate energy systems were detected most accurately, while
transformers and reactors were more difficult to identify due to their smaller size and greater visual variability. Applying our best-performing model to nationwide imagery yielded approximately 22,591 component detections across 11,083 unique substations within the United States. These detections were broken down by state and Federal Energy Regulatory Commission (FERC) regions, with Florida (2,478 detections) and Midcontinent Independent System Operator (MISO; 4,329 detections) having the largest number of detections in their respective categories.
Read the full abstract ▾
All authors Namish Bansal, Haley Mody, Dennies Kiprono Bor, Dante Groccia, Edward J. Oughton
Computer Vision
SAM 3D for 3D Object Reconstruction from Remote Sensing Images
Junsheng Yao, Lichao Mou and Qingyu Li
Monocular 3D building reconstruction from remote sensing imagery is essential for scalable urban modeling, yet existing methods often require task-specific architectures and intensive supervision. This paper presents the first systematic evaluation of SAM 3D, a general-purpose image-to-3D foundation model, for
monocular remote sensing building reconstruction. We
benchmark SAM 3D against TRELLIS on samples from the NYC Urban
Dataset, employing Frechet Inception Distance (FID) and CLIP-based Maximum Mean Discrepancy (CMMD) as evaluation metrics. Experimental results demonstrate that SAM 3D produces more coherent roof geometry and sharper boundaries compared to TRELLIS. We further extend SAM 3D to urban scene reconstruction through a segment-reconstruct-compose pipeline, demonstrating its potential for urban scene modeling. We also analyze practical limitations and discuss future research directions. These findings provide practical guidance for deploying foundation models in urban 3D reconstruction and motivate future integration of scene-level structural priors.
Read the full abstract ▾
Computer Vision
SonoVision: A Computer Vision Approach for Helping Visually Challenged Individuals Locate Objects with the Help of Sound Cues
Md Abu Obaida Zishan and Annajiat Alim Rasel
Locating objects for the visually impaired is a significant challenge and is something no one can get used to over time. However, this hinders their independence and could push them towards risky and dangerous scenarios. Hence, in the spirit of making the visually challenged more self-sufficient, we present SonoVision, a smart-phone application that helps them find everyday objects using sound cues through earphones/headphones. This simply means, if an object is on the right or left side of a user, the app makes a sinusoidal sound in a user's respective ear through ear/headphones. However, to indicate objects located directly in front, both the left and right earphones are rung simultaneously. These sound cues could easily help a visually impaired individual locate objects with the help of their smartphones and reduce the reliance on people in their surroundings, consequently making them more independent. This application is made with the flutter development platform and uses the Efficientdet-D2 model for object detection in the backend. We believe the app will significantly assist the visually impaired in a safe and user-friendly manner with its capacity to work completely offline. Our application can be accessed here https://github.com/MohammedZ666/SonoVision.git.
Read the full abstract ▾
Computer Vision
Towards Robust Optical-SAR Object Detection under Missing Modalities: A Dynamic Quality-Aware Fusion Framework
Zhicheng Zhao, Yuancheng Xu, Andong Lu et al. (+2 more)
Optical and Synthetic Aperture Radar (SAR) fusion-based object detection has attracted significant research interest in remote sensing, as these modalities provide complementary information for all-weather monitoring. However, practical deployment is severely limited by inherent challenges. Due to distinct imaging mechanisms, temporal asynchrony, and registration difficulties, obtaining well-aligned optical-SAR image pairs remains extremely difficult, frequently resulting in missing or degraded modality data. Although recent approaches have attempted to address this issue, they still suffer from limited robustness to random missing modalities and lack effective mechanisms to ensure consistent performance improvement in fusion-based detection. To address these limitations, we propose a novel Quality-Aware Dynamic Fusion Network (QDFNet) for robust optical-SAR object detection. Our proposed method leverages learnable reference
tokens to dynamically assess feature reliability and guide adaptive fusion in the presence of missing modalities. In particular, we design a Dynamic Modality Quality Assessment (DMQA) module that employs learnable reference
tokens to iteratively refine feature reliability assessment, enabling precise identification of degraded regions and providing quality guidance for subsequent fusion. Moreover, we develop an Orthogonal Constraint Normalization Fusion (OCNF) module that employs orthogonal constraints to preserve modality independence while dynamically adjusting fusion weights based on reliability scores, effectively suppressing unreliable feature propagation. Extensive experiments on the SpaceNet6-OTD and OGSOD-2.0
datasets demonstrate the superiority and effectiveness of QDFNet compared to state-of-the-art methods, particularly under partial modality corruption or missing data scenarios.
Read the full abstract ▾
All authors Zhicheng Zhao, Yuancheng Xu, Andong Lu, Chenglong Li, Jin Tang
Computer Vision
LECalib: Line-Based Event Camera Calibration
Zibin Liu, Banglei Guan, Yang Shang et al. (+3 more)
Camera calibration is an essential prerequisite for event-based
vision applications. Current event camera calibration methods typically involve using flashing patterns, reconstructing intensity images, and utilizing the features extracted from events. Existing methods are generally time-consuming and require manually placed calibration objects, which cannot meet the needs of rapidly changing scenarios. In this paper, we propose a line-based event camera calibration framework exploiting the
geometric lines of commonly-encountered objects in man-made environments, e.g., doors, windows, boxes, etc. Different from previous methods, our method detects lines directly from event streams and leverages an event-line calibration model to generate the initial guess of camera parameters, which is suitable for both planar and non-planar lines. Then, a non-linear optimization is adopted to refine camera parameters. Both simulation and real-world experiments have demonstrated the feasibility and accuracy of our method, with validation performed on
monocular and stereo event cameras. The source code is released at https://github.com/Zibin6/line_based_event_camera_calib.
Read the full abstract ▾
All authors Zibin Liu, Banglei Guan, Yang Shang, Zhenbao Yu, Yifei Bian, Qifeng Yu
Computer Vision
SuperiorGAT: Graph Attention Networks for Sparse LiDAR Point Cloud Reconstruction in Autonomous Systems
Khalfalla Awedat, Mohamed Abidalrekab, Gurcan Comert et al. (+1 more)
LiDAR-based perception in autonomous systems is constrained by fixed vertical beam resolution and further compromised by beam dropout resulting from environmental occlusions. This paper introduces SuperiorGAT, a graph
attention-based framework designed to reconstruct missing elevation information in sparse
LiDAR point clouds. By modeling
LiDAR scans as beam-aware graphs and incorporating gated residual fusion with feed-forward refinement, SuperiorGAT enables accurate reconstruction without increasing network depth. To evaluate performance, structured beam dropout is simulated by removing every fourth vertical scanning beam. Extensive experiments across diverse KITTI environments, including Person, Road, Campus, and City sequences, demonstrate that SuperiorGAT consistently achieves lower reconstruction error and improved
geometric consistency compared to PointNet-based models and deeper GAT baselines. Qualitative X-Z projections further confirm the model's ability to preserve structural integrity with minimal vertical distortion. These results suggest that architectural refinement offers a computationally efficient method for improving
LiDAR resolution without requiring additional sensor hardware.
Read the full abstract ▾
All authors Khalfalla Awedat, Mohamed Abidalrekab, Gurcan Comert, Mustafa Ayad
Computer Vision
EmoCtrl: Controllable Emotional Image Content Generation
Jingyuan Yang, Weibin Luo and Hui Huang
An image conveys meaning through both its visual content and emotional tone, jointly shaping human perception. We introduce Controllable Emotional Image Content Generation (C-EICG), which aims to generate images that remain faithful to a given content description while expressing a target
emotion. Existing text-to-image models ensure content consistency but lack emotional awareness, whereas
emotion-driven models generate affective results at the cost of content distortion. To address this gap, we propose EmoCtrl, supported by a
dataset annotated with content,
emotion, and affective
prompts, bridging abstract
emotions to visual cues. EmoCtrl incorporates textual and visual
emotion enhancement modules that enrich affective expression via descriptive semantics and perceptual cues. To align with human preference, we further introduce an
emotion-driven preference optimization with specifically designed
emotion reward. Comprehensive experiments demonstrate that EmoCtrl achieves faithful content and expressive
emotion control, outperforming existing methods. User studies confirm EmoCtrl's strong
alignment with human preference. Moreover, EmoCtrl generalizes well to creative applications, further demonstrating the robustness and adaptability of the learned
emotion tokens.
Read the full abstract ▾
Computer Vision
FluenceFormer: Transformer-Driven Multi-Beam Fluence Map Regression for Radiotherapy Planning
Ujunwa Mgboh, Rafi Ibn Sultan, Joshua Kim et al. (+2 more)
Fluence map prediction is central to automated radiotherapy planning but remains an ill-posed inverse problem due to the complex relationship between volumetric anatomy and beam-intensity modulation. Convolutional methods in prior work often struggle to capture long-range dependencies, which can lead to structurally inconsistent or physically unrealizable plans. We introduce FluenceFormer, a backbone-agnostic
transformer framework for direct, geometry-aware fluence regression. The model uses a unified two-stage design: Stage~1 predicts a global dose prior from anatomical inputs, and Stage~2 conditions this prior on explicit beam geometry to regress physically calibrated fluence maps. Central to the approach is the Fluence-Aware Regression (FAR) loss, a physics-informed objective that integrates voxel-level fidelity, gradient smoothness, structural consistency, and beam-wise energy conservation. We evaluate the generality of the framework across multiple
transformer backbones, including Swin UNETR, UNETR, nnFormer, and MedFormer, using a prostate IMRT
dataset. FluenceFormer with Swin UNETR achieves the strongest performance among the evaluated models and improves over existing
benchmark CNN and single-stage methods, reducing Energy Error to $\mathbf{4.5\%}$ and yielding statistically significant gains in structural fidelity ($p < 0.05$).
Read the full abstract ▾
All authors Ujunwa Mgboh, Rafi Ibn Sultan, Joshua Kim, Kundan Thind, Dongxiao Zhu
Computer Vision
Bright 4B: Scaling Hyperspherical Learning for Segmentation in 3D Brightfield Microscopy
Amil Khan, Matheus Palhares Viana, Suraj Mishra et al. (+1 more)
Label-free 3D brightfield microscopy offers a fast and noninvasive way to visualize cellular morphology, yet robust volumetric
segmentation still typically depends on fluorescence or heavy post-processing. We address this gap by introducing Bright-4B, a 4 billion parameter foundation model that learns on the unit hypersphere to segment subcellular structures directly from 3D brightfield volumes. Bright-4B combines a hardware-aligned Native Sparse
Attention mechanism (capturing local, coarse, and selected global context), depth-width residual HyperConnections that stabilize representation flow, and a soft Mixture-of-Experts for adaptive capacity. A plug-and-play anisotropic patch embed further respects confocal point-spread and axial thinning, enabling geometry-faithful 3D tokenization. The resulting model produces morphology-accurate
segmentations of
nuclei, mitochondria, and other organelles from brightfield stacks alone–without fluorescence, auxiliary channels, or handcrafted post-processing. Across multiple confocal
datasets, Bright-4B preserves fine structural detail across depth and
cell types, outperforming contemporary CNN and
Transformer baselines. All code, pretrained weights, and models for downstream finetuning will be released to advance large-scale, label-free 3D
cell mapping.
Read the full abstract ▾
All authors Amil Khan, Matheus Palhares Viana, Suraj Mishra, B.S. Manjunath