Computer Vision
ALOOD: Exploiting Language Representations for LiDAR-based Out-of-Distribution Object Detection
Michael Kösel, Marcel Schreiber, Michael Ulrich et al. (+2 more)
LiDAR-based 3D object detection plays a critical role for reliable and safe autonomous driving systems. However, existing detectors often produce overly confident predictions for objects not belonging to known categories, posing significant safety risks. This is caused by so-called out-of-distribution (OOD) objects, which were not part of the training data, resulting in incorrect predictions. To address this challenge, we propose ALOOD (Aligned
LiDAR representations for Out-Of-Distribution Detection), a novel approach that incorporates
language representations from a
vision-
language model (VLM). By aligning the object features from the object detector to the feature space of the VLM, we can treat the detection of OOD objects as a zero-shot classification task. We demonstrate competitive performance on the nuScenes OOD
benchmark, establishing a novel approach to OOD object detection in
LiDAR using
language representations. The source code is available at https://github.com/uulm-mrm/mmood3d.
Read the full abstract ▾
All authors Michael Kösel, Marcel Schreiber, Michael Ulrich, Claudius Gläser, Klaus Dietmayer
Computer Vision
MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals
Junyu Shen, Zhendong She, Chenghanyu Zhang et al. (+13 more)
The paradigm of
Multimodal Large
Language Models (MLLMs) offers a promising blueprint for advancing the electromagnetic (EM) domain. However, prevailing approaches often deviate from the native MLLM paradigm, instead using task-specific or pipelined architectures that lead to fundamental limitations in model performance and generalization. Fully realizing the MLLM potential in EM domain requires overcoming three main challenges: (1) Data. The scarcity of high-quality
datasets with paired EM signals and descriptive text annotations used for MLLMs pre-training; (2)
Benchmark. The absence of comprehensive
benchmarks to systematically evaluate and compare the performance of models on EM signal-to-text tasks; (3) Model. A critical fragility in low Signal-to-Noise Ratio (SNR) environments, where critical signal features can be obscured, leading to significant performance degradation. To address these challenges, we introduce a tripartite contribution to establish a foundation for MLLMs in the EM domain. First, to overcome data scarcity, we construct and release EM-100k, a large-scale
dataset comprising over 100,000 EM signal-text pairs. Second, to enable rigorous and standardized evaluation, we propose EM-Bench, the most comprehensive
benchmark featuring diverse downstream tasks spanning from perception to
reasoning. Finally, to tackle the core modeling challenge, we present MERLIN, a novel training framework designed not only to align low-level signal representations with high-level semantic text, but also to explicitly enhance model robustness and performance in challenging low-SNR environments. Comprehensive experiments validate our method, showing that MERLIN is state-of-the-art in the EM-Bench and exhibits remarkable robustness in low-SNR settings.
Read the full abstract ▾
All authors Junyu Shen, Zhendong She, Chenghanyu Zhang, Yuchuang Sun, Luqing Luo, Dingwei Tan, Zonghao Guo, Bo Guo, Zehua Han, Wupeng Xie, Yaxin Mu, Peng Zhang, Peipei Li, Fengxiang Wang, Yangang Sun, Maosong Sun
Computer Vision
Edged USLAM: Edge-Aware Event-Based SLAM with Learning-Based Depth Priors
Şebnem Sar{\i}özkan, Hürkan Şahin, Olaya Álvarez-Tuñón et al. (+1 more)
Conventional visual simultaneous localization and mapping (SLAM)
algorithms often fail under rapid motion, low illumination, or abrupt lighting transitions due to motion blur and limited dynamic range. Event cameras mitigate these issues with high temporal resolution and high dynamic range (HDR), but their sparse, asynchronous outputs complicate feature extraction and integration with other sensors; e.g. inertial measurement units (IMUs) and standard cameras. We present Edged USLAM, a hybrid visual-inertial system that extends Ultimate SLAM (USLAM) with an edge-aware front-end and a lightweight depth module. The frontend enhances event frames for robust feature tracking and nonlinear motion compensation, while the depth module provides coarse, region-of-interest (ROI)-based scene depth to improve motion compensation and scale consistency. Evaluations across public
benchmarks and real-world unmanned air vehicle (
UAV) flights demonstrate that performance varies significantly by scenario. For instance, event-only methods like point-line event-based visual-inertial odometry (PL-EVIO) or learning-based pipelines such as deep event-based visual odometry (DEVO) excel in highly aggressive or extreme HDR conditions. In contrast, Edged USLAM provides superior stability and minimal drift in slow or structured trajectories, ensuring consistently accurate localization on real flights under challenging illumination. These findings highlight the complementary strengths of event-only, learning-based, and hybrid approaches, while positioning Edged USLAM as a robust solution for diverse aerial
navigation tasks.
Read the full abstract ▾
All authors Şebnem Sar{\i}özkan, Hürkan Şahin, Olaya Álvarez-Tuñón, Erdal Kayacan
Computer Vision
MV-Fashion: Towards Enabling Virtual Try-On and Size Estimation with Multi-View Paired Data
Hunor Laczkó, Libang Jia, Loc-Phat Truong et al. (+4 more)
Existing 4D human
datasets fall short for fashion-specific research, lacking either realistic garment dynamics or task-specific annotations. Synthetic
datasets suffer from a realism gap, whereas real-world captures lack the detailed annotations and paired data required for virtual try-on (VTON) and size estimation tasks. To bridge this gap, we introduce MV-Fashion, a large-scale, multi-view
video dataset engineered for domain-specific fashion analysis. MV-Fashion features 3,273 sequences (72.5 million frames) from 80 diverse subjects wearing 3-10 outfits each. It is designed to capture complex, real-world garment dynamics, including multiple layers and varied styling (e.g. rolled sleeves, tucked shirt). A core contribution is a rich data representation that includes pixel-level semantic annotations, ground-truth material properties like elasticity, and 3D point clouds. Crucially for VTON applications, MV-Fashion provides paired data: multi-view synchronized captures of worn garments alongside their corresponding flat, catalogue images. We leverage this
dataset to establish baselines for fashion-centric tasks, including virtual try-on, clothing size estimation, and novel view synthesis. The
dataset is available at https://hunorlaczko.github.io/MV-Fashion .
Read the full abstract ▾
All authors Hunor Laczkó, Libang Jia, Loc-Phat Truong, Diego Hernández, Sergio Escalera, Jordi Gonzalez, Meysam Madadi
Computer Vision
VesselFusion: Diffusion Models for Vessel Centerline Extraction from 3D CT Images
Soichi Mita, Shumpei Takezaki and Ryoma Bise
Vessel centerline extraction from 3D CT images is an important task because it reduces annotation effort to build a model that estimates a vessel structure. It is challenging to estimate natural vessel structures since conventional approaches are deterministic models, which cannot capture a complex human structure. In this study, we propose VesselFusion, which is a
diffusion model to extract the vessel centerline from 3D CT image. The proposed method uses a coarse-to-fine representation of the centerline and a voting-based aggregation for a natural and stable extraction. VesselFusion was evaluated on a publicly available CT image
dataset and achieved higher extraction accuracy and a more natural result than conventional approaches.
Read the full abstract ▾
Computer Vision
Fast Low-light Enhancement and Deblurring for 3D Dark Scenes
Feng Zhang, Jinglong Wang, Ze Li et al. (+4 more)
Novel view synthesis from low-light, noisy, and motion-blurred imagery remains a valuable and challenging task. Current volumetric rendering methods struggle with compound degradation, and sequential 2D preprocessing introduces artifacts due to interdependencies. In this work, we introduce FLED-GS, a fast low-light enhancement and deblurring framework that reformulates 3D scene restoration as an alternating cycle of enhancement and reconstruction. Specifically, FLED-GS inserts several intermediate brightness anchors to enable progressive recovery, preventing noise blow-up from harming deblurring or geometry. Each iteration sharpens inputs with an off-the-shelf 2D deblurrer and then performs noise-aware 3DGS reconstruction that estimates and suppresses noise while producing clean priors for the next level. Experiments show FLED-GS outperforms state-of-the-art LuSh-NeRF, achieving 21$\times$ faster training and 11$\times$ faster rendering.
Read the full abstract ▾
All authors Feng Zhang, Jinglong Wang, Ze Li, Yanghong Zhou, Yang Chen, Lei Chen, Xiatian Zhu
Computer Vision · Cross-listed from cs.RO
UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing
Jiaxi Zhang, Yunheng Wang, Wei Lu et al. (+8 more)
3D Visual Grounding (3DVG) localizes objects from natural-
language descriptions in 3D scenes and is fundamental to
embodied AI applications. Although foundation models enable open-vocabulary
reasoning, they typically rely on pre-generated candidates, creating two sequential bottlenecks. The candidate bottleneck occurs when
dataset-specific 3D proposal models miss, fragment, or incorrectly group targets under distribution shifts, excluding them from VLM
reasoning. The evidence bottleneck stems from incomplete visual evidence: global renderings preserve spatial context but obscure object details, whereas candidate-centric views capture local appearance but lack global context. To address these bottlenecks, we propose UniGround, a zero-shot 3DVG framework that addresses both bottlenecks through Global Candidate Filtering and Contextual Precision Grounding. Global Candidate Filtering constructs
topology-consistent, class-agnostic candidates from 3D
topology and multi-view semantic cues, without
dataset-trained 3D detectors, task-specific proposal supervision, or predefined box and category priors. Contextual Precision Grounding jointly reasons over global spatial context and candidate-centric visual evidence, followed by closed-loop consistency verification for reliable target identification. UniGround achieves 46.1%/34.1% Acc@0.25/0.5 on ScanRefer and 28.7% Acc@0.25 on the evaluated ARKitScenes subset of EmbodiedScan. Further experiments demonstrate competitive grounding without
dataset-specific 3D priors, cross-
dataset generalization to unseen indoor scenes, and robustness to real-world reconstruction noise and practical domain shifts.
Read the full abstract ▾
All authors Jiaxi Zhang, Yunheng Wang, Wei Lu, Taowen Wang, Shuning Zhang, Yixiao Feng, Junzhe Xu, Shunwang Sun, Weisheng Xu, Yuetong Fang, Renjing Xu
Computer Vision
Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows
Shentong Mo and Yibing Song
Coordinated audio generation based on
video inputs typically requires a strict audio-visual (AV)
alignment, where both semantics and rhythmics of the generated audio segments shall correspond to those in the
video frames. Previous studies leverage a two-stage design where the AV encoders are firstly aligned via contrastive learning, then the encoded
video representations guide the audio generation process. We observe that both contrastive learning and global
video guidance are effective in aligning overall AV semantics while limiting temporally rhythmic synchronization. In this work, we propose FoleyFlow to first align unimodal AV encoders via masked modeling training, where the masked audio segments are recovered under the guidance of the corresponding
video segments. After training, the AV encoders which are separately pretrained using only unimodal data are aligned with semantic and rhythmic consistency. Then, we develop a dynamic conditional flow for the final audio generation. Built upon the efficient velocity flow generation framework, our dynamic conditional flow utilizes temporally varying
video features as the dynamic condition to guide corresponding audio segment generations. To this end, we extract coherent semantic and rhythmic representations during masked AV
alignment, and use this representation of
video segments to guide audio generation temporally. Our audio results are evaluated on the standard
benchmarks and largely surpass existing results under several metrics. The superior performance indicates that FoleyFlow is effective in generating coordinated audios that are both semantically and rhythmically coherent to various
video sequences.
Read the full abstract ▾
Computer Vision
SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving
Zihan You, Hongwei Liu, Chenxu Dang et al. (+4 more)
Recent advances in
Vision-
Language-Action (VLA) models have shown promising capabilities in autonomous driving by leveraging the understanding and
reasoning strengths of Large
Language Models(LLMs).However, our empirical analysis reveals that directly applying existing
token-level MoE mechanisms–which are inherited from LLM architectures–to VLA models results in unstable performance and safety degradation in autonomous driving, highlighting a misalignment between
token-based expert specialization and scene-level decision-making.To address this, we propose SAMoE-VLA, a scene-adaptive
Vision-
Language-Action framework that conditions expert selection on structured scene representations instead of
token embeddings. Our key idea is to derive the MoE routing signal from bird's-eye-view (BEV) features that encapsulates traffic scene context, enabling scenario-dependent expert weighting and merging tailored to distinct driving conditions. Furthermore, to support temporally consistent
reasoning across world-knowledge, perception,
language, and action, we introduce a Conditional Cross-Modal Causal
Attention mechanism that integrates world state, linguistic intent, and action history into a unified causal
reasoning process. Extensive experiments on the nuScenes open loop planning
dataset and LangAuto closed-loop
benchmark demonstrate that SAMoE-VLA achieves state-of-the-art performance, outperforming prior VLA-based and world-model-based approaches with fewer parameters.Our code will be released soon.
Read the full abstract ▾
All authors Zihan You, Hongwei Liu, Chenxu Dang, Zhe Wang, Sining Ang, Aoqi Wang, Yan Wang
Computer Vision
Chengchao Shen
Large
vision transformers present impressive scalability, as their performance can be well improved with increased model capacity. Nevertheless, their cumbersome parameters results in exorbitant computational and
memory demands. By analyzing prevalent
transformer structures, we find that multilayer perceptron (MLP) modules constitute the largest share of the model's parameters. In this paper, we propose an Adaptive MLP
Pruning (AMP) method to substantially reduce the parameters of large
vision transformers without obvious performance degradation. First, we adopt Taylor based method to evaluate
neuron importance of MLP. However, the importance computation using one-hot cross
entropy loss ignores the potential predictions on other categories, thus degrading the quality of the evaluated importance scores. To address this issue, we introduce label-free information
entropy criterion to fully model the predictions of the original model for more accurate importance evaluation. Second, we rank the hidden
neurons of MLP by the above importance scores and apply binary search
algorithm to adaptively prune the ranked
neurons according to the redundancy of different MLP modules, thereby avoiding the predefined compression ratio. Experimental results on several state-of-the-art large
vision transformers, including CLIP and DINOv2, demonstrate that our method achieves roughly 40% parameter and FLOPs reduction in a near lossless manner. Moreover, when the models are not finetuned after
pruning, our method outperforms other
pruning methods by significantly large margin. The source code and trained weights are available at https://github.com/visresearch/AMP.
Read the full abstract ▾
Computer Vision
TrianguLang: Geometry-Aware Semantic Consensus for Pose-Free 3D Localization
Bryce Grant, Aryeh Rothenberg, Atri Banerjee et al. (+1 more)
Localizing objects and parts from natural
language in 3D space is essential for robotics, AR, and
embodied AI, yet existing methods face a trade-off between the accuracy and
geometric consistency of per-scene optimization and the efficiency of feed-forward inference. We present TrianguLang, a feed-forward framework for 3D localization that requires no camera calibration at inference. Unlike prior methods that treat views independently, we introduce Geometry-Aware Semantic
Attention (GASA), which utilizes predicted geometry to gate cross-view feature correspondence, suppressing semantically-plausible but geometrically-inconsistent matches without requiring ground-truth poses. Validated on five
benchmarks including ScanNet++ and uCO3D, TrianguLang achieves state-of-the-art feed-forward text-guided
segmentation and localization, reducing user effort from $O(N)$ clicks to a single text query. The model processes each frame at 1008x1008 resolution in $\sim$57ms ($\sim$18 FPS) without optimization, enabling practical deployment for interactive robotics and AR applications. Code and checkpoints are available at https://cwru-aism.github.io/triangulang/.
Read the full abstract ▾
All authors Bryce Grant, Aryeh Rothenberg, Atri Banerjee, Peng Wang
Computer Vision
DSH-Bench: A Difficulty- and Scenario-Aware Benchmark with Hierarchical Subject Taxonomy for Subject-Driven Text-to-Image Generation
Zhenyu Hu, Qing Wang, Te Cao et al. (+11 more)
Significant progress has been achieved in subject-driven text-to-image (T2I) generation, which aims to synthesize new images depicting target subjects according to user instructions. However, evaluating these models remains a significant challenge. Existing
benchmarks exhibit critical limitations: 1) insufficient diversity and comprehensiveness in subject images, 2) inadequate granularity in assessing model performance across different subject difficulty levels and
prompt scenarios, and 3) a profound lack of actionable insights and diagnostic guidance for subsequent model refinement. To address these limitations, we propose DSH-Bench, a comprehensive
benchmark that enables systematic multi-perspective analysis of subject-driven T2I models through four principal innovations: 1) a hierarchical taxonomy sampling mechanism ensuring comprehensive subject representation across 58 fine-grained categories, 2) an innovative classification scheme categorizing both subject difficulty level and
prompt scenario for granular capability assessment, 3) a novel Subject Identity Consistency Score (SICS) metric demonstrating a 9.4% higher correlation with human evaluation compared to existing measures in quantifying subject preservation, and 4) a comprehensive set of diagnostic insights derived from the
benchmark, offering critical guidance for optimizing future model training paradigms and data construction strategies. Through an extensive empirical evaluation of 19 leading models, DSH-Bench uncovers previously obscured limitations in current approaches, establishing concrete directions for future research and development.
Read the full abstract ▾
All authors Zhenyu Hu, Qing Wang, Te Cao, Luo Liao, Longfei Lu, Liqun Liu, Shuang Li, Hang Chen, Mengge Xue, Yuan Chen, Chao Deng, Peng Shu, Huan Yu, Jie Jiang
Computer Vision
From Reactive to Map-Based AI: Tuned Local LLMs for Semantic Zone Inference in Object-Goal Navigation
Yudai Noda and Kanji Tanaka
Object-Goal
Navigation (ObjectNav) requires an
agent to find and navigate to a target object category in unknown environments. While recent Large
Language Model (LLM)-based
agents exhibit zero-shot
reasoning, they often rely on a "reactive" paradigm that lacks explicit spatial
memory, leading to redundant exploration and myopic behaviors. To address these limitations, we propose a transition from reactive AI to "Map-Based AI" by integrating LLM-based semantic inference with a hybrid topological-grid mapping system. Our framework employs a fine-tuned Llama-2 model via Low-Rank Adaptation (LoRA) to infer semantic zone categories and target existence probabilities from verbalized object observations. In this study, a "zone" is defined as a functional area described by the set of observed objects, providing crucial semantic co-occurrence cues for finding the target. This semantic information is integrated into a topological graph, enabling the
agent to prioritize high-probability areas and perform systematic exploration via Traveling Salesman Problem (TSP) optimization. Evaluations in the AI2-THOR simulator demonstrate that our approach significantly outperforms traditional frontier exploration and reactive LLM baselines, achieving a superior Success Rate (SR) and Success weighted by Path Length (SPL).
Read the full abstract ▾
Computer Vision
TALON: Test-time Adaptive Learning for On-the-Fly Category Discovery
Yanan Wu, Yuhan Yan, Tailai Chen et al. (+5 more)
On-the-fly category discovery (OCD) aims to recognize known categories while simultaneously discovering novel ones from an unlabeled online stream, using a model trained only on labeled data. Existing approaches freeze the feature extractor trained offline and employ a hash-based framework that quantizes features into binary codes as class prototypes. However, discovering novel categories with a fixed knowledge base is counterintuitive, as the learning potential of incoming data is entirely neglected. In addition, feature quantization introduces information loss, diminishes representational expressiveness, and amplifies intra-class variance. It often results in category explosion, where a single class is fragmented into multiple pseudo-classes. To overcome these limitations, we propose a test-time adaptation framework that enables learning through discovery. It incorporates two complementary strategies: a semantic-aware prototype update and a stable test-time encoder update. The former dynamically refines class prototypes to enhance classification, whereas the latter integrates new information directly into the parameter space. Together, these components allow the model to continuously expand its knowledge base with newly encountered samples. Furthermore, we introduce a margin-aware logit calibration in the offline stage to enlarge inter-class margins and improve intra-class compactness, thereby reserving
embedding space for future class discovery. Experiments on standard OCD
benchmarks demonstrate that our method substantially outperforms existing hash-based state-of-the-art approaches, yielding notable improvements in novel-class accuracy and effectively mitigating category explosion. The code is publicly available at \textcolor{blue}{https://github.com/ynanwu/TALON}.
Read the full abstract ▾
All authors Yanan Wu, Yuhan Yan, Tailai Chen, Zhixiang Chi, ZiZhang Wu, Yi Jin, Yang Wang, Zhenbo Li
Computer Vision
Synthetic Defect Image Generation for Power Line Insulator Inspection Using Multimodal Large Language Models
Xuesong Wang and Caisheng Wang
Utility companies increasingly rely on drone imagery for post-event and routine inspection, but training accurate defect-type classifiers remains difficult because defect examples are rare and inspection
datasets are often limited or proprietary. We address this data-scarcity setting by using an off-the-shelf
multimodal large
language model (MLLM) as a training-free image generator to synthesize defect images from visual references and text
prompts. Our pipeline increases diversity via dual-reference conditioning, improves label fidelity with lightweight human verification and
prompt refinement, and filters the resulting synthetic pool using an
embedding-based selection rule based on distances to class centroids computed from the real training split. We evaluate on ceramic
insulator defect-type classification (shell vs. glaze) using a public
dataset with a realistic low training-data regime (104 real training images; 152 validation; 308 test). Augmenting the 10% real training set with
embedding-selected synthetic images improves test F1 score (harmonic mean of precision and recall) from 0.615 to 0.739 (20% relative), corresponding to an estimated 4–5x data-efficiency gain, and the gains persist with stronger backbone models and frozen-feature linear-probe baselines. These results suggest a practical, low-barrier path for improving defect recognition when collecting additional real defects is slow or infeasible.
Read the full abstract ▾
Computer Vision
Evaluating Generative Models via One-Dimensional Code Distributions
Zexi Jia, Pengcheng Luo, Yijia Zhong et al. (+2 more)
Most evaluations of generative models rely on feature-distribution metrics such as FID, which operate on continuous recognition features that are explicitly trained to be invariant to appearance variations, and thus discard cues critical for perceptual quality. We instead evaluate models in the space of discrete visual
tokens, where modern 1D image tokenizers compactly encode both semantic and perceptual information and quality manifests as predictable
token statistics. We introduce Codebook Histogram Distance (CHD), a training-free distribution metric in
token space, and Code Mixture Model Score (CMMS), a no-reference quality metric learned from synthetic degradations of
token sequences. To stress-test metrics under broad distribution shifts, we further propose VisForm, a
benchmark of 210K images spanning 62 visual forms and 12 generative models with expert annotations. Across AGIQA, HPDv2/3, and VisForm, our
token-based metrics achieve state-of-the-art correlation with human judgments. We will release all code and
datasets to facilitate future research, with the code publicly available at https://github.com/zexiJia/1d-Distance.
Read the full abstract ▾
All authors Zexi Jia, Pengcheng Luo, Yijia Zhong, Jinchao Zhang, Jie Zhou
Computer Vision
SkyLink: A Large Vision-Language Model Driven Re-ranking Framework for Cross-View UAV geolocalization
Bowen Liu, Pengyue Jia, Wanyu Wang et al. (+9 more)
Cross-view
UAV geolocalization is fundamentally a challenging large-scale image
retrieval task, aiming to determine the geographic coordinates of Unmanned Aerial Vehicle (
UAV) queries by matching them against an extensive geo-tagged satellite image database. Most existing methods learn separate feature representations for each view and determine the final prediction using naive heuristics to assess feature similarity, thereby neglecting to model the crucial cross-view relationships. In this paper, we propose SkyLink, a novel plug-and-play ranking framework that pioneers joint relational modeling of inter-view relationships to enhance cross-view
UAV geolocalization. SkyLink leverages a Large
Vision-
Language Model (LVLM) to model the intricate visual-semantic relationships between
UAV and satellite views, facilitating effective cross-view matching. To further refine the learning process, we introduce a relational-aware loss. It leverages soft labels to provide a more nuanced supervision signal, mitigating the harsh penalty on near-positive pairs. This approach enhances both training stability and the model's discriminative capacity. Extensive experiments conducted across multiple base
retrieval architectures and
benchmark datasets demonstrate that SkyLink significantly boosts the ranking effectiveness of existing models, consistently achieving superior performance in various challenging scenarios.
Read the full abstract ▾
All authors Bowen Liu, Pengyue Jia, Wanyu Wang, Derong Xu, Jiawei Cheng, Jiancheng Dong, Xiao Han, Zimo Zhao, Chao Zhang, Bowen Yu, Fangyu Hong, Xiangyu Zhao
Computer Vision
ImageEdit-R1: Boosting Multi-Agent Image Editing via Reinforcement Learning
Yiran Zhao, Yaoqi Ye, Xiang Liu et al. (+2 more)
With the rapid advancement of commercial multi-modal models, image editing has garnered significant
attention due to its widespread applicability in daily life. Despite impressive progress, existing image editing systems, particularly closed-source or proprietary models, often struggle with complex, indirect, or multi-step user instructions. These limitations hinder their ability to perform nuanced, context-aware edits that align with human intent. In this work, we propose ImageEdit-R1, a multi-
agent framework for intelligent image editing that leverages reinforcement learning to coordinate high-level decision-making across a set of specialized, pretrained
vision-
language and generative
agents. Each
agent is responsible for distinct capabilities–such as understanding user intent, identifying regions of interest, selecting appropriate editing actions, and synthesizing visual content–while reinforcement learning governs their collaboration to ensure coherent and goal-directed behavior. Unlike existing approaches that rely on monolithic models or hand-crafted pipelines, our method treats image editing as a sequential decision-making problem, enabling dynamic and context-aware editing strategies. Experimental results demonstrate that ImageEdit-R1 consistently outperforms both individual closed-source
diffusion models and alternative multi-
agent framework baselines across multiple image editing
datasets.
Read the full abstract ▾
All authors Yiran Zhao, Yaoqi Ye, Xiang Liu, Michael Qizhe Shieh, Trung Bui
Computer Vision · Cross-listed from cs.RO
See and Switch: Vision-Based Branching for Interactive Robot-Skill Programming
Petr Vanc, Jan Kristof Behrens, Václav Hlaváč et al. (+1 more)
Programming by demonstration (PbD) makes
robot programming accessible to non-experts, but scaling it to real-world variability remains a challenge for current teaching frameworks, especially when a
robot must select suitable task variants online from visual input. We present See & Switch, an interactive teaching-and-execution framework that represents tasks as graphs of skill parts connected by decision states, enabling conditional branching during replay. Its
vision-based Switcher uses eye-in-hand images to select the appropriate successor skill part and detect novel situations that require new demonstrations. The framework supports recovery demonstrations during execution through kinesthetic teaching, joystick control, and hand gestures. We evaluate See & Switch on three dexterous manipulation tasks with 8 novice users, collecting approx. 900 real-
robot execution rollouts. To isolate visual decision performance from timing errors during decision states, we evaluate the Switcher offline using user-gated decision state windows. In the evaluation within the decision state windows, the method achieves up to 90.6% branch-selection accuracy and detects anomalies with >90% accuracy in 47 of 79 decision states, demonstrating reliable switching based on visual input for conditional
robot-skill programming. We provide all code and experiment data at http://imitrob.ciirc.cvut.cz/publications/seeandswitch.
Read the full abstract ▾
All authors Petr Vanc, Jan Kristof Behrens, Václav Hlaváč, Karla Stepanova
Computer Vision
Speed3R: Sparse Feed-forward 3D Reconstruction Models
Weining Ren, Xiao Tan and Kai Han
While recent feed-forward 3D reconstruction models accelerate 3D reconstruction by jointly inferring dense geometry and camera poses in a single pass, their reliance on dense
attention imposes a quadratic complexity, creating a prohibitive computational bottleneck that severely limits inference speed. To resolve this, we introduce Speed3R, an end-to-end trainable model inspired by the core principle of Structure-from-Motion: that a sparse set of keypoints is sufficient for robust pose estimation. Speed3R features a dual-branch
attention mechanism where a compression branch creates a coarse contextual prior to guide a selection branch, which performs fine-grained
attention only on the most informative image
tokens. This strategy mimics the efficiency of traditional keypoint matching, achieving a remarkable 12.4x inference speedup on 1000-view sequences, while introducing a minimal, controlled trade-off in
geometric accuracy. Validated on standard
benchmarks with both VGGT and $\pi^3$ backbones, our method delivers high-quality reconstructions at a fraction of computational cost, paving the way for efficient large-scale scene modeling.
Read the full abstract ▾
Computer Vision
Solution to the 10th ABAW Expression Recognition Challenge: A Robust Multimodal Framework with Safe Cross-Attention and Modality Dropout
Jun Yu, Naixiang Zheng, Guoyuan Wang et al. (+5 more)
Emotion recognition in real-world environments is hindered by partial occlusions, missing modalities, and severe class imbalance. To address these issues, particularly for the Affective Behavior Analysis in-the-wild (ABAW) Expression challenge, we propose a
multimodal framework that dynamically fuses visual and audio representations. Our approach uses a dual-branch
Transformer architecture featuring a safe cross-
attention mechanism and a modality dropout strategy. This design allows the network to rely on audio-based predictions when visual cues are absent. To mitigate the long-tail distribution of the Aff-Wild2
dataset, we apply focal loss optimization, combined with a sliding-window soft voting strategy to capture dynamic emotional transitions and reduce frame-level classification jitter. Experiments demonstrate that our framework effectively handles missing modalities and complex spatiotemporal dependencies, achieving an accuracy of 60.79% and an F1-score of 0.5029 on the Aff-Wild2 validation set.
Read the full abstract ▾
All authors Jun Yu, Naixiang Zheng, Guoyuan Wang, Yunxiang Zhang, Lingsi Zhu, Jiaen Liang, Wei Huang, Shengping Liu
Computer Vision
QualiTeacher: Quality-Conditioned Pseudo-Labeling for Real-World Image Restoration
Fengyang Xiao, Jingjia Feng, Peng Hu et al. (+6 more)
Real-world image restoration (RWIR) is a highly challenging task due to the absence of clean ground-truth images. Many recent methods resort to pseudo-label (PL) supervision, often within a Mean-Teacher (MT) framework. However, these methods face a critical paradox: unconditionally trusting the often imperfect, low-quality PLs forces the student model to learn undesirable artifacts, while discarding them severely limits data diversity and impairs model generalization. In this paper, we propose QualiTeacher, a novel framework that transforms pseudo-label quality from a noisy liability into a conditional supervisory signal. Instead of filtering, QualiTeacher explicitly conditions the student model on the quality of the PLs, estimated by an ensemble of complementary non-reference image quality assessment (NR-IQA) models spanning low-level distortion and semantic-level assessment. This strategy teaches the student network to learn a quality-graded restoration
manifold, enabling it to understand what constitutes different quality levels. Consequently, it can not only avoid mimicking artifacts from low-quality labels but also extrapolate to generate results of higher quality than the teacher itself. To ensure the robustness and accuracy of this quality-driven learning, we further enhance the process with a multi-augmentation scheme to diversify the PL quality spectrum, a score-based preference optimization strategy inspired by Direct Preference Optimization (DPO) to enforce a monotonically ordered quality separation, and a cropped consistency loss to prevent
adversarial over-optimization (reward hacking) of the IQA models. Experiments on standard RWIR
benchmarks demonstrate that QualiTeacher can serve as a plug-and-play strategy to improve the quality of the existing pseudo-labeling framework, establishing a new paradigm for learning from imperfect supervision. Code will be released.
Read the full abstract ▾
All authors Fengyang Xiao, Jingjia Feng, Peng Hu, Dingming Zhang, Lei Xu, Guanyi Qin, Lu Li, Chunming He, Sina Farsiu
Computer Vision
Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades
Ashkan Taghipour, Morteza Ghahremani, Zinuo Li et al. (+3 more)
Generating
videos of complex human motions such as flips, cartwheels, and martial arts remains challenging for current
video diffusion models. Text-only conditioning is temporally ambiguous for fine-grained motion control, while explicit pose-based controls, though effective, require users to provide complete skeleton sequences that are costly to produce for long and dynamic actions. We propose a two-stage cascaded framework that addresses both limitations. First, an autoregressive text-to-skeleton model generates 2D pose sequences from natural
language descriptions by predicting each joint conditioned on previously generated poses. This design captures long-range temporal dependencies and inter-joint coordination required for complex motions. Second, a pose-conditioned
video diffusion model synthesizes
videos from a reference image and the generated skeleton sequence. It employs DINO-ALF (Adaptive Layer Fusion), a multi-level reference encoder that preserves appearance and clothing details under large pose changes and self-occlusions. To address the lack of publicly available
datasets for complex human motion
video generation, we introduce a Blender-based synthetic
dataset containing 2,000
videos with diverse characters performing acrobatic and stunt-like motions. The
dataset provides full control over appearance, motion, and environment. It fills an important gap because existing
benchmarks significantly under-represent acrobatic motions while web-collected
datasets raise copyright and
privacy concerns. Experiments on our synthetic
dataset and the Motion-X Fitness
benchmark show that our text-to-skeleton model outperforms prior methods on FID, R-precision, and motion diversity. Our pose-to-
video model also achieves the best results among all compared methods on VBench metrics for temporal consistency, motion smoothness, and subject preservation.
Read the full abstract ▾
All authors Ashkan Taghipour, Morteza Ghahremani, Zinuo Li, Hamid Laga, Farid Boussaid, Mohammed Bennamoun
Computer Vision
Not Like Transformers: Drop the Beat Representation for Dance Generation with Mamba-Based Diffusion Model
Sangjune Park, Inhyeok Choi, Donghyeon Soon et al. (+2 more)
Dance is a form of human motion characterized by emotional expression and communication, playing a role in various fields such as music, virtual reality, and content creation. Existing methods for dance generation often fail to adequately capture the inherently sequential, rhythmical, and music-synchronized characteristics of dance. In this paper, we propose MambaDance, a new dance generation approach that leverages a Mamba-based
diffusion model. Mamba, well-suited to handling long and autoregressive sequences, is integrated into our two-stage
diffusion architecture, substituting off-the-shelf
Transformer. Additionally, considering the critical role of musical beats in dance choreography, we propose a Gaussian-based beat representation to explicitly guide the decoding of dance sequences. Experiments on AIST++ and FineDance
datasets for each sequence length show that our proposed method effectively generates plausible dance movements while reflecting essential characteristics, consistently from short to long dances, compared to the previous methods. Additional qualitative results and demo
videos are available at \small{https://
vision3d-lab.github.io/mambadance}.
Read the full abstract ▾
All authors Sangjune Park, Inhyeok Choi, Donghyeon Soon, Youngwoo Jeon, Kyungdon Joo
Computer Vision · Cross-listed from cs.RO
AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis
Xiaofei Wu, Yi Zhang, Yumeng Liu et al. (+3 more)
Generating human grasping poses that accurately reflect both object geometry and user-specified interaction semantics is essential for natural hand-object interactions in AR/VR and
embodied AI. However, existing semantic grasping approaches struggle with the large modality gap between 3D object representations and textual instructions, and often lack explicit spatial or semantic constraints, leading to physically invalid or semantically inconsistent grasps. In this work, we present AffordGrasp, a
diffusion-based framework that produces physically stable and semantically faithful human grasps with high precision. We first introduce a scalable annotation pipeline that automatically enriches hand-object interaction
datasets with fine-grained structured
language labels capturing interaction intent. Building upon these annotations, AffordGrasp integrates an affordance-aware latent representation of hand poses with a dual-conditioning
diffusion process, enabling the model to jointly reason over object geometry, spatial affordances, and instruction semantics. A distribution adjustment module further enforces physical contact consistency and semantic
alignment. We evaluate AffordGrasp across four instruction-augmented
benchmarks derived from HO-3D, OakInk, GRAB, and AffordPose, and observe substantial improvements over state-of-the-art methods in grasp quality, semantic accuracy, and diversity.
Read the full abstract ▾
All authors Xiaofei Wu, Yi Zhang, Yumeng Liu, Yuexin Ma, Yujiao Shi, Xuming He
Computer Vision
VSDiffusion: Taming Ill-Posed Shadow Generation via Visibility-Constrained Diffusion
Jing Li and Jing Zhang
Generating realistic cast shadows for inserted foreground objects is a crucial yet challenging problem in image composition, where maintaining
geometric consistency of shadow and object in complex scenes remains difficult due to the ill-posed nature of shadow formation. To address this issue, we propose VSDiffusion, a visibility-constrained two-stage framework designed to narrow the solution space by incorporating visibility priors. In Stage I, we predict a coarse shadow mask to localize plausible shadow generated regions. And in Stage II, conditional
diffusion is performed guided by lighting and depth cues estimated from the composite to generate accurate shadows. In VSDiffusion, we inject visibility priors through two complementary pathways. First, a visibility control branch with shadow-gated cross
attention that provides multi-scale structural guidance. Then, a learned soft prior map that reweights training loss in error-prone regions to enhance
geometric correction. Additionally, we also introduce high-frequency guided enhancement module to sharpen boundaries and improve texture interaction with the background. Experiments on widely used public DESOBAv2
dataset demonstrated that our proposed VSDiffusion can generate accurate shadow, and establishes new SOTA results across most evaluation metrics.
Read the full abstract ▾
Computer Vision
Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared
Yafei Zhang, Meng Ma, Huafeng Li et al. (+1 more)
Infrared-visible (IR-VIS) image fusion is vital for perception and security, yet most methods rely on the availability of both modalities during training and inference. When the infrared modality is absent, pixel-space generative substitutes become hard to control and inherently lack interpretability. We address missing-IR fusion by proposing a dictionary-guided, coefficient-domain framework built upon a shared convolutional dictionary. The pipeline comprises three key components: (1) Joint Shared-dictionary Representation Learning (JSRL) learns a unified and interpretable atom space shared by both IR and VIS modalities; (2) VIS-Guided IR Inference (VGII) transfers VIS coefficients to pseudo-IR coefficients in the coefficient domain and performs a one-step closed-loop refinement guided by a frozen large
language model as a weak semantic prior; and (3) Adaptive Fusion via Representation Inference (AFRI) merges VIS structures and inferred IR cues at the atom level through window
attention and convolutional mixing, followed by reconstruction with the shared dictionary. This encode-transfer-fuse-reconstruct pipeline avoids uncontrolled pixel-space generation while ensuring prior preservation within interpretable dictionary-coefficient representation. Experiments under missing-IR settings demonstrate consistent improvements in perceptual quality and downstream detection performance. To our knowledge, this represents the first framework that jointly learns a shared dictionary and performs coefficient-domain inference-fusion to tackle missing-IR fusion. The source code is publicly available at https://github.com/harukiv/DCMIF.
Read the full abstract ▾
All authors Yafei Zhang, Meng Ma, Huafeng Li, Yu Liu
Computer Vision
It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Models
Jaeha Choi, Jin Won Lee, Siwoo You et al. (+1 more)
Advances in
vision-
language models (VLMs) have achieved remarkable success on complex
multimodal reasoning tasks, leading to the assumption that they should also excel at reading analog clocks. However, contrary to this expectation, our study reveals that reading analog clocks in real-world environments remains a significant challenge for state-of-the-art VLMs. Existing analog clock
datasets are largely synthetic or planar with limited stylistic diversity and minimal background context, failing to capture the visual variability of real-world scenes. As a result, VLMs trained on such data exhibit weak spatiotemporal
reasoning, frequently confusing the hour and minute hands and struggling under common visual conditions such as occlusion, lighting variation, and cluttered backgrounds. To address this issue, we introduce TickTockVQA, a human-annotated
dataset containing analog clocks in diverse real-world scenarios. TickTockVQA provides explicit hour and minute annotations, and includes an AM/PM tag when it is inferable from the visual context. Furthermore, we propose Swap-DPO, a direct preference optimization-based fine-tuning framework to align model
reasoning toward accurate time interpretation. Experimental results demonstrate that our approach substantially enhances clock reading accuracy and robustness under real-world conditions, establishing a foundation for future research on spatiotemporal
reasoning and visual understanding in VLMs.
Read the full abstract ▾
All authors Jaeha Choi, Jin Won Lee, Siwoo You, Jangho Lee
Computer Vision
ViSA-Enhanced Aerial VLN: A Visual-Spatial Reasoning Enhanced Framework for Aerial Vision-Language Navigation
Haoyu Tong, Xiangyu Dong, Xiaoguang Ma et al. (+3 more)
Existing aerial
Vision-
Language Navigation (VLN) methods predominantly adopt a detection-and-planning pipeline, which converts open-vocabulary detections into discrete textual scene graphs. These approaches are plagued by inadequate spatial
reasoning capabilities and inherent linguistic ambiguities. To address these bottlenecks, we propose a Visual-Spatial
Reasoning (ViSA) enhanced framework for aerial VLN. Specifically, a triple-phase collaborative architecture is designed to leverage structured visual prompting, enabling
Vision-
Language Models (VLMs) to perform direct
reasoning on image planes without the need for additional training or complex intermediate representations. Comprehensive evaluations on the CityNav
benchmark demonstrate that the ViSA-enhanced VLN achieves a 70.3% improvement in success rate compared to the fully trained state-of-the-art (SOTA) method, elucidating its great potential as a backbone for aerial VLN systems.
Read the full abstract ▾
All authors Haoyu Tong, Xiangyu Dong, Xiaoguang Ma, Haoran Zhao, Yaoming Zhou, Chenghao Lin
Computer Vision
Teng Wang, Yanting Lu and Ruize Wang
We present AutoTraces, an autoregressive
vision-
language-trajectory model for
robot trajectory
forecasting in humam-populated environments, which harnesses the inherent
reasoning capabilities of large
language models (LLMs) to model complex human behaviors. In contrast to prior works that rely solely on textual representations, our key innovation lies in a novel trajectory tokenization scheme, which represents waypoints with point
tokens as categorical and positional markers while encoding waypoint numerical values as corresponding point
embeddings, seamlessly integrated into the LLM's space through a lightweight encoder-decoder architecture. This design preserves the LLM's native autoregressive generation mechanism while extending it to physical coordinate spaces, facilitates modeling of long-term interactions in trajectory data. We further introduce an automated chain-of-thought (CoT) generation mechanism that leverages a
multimodal LLM to infer spatio-temporal relationships from visual observations and trajectory data, eliminating reliance on manual annotation. Through a two-stage training strategy, our AutoTraces achieves SOTA
forecasting accuracy, particularly in long-horizon prediction, while exhibiting strong cross-scene generalization and supporting flexible-length
forecasting.
Read the full abstract ▾