Computer Vision
Extending Decoupled Attention to Dense Prediction and Masked Training for Multi-Channel Images
Umar Marikkar, Sameed Husain, Muhammad Awais et al. (+1 more)
Multi-Channel imaging (MCI) data differs fundamentally from natural images, as each channel records a semantically distinct signal rather than a colour band. To adapt
vision encoders to MCI data, Multi-Channel
Vision Transformers (MC-ViTs) tokenize each channel independently and concatenate the resulting
tokens into one sequence, and the channel count is no longer fixed by the architecture. Self-
attention is then computed across all channel-patch
tokens with no restriction on which channels attend to which, which dilutes the features of individual channels. The Decoupled
Vision Transformer (DC-ViT) regulates this by separating updates computed within a channel from updates computed across channels, and by forming a representation per channel before the channels are combined. Its formulation, however, pairs
tokens by spatial position, and thus requires the same visible
tokens in every channel. Correspondence under independent per-channel masking is recovered by solving a linear assignment between the retained patches of each channel, which allows decoupled
attention to be combined with current masked multi-channel training in its standard configuration rather than a restricted one. Across three classification and three
segmentation benchmarks spanning fluorescence microscopy, imaging mass cytometry and satellite imaging, including dense prediction at high channel counts, the resulting formulation outperforms the strongest MC-ViT baseline.
Read the full abstract ▾
All authors Umar Marikkar, Sameed Husain, Muhammad Awais, Sara Atito
Computer Vision
Detection is solved, delineation is not: what governs tooth segmentation on panoramic radiographs
Muhammad Rehan, Moaz Amjad, Syed Danial Ahmed et al. (+2 more)
Automatic tooth
segmentation and FDI numbering on panoramic radiographs underpins computer-assisted dental diagnosis, yet which factors govern performance remains unclear. We assemble a corpus of 1,422 panoramic radiographs containing 42,142 expert-delineated tooth polygons across the 32-class FDI taxonomy, annotated by 30 dental practitioners and independently reviewed by two others, and use it to isolate input resolution, architecture and anatomical priors under a single evaluation protocol. First, resolution dominates: across a controlled 640/1024/1280 ablation, mask mAP50-95 rises 0.656 -> 0.710 -> 0.717 while mAP50 stays flat at ~0.982. Both gains are significant under a paired bootstrap over images (p < 0.001, p = 0.024); neither mAP50 change is distinguishable from zero. Added resolution buys boundary precision, not detection. Second, architecture is nearly irrelevant in-domain: a query-based
transformer with 2.1x the parameters is statistically equivalent to a one-stage detector (95% CI [-0.0064, +0.0064]), only marginally better under domain shift, 5.5x slower on CPU and not executable under standard ONNX runtimes. Third, three targeted interventions fail: a LoRA-adapted self-supervised encoder underperforms, a promptable foundation segmenter degrades masks by 39%, and globally optimal anatomical label assignment yields +0.0007 despite correcting a constraint violated in 40% of out-of-domain predictions. Zero-shot transfer to an independent multi-centre cohort, verified overlap-free, costs 62% of mask mAP50-95 but only 18% of mAP50, reproducing the dissociation. Decomposing masks along the tooth axis localises the residual error to the apical third. Boundary precision is therefore the binding constraint, and effort is better directed at resolution and acquisition diversity than at architectural novelty.
Read the full abstract ▾
All authors Muhammad Rehan, Moaz Amjad, Syed Danial Ahmed, Mariam Adnan, Haider Ali
Computer Vision
Learned Parametric Emotion Editing: Real-Time Affective Filtering for On-Device Social Media Video
Musa Rochi, Marcel Schubert and Christoph Gebhardt
Problematic internet use affects a growing share of the population, yet common interventions, e.g., time limits, blocking, forced breaks, are coercive and easily circumvented. We explore a less restrictive alternative: adapting the emotional intensity of visual content. Prior work has shown that optimization can steer an image's affective content, but its per-image optimization cost makes it impractical for real-time deployment. We instead learn a model that predicts this transformation in a single forward pass: a MobileNetV4 backbone with FiLM-based
emotion conditioning outputs parameters for differentiable global transformations. This replaces prior iterative optimization (80 s per image) with a single 3.7 ms forward pass. In a user study (N = 54), the model reduced viewer-reported arousal relative to unedited images, comparably to the grayscale well-being filter, while being rated higher in perceived quality. We integrate the model into an Android app that adapts Instagram
video in real time, sustaining 60 fps on a Samsung
Galaxy S23.
Read the full abstract ▾
Computer Vision
HAT: Hypothesis-Anchored Tracking for Video Monocular Spacecraft Pose Estimation
André Lopo, Atabak Dehban and Rodrigo Ventura
Monocular 6-DoF pose estimation of non-cooperative targets is important for on-orbit servicing and debris removal. A single-image
estimator can confuse near-symmetric spacecraft orientations, and tracking can preserve an incorrect pose. We present Hypothesis-Anchored Tracking (HAT), a causal framework that uses inter-frame motion to select among competing CAD-based pose hypotheses before
alignment and fusion. Rather than independently choosing the highest-scoring hypothesis in each image, HAT retains competing orientation histories and selects a pose to anchor the relative trajectory estimated by
monocular SLAM. Sparse anchors and pose fusion provide per-frame estimates after initialization without revising past outputs. The method requires only a calibrated RGB sequence, a metric CAD model, and target image regions, which can be supplied by detection or
segmentation. The pretrained pose and SLAM networks require no target-specific training or fine-tuning. We evaluate two versions, Mega-HAT and Pico-HAT, using MegaPose and PicoPose, on SPARK-2024, SwissCube and SHIRT, with YCB-
Video assessing performance outside the space domain. Using one temporal configuration per method, the arithmetic means of the four
dataset-wise comparisons show 9.4% lower mean pose error and 3.76 times the sustained input FPS for Mega-HAT relative to independent MegaPose, and 23.9% lower mean pose error and 2.42 times the FPS for Pico-HAT relative to independent PicoPose. Mega-HAT ablations on SPARK and an offline reference examine component contributions and the effect of revising past estimates.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Evaluating In-Context Learning and Retrieval Strategies for Devanagari Post-OCR Correction
Abhishek Bhandari and Gaurav Harit
In-context learning using Large
Language Models (LLMs) offers a compelling path to training-free post-OCR correction, yet its effectiveness for Devanagari script remains entirely unexplored. We present the first systematic evaluation of LLMs (3B-32B) for post-OCR correction in Hindi and Marathi, comparing three in-context example
retrieval strategies: domain-random selection, dense semantic
retrieval, and our proposed CharBM25, which retrieves examples by character n-gram BM25 similarity over OCR inputs to target shared error patterns with the test sentence. Across a 20,000-sentence
benchmark spanning five news domains,
retrieval strategy is the decisive factor in correction quality: CharBM25 outperforms domain-random selection by 2.8-4.0pp absolute WER on Hindi and 2.9-3.8pp on Marathi, using character trigrams, which consistently outperform bigrams and unigrams. Scale dominates performance: Gemma-3-27B achieves WER reductions of 55.0% for Hindi and 33.3% for Marathi under CharBM25-5. Few-shot gains are capacity-gated: models below 8B do not reliably improve over the OCR baseline, and on Marathi the smallest models (3B) degrade more sentences than they improve. Marathi is persistently harder to correct than Hindi across all scales, reflecting its greater morphological complexity. These findings establish CharBM25 as an effective, GPU-free
retrieval strategy that matches or exceeds dense
retrieval at negligible computational cost, and show that combining it with a general-purpose LLM of 12B+ parameters delivers reliable, training-free Devanagari post-OCR correction without task-specific fine-tuning.
Dataset: https://huggingface.co/
datasets/AbhishekBhandari/Devanagari-OCR-ICL-
Benchmark
Read the full abstract ▾
Computer Vision
A benchmark dataset and baseline methods for four-dimensional STEM diffraction patterns
Yuyan Guan, Haoran Zhang, Zian Mao et al. (+7 more)
Four-dimensional scanning transmission electron microscopy (4D-STEM) records a two-dimensional diffraction pattern at each electron-probe position, yielding spatially resolved reciprocal-space information but large, heterogeneous data volumes. Here we describe 4D-ImageNet, a collection of 174,000 diffraction patterns comprising 145,000 experimental patterns selected from 29 acquisitions and 29,000 multislice simulations. The experimental data cover acquisition-level labels for Ag, Au, mixed Au-Ag, CoO, Pd and ZnO specimens across multiple fields of view, scan dimensions, camera lengths and exposure times. Each acquisition contributes 5,000 quality-ranked patterns with source scan coordinates and acquisition metadata. A set-prediction detector provides model-derived pseudo-labels for the direct-beam position and Bragg-disk centres, with a confidence score for each disk. The simulation data cover 13
crystal structures and include Euler rotations, reciprocal-space sampling and approximate low-index beam directions. A grouped mixed-domain masked-reconstruction
benchmark is provided to assess leakage-resistant loading and evaluation across experimental and simulated data. The
dataset is intended for representation learning, disk detection, diffraction-pattern
retrieval, orientation analysis and simulation-to-experiment studies.
Read the full abstract ▾
All authors Yuyan Guan, Haoran Zhang, Zian Mao, Antong Yang, Caifei Li, Jialong Wang, Chuying Ouyang, Hong Wang, Xiaoqin Zeng, Yujun Xie
Computer Vision
GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
Pinxin Liu, Haiyang Liu, Jiahao Luo et al. (+3 more)
Generating natural co-
speech gestures from streaming
speech is essential for
embodied conversational
agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion
tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-
speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a
transformer to model streaming audio-motion context and a per-
token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow
distillation strategy that freezes the causal backbone and distills the multi-step per-
token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model
token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality–latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time
token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR
Read the full abstract ▾
All authors Pinxin Liu, Haiyang Liu, Jiahao Luo, Junhua Huang, Chunhao Zou, Luchuan Song
Computer Vision
From Retrieval to Recognition:How Vision–Language Models Become OCR Specialists
Yuanxiang Huangfu, Hanmeng Zhong, Linqing Chen et al. (+1 more)
Does a general
vision–
language model acquire specialized OCR ability by developing a new reading
circuit or by reusing an existing mechanism? We address this question in the setting of full-sequence OCR, rather than local-answer
retrieval. Using an evidence-grounded protocol with held-out causal interventions, we identify sparse and stable OCR-head sets in GLM-OCR, MinerU2.5, and PaddleOCR-VL-1.6. We then investigate the mechanistic origin of these OCR heads by comparing them with independently identified textual
retrieval/copy heads in general VLMs. Across two general VLMs, visual OCR heads strongly overlap independently identified textual
retrieval/copy heads, yielding untuned top-20 intersections of 73.3% and all-head Spearman correlations of 0.677-0.886. The overlap and causal interventions suggest that full-sequence OCR operates as dense sequential
multimodal copy-and-paste, repeatedly retrieving visual evidence and routing it to the current output position. Finally, we examine how this shared
circuit changes as a general VLM becomes an OCR specialist. Matched base-to-specialized comparisons show that OCR specialization largely preserves head identity, retaining 17-20 of the top 20 heads per task with all-head rank correlations of 0.874-0.942, while redistributing their functional and causal strengths.
Read the full abstract ▾
All authors Yuanxiang Huangfu, Hanmeng Zhong, Linqing Chen, Jeffrey Tiong Jee Hui
Computer Vision
Purification and Regulation: Comorbidity-Aware Multi-Label Few-Shot Learning for Medical Image Classification
Ying-Chih Lin, Po-Chih Kuo and Yong-Sheng Chen
Multi-label few-shot learning (MLFSL) remains a significant challenge in medical image analysis (MIA). Current metric-based meta-learning methods face two critical limitations in MIA. First, conventional prototype generation often entangles irrelevant disease information, leading to contaminated prototypes and degraded performance. Second, prior studies typically enforce inter-class separability in
embedding space, largely neglecting the inherent correlations among diseases. To overcome these challenges, we propose Prototype Purification and Regulation (PPR), a novel MLFSL framework for MIA. PPR first performs prototype purification by leveraging sample-level comorbidity scores to emphasize disease-specific features, producing purified prototypes that better characterize each disease. Building upon these purified prototypes, PPR further addresses the underexplored problem of inter-class prototype distance in MIA by incorporating disease-level comorbidity statistics to adaptively regulate inter-class similarity, forming a comorbidity-aware
embedding space. Overall, PPR sequentially enables the model to capture pure disease features and inter-class relationships for reliable MLFSL in MIA. Extensive experiments across four chest X-ray
benchmark datasets, including cross-domain evaluation, show that PPR consistently outperforms state-of-the-art methods, significantly improving disease detection while demonstrating robust generalization and clinical applicability.
Read the full abstract ▾
Computer Vision
Refine Then Fusion: Training-Free 3D Point Cloud Adaptation with Priority Refinement and Multi-Modal Knowledge Fusion
Hang Cheng, Yan Chen, Mingyu Fan et al. (+1 more)
Recent pre-trained foundation models provide rich multi-modal priors for downstream 3D
vision tasks. However, the effectiveness of these representations in few-shot scenarios is limited by two fundamental challenges: High-dimensional features often contain substantial channel redundancy and task-irrelevant noise, while the reliability of different modalities varies across samples. Consequently, direct aggregation of heterogeneous representations overlooks sample-dependent modality reliability and may obscure the discriminative cues essential. To address these limitations, we propose Refine Then Fusion(RTF), a training-free framework for few-shot 3D recognition. RTF first identifies discriminative feature channels by jointly modeling inter-class similarity and intra-class stability, thereby decoupling domain-specific knowledge refinement from the cached representations of pre-trained models. It then introduces a reliability-aware fusion mechanism that estimates sample-wise modality reliability from the distribution shifts induced by feature refinement, enabling adaptive aggregation of multi-modal representations. Furthermore, RTF constructs a
memory cache that integrates instance-level support features with class-level prototypes to infer query labels. Extensive experiments on five
benchmarks demonstrate that RTF consistently outperforms single-modal baselines, partial-fusion variants, and existing lightweight adaptation methods, achieving state-of-the-art few-shot 3D recognition performance without gradient optimization, additional training data, auxiliary training, or parameter updates.
Read the full abstract ▾
All authors Hang Cheng, Yan Chen, Mingyu Fan, Long Zeng
Computer Vision
VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration
Changbeen Kim, Junwon Chang, Kipyo Kim et al. (+3 more)
While
Video Large
Language Models (
Video-LLMs) have recently demonstrated strong performance, reliably evaluating their fine-grained
video understanding remains challenging. Existing
benchmarks often rely on question answering or ground-truth caption matching, where models may succeed through superficial cues and incomplete annotations. To this end, we introduce VidOmni-Bench, a
benchmark that requires models to verify whether each event in dense
video captions is supported by the
video. VidOmni-Bench consists of 500
videos spanning five complexity types and diverse durations from 4 seconds to 90 minutes. After collecting
videos along these axes, we use diverse
Video-LLMs to generate dense captions and obtain human-verified sentence-level labels, where sentences containing incorrect events serve as hard negatives for evaluation. Our experiments on VidOmni-Bench reveal three key findings: (i)
Video-LLMs frequently generate hallucinated descriptions in dense
video captioning; (ii) they also struggle as verifiers, failing to reliably detect plausible but incorrect event descriptions; and (iii) model weaknesses vary across
video complexity and duration, revealing diverse, model-specific bottlenecks in current
Video-LLMs.
Read the full abstract ▾
All authors Changbeen Kim, Junwon Chang, Kipyo Kim, Risa Shinoda, Kuniaki Saito, Donghyun Kim
Computer Vision
2D GauSS-MI: Efficient Active Scene Reconstruction with Balanced Visual and Geometric Quality
Yuhan Xie and Jia Pan
Active reconstruction requires efficient active view selection to achieve high-quality reconstruction within limited onboard computational resources. Existing methods face challenges in adequately balancing visual and
geometric quality with the computational efficiency required for real-time operation. In this work, we present an active reconstruction framework based on 2D Gaussian Splatting (2DGS). We develop an efficient online 2DGS mapping pipeline for incremental RGB-D observations and introduce a probabilistic reliability model that characterizes the view-dependent reconstruction quality of individual 2D Gaussian splats. Building on this model, we formulate 2D Gaussian Splatting Shannon Mutual Information (2D GauSS-MI), a mutual-information-based metric that exploits the explicit surface orientation of 2DGS to evaluate the expected information gain of candidate views. The proposed metric enables active view selection to account for both visual and
geometric reconstruction quality. We evaluate the proposed system against three state-of-the-art baselines on eight Replica scenes. Experimental results demonstrate that our method achieves a favorable balance between visual and
geometric reconstruction quality with substantially lower computational cost and competitive model storage.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
2nd Place Solution to the HANDS 2026 Workshop Challenge-Dexterous Grasp Motion Track: Single-Shot Trajectory Warping for Grasp Motion Generation
Muneeb A. Khan, Woojin Kim, Shinwoo Kim et al. (+3 more)
This report describes our 2nd place solution to the HANDS 2026 workshop challenge (Dexterous Grasp Motion track) in conjunction with ECCV 2026. In this challenge, we address grasp motion generation for the 12-DoF LinkerHand O6, aiming to produce physically plausible reach-and-lift trajectories for unseen objects from randomized initial hand poses in simulation. This task is particularly challenging because each grasp requires a per-step policy to make approximately $70$ twelve-dimensional decisions, with errors accumulating over time, while test objects and physical dynamics may differ from those encountered during training. To address these challenges, we propose editing a single successful GraspM3 demonstration instead of generating the motion step by step: a policy observes the object once and outputs a 12-D warp of the demonstration, which is then replayed open-loop. Moreover, we train the warp policy with one-step PPO over all $4{,}824$ training objects in parallel. As a result, our method achieved success rates of $94.61\%$ on the easy track, the highest of all submissions, and $57.18\%$ on the hard track of the private test set.
Read the full abstract ▾
All authors Muneeb A. Khan, Woojin Kim, Shinwoo Kim, Muhammad Munsif, Binod Bhattarai, Seungryul Baek
Computer Vision
Adaptive World Memory 3D Foundation Model for Scalable 3D Mapping, Localization, and Rendering
Tianchen Deng, Guole Shen, Yilin Shen et al. (+7 more)
Recent 3D foundation models enable generalizable
geometric reasoning from RGB images but remain limited in persistent
memory, scalability, and renderable scene modeling. We present a
memory-centric 3D foundation model for scalable robotic localization, reconstruction, and Gaussian rendering. Its core is an adaptive world
memory mechanism that combines
transformer-based gated updates with test-time temporal-spatial regulation. Learned gates control recurrent
memory propagation, while temporal state
evolution and spatial observation-state consistency regulate
token-wise updates and forgetting over long image sequences. To support large-scale mapping, we organize
memory into local submaps and integrate progressive mapping and tracking, loop closure, and SL(4)-based global refinement to maintain local accuracy and global consistency. A Gaussian reconstruction head decodes
memory-enhanced features into renderable primitives, unifying camera pose estimation, dense point-cloud reconstruction, and photorealistic rendering within a single model. Experiments on public
benchmarks and self-collected
datasets from diverse robotic platforms demonstrate improved trajectory accuracy, reconstruction completeness, and rendering quality over existing 3D foundation reconstruction and SLAM baselines. These results support adaptive
memory as a foundation for persistent robotic world modeling. The
dataset and code will be made publicly available at \href{https://github.com/dtc111111/AWM-3DFM}{https://github.com/dtc111111/AWM-3DFM}.
Read the full abstract ▾
All authors Tianchen Deng, Guole Shen, Yilin Shen, Wenhua Wu, Yilin Fang, Ziqi Ma, Tianjun Zhang, Shenghai Yuan, Wolfram Burgard, Hesheng Wang
Computer Vision
VoxelTTO: Voxel-Aligned Feed-Forward 3D Gaussian Splatting with Test-Time Optimization
Yibin Zhao, Yihan Pan, Yangwen Li et al. (+2 more)
Recent feed-forward 3D Gaussian Splatting (3DGS) methods typically regress pixel-aligned Gaussian primitives, often causing excessive overlap and artifacts, while inaccuracies in predicted camera poses can lead to misalignment in novel-view synthesis (NVS). We present VoxelTTO, a feed-forward framework for reconstructing geometrically accurate 3DGS scenes from an arbitrary number of images and optional camera parameters. VoxelTTO aggregates dense image features into a global voxel representation and decodes Gaussians from voxel features, breaking the pixel-to-Gaussian correspondence. To exploit known camera parameters while keeping the pretrained visual foundation model (VFM) parameters frozen, we introduce test-time optimization (TTO) that adapts lightweight LoRA modules using pose supervision. We further replace vanilla 3DGS rasterization with
stochastic solid volume rendering during training and inference, improving
geometric fidelity. Training updates only the voxel-aligned Gaussian reconstruction modules, requiring 80 GPU hours. Experiments on Replica, Tanks and Temples, and DTU demonstrate improved RGB-D NVS and camera-pose estimation relative to prior methods.
Read the full abstract ▾
All authors Yibin Zhao, Yihan Pan, Yangwen Li, Jun Nan, Jianjun Yi
Computer Vision
OpenSAL360: Open-Source Crowdsourcing Platform for Omnidirectional Video Saliency Collection
Alexey Bryncev, Andrey Moskalenko, Kira Shilovskaya et al. (+2 more)
Omnidirectional
video saliency prediction plays an important role in many immersive multimedia applications, including viewport-adaptive streaming and compression, foveated rendering, mesh simplification, perceptual quality assessment. Yet progress in this area remains constrained by the cost and complexity of collecting eye-tracking data with VR headsets, which makes large-scale
dataset creation difficult to extend. We present OpenSAL360, the first open-source platform for scalable, low-cost 360{\deg}
video saliency collection. Unlike conventional VR-based protocols, it requires only a standard screen, mouse, and internet connection, enabling parallel saliency data collection from common crowdsourcing assessors without specialized hardware. We validate our collection protocol against seven well-established VR eye-tracking
datasets and conduct ablation studies on key interface, pre-, and post-processing parameters. To demonstrate the effectiveness and scalability of the proposed methodology, we collect and publicly release a saliency
dataset covering 500 omnidirectional
videos annotated by 2,000+ crowdsourcing assessors, making it, to the best of our knowledge, the largest
dataset in this field. We make OpenSAL360 publicly available at https://github.com/msu-
video-group/OpenSAL360.
Read the full abstract ▾
All authors Alexey Bryncev, Andrey Moskalenko, Kira Shilovskaya, Ivan Kosmynin, Dmitriy Vatolin
Computer Vision
MT-WAM: Reorienting the One-Pass Predictive Representation Toward Action Generation
Yiguang Yang, Jiankun Peng, Xiaoming Wang et al. (+2 more)
Fast-WAM shows that
video-action co-training improves control without generating future
video at inference, making the representation from a single
video diffusion Transformer forward central to action generation. However, future-observation prediction does not explicitly prioritize the future dynamics and visual structure needed for control. We present MT-WAM, which retains the original training objectives and adds complementary supervision for future two-dimensional point trajectories and visual features. A lightweight dual-stream branch copied from the
video backbone's final blocks provides target-specific processing, while a structured
attention mask prevents cross-stream
attention. Motion-stream
tokens supply additional dynamics conditions to the action expert. Future visual-feature prediction provides supervision in a feature space that captures object and spatial structure. This supervision trains the
video backbone to provide more informative visual context for action generation under changing visual conditions, without adding visual-feature-stream
tokens to action conditioning. At inference, MT-WAM uses
video and motion caches computed once per replan and skips future-
video prediction. Without additional
embodied policy pretraining, MT-WAM achieves 98.2% success on LIBERO and 73.7% on LIBERO-Plus, exceeding Fast-WAM by 23.8 percentage points on the latter. On RoboTwin 2.0 Clean2Rand, Random success increases from 6.30% to 19.40%; across four real-world tasks, average success increases from 67.0% to 77.8%.
Read the full abstract ▾
All authors Yiguang Yang, Jiankun Peng, Xiaoming Wang, Yiran Zhang, Zhibo Fang
Computer Vision
SkillIR: Evolving Scene-Aware Skills for Agentic Image Restoration
Jie Shao, Shengkai Hu, Xu Zhang et al. (+4 more)
This paper studies agentic image restoration, in which
multimodal agents coordinate specialized restoration tools to recover images affected by complex degradations. Existing restoration
agents often derive complete tool-use plans from the original degraded image or retrieve previously successful trajectories, providing limited support for adapting individual actions to evolving intermediate restoration states. We find that accepted tool executions can change the residual degradation state and, consequently, the applicability of subsequent tools. To address this issue, we propose SkillIR, a skill-guided framework that represents restoration experience as degradation-centered action evidence rather than complete tool-use trajectories. SkillIR consolidates context-dependent action outcomes into scene-aware restoration skills that characterize applicable conditions, expected effects, and attributable failure cases. Instead of prescribing a complete restoration plan, the retrieved skills guide one bounded action at a time within a verified residual-state loop: each tool output is treated as a candidate, committed only after transition verification, and followed by reassessment of the active residual degradations. After each rollout, the resulting evidence is used to create, refine, or patch dynamic skills, enabling accumulated restoration experience to improve decision-making for subsequent inputs. Experiments on synthetic and real-world multi-degradation
datasets demonstrate that SkillIR improves restoration quality and enables more reliable and effective tool use.
Read the full abstract ▾
All authors Jie Shao, Shengkai Hu, Xu Zhang, Beihang Song, Yongcheng Jing, Xu Wu, Jun Wan
Computer Vision
PSEE: Progressive Sensor Event Expansion for Point-Supervised Temporal Action Localization
Jiaxi Yin, Ge Wang, Han Ding et al. (+1 more)
Temporal action localization (TAL) in wearable sensor streams identifies action classes and temporal boundaries, enabling finer-grained activity understanding than conventional action recognition. However, training typically requires costly start–end annotations for every action instance. To reduce this burden, we study point-supervised TAL, where each instance is labeled with only one timestamp and its class. We propose Progressive Sensor Event Expansion (PSEE), which combines semantic activations, sensor-specific transition evidence, and adaptive temporal ownership to recover point-supervised pseudo segments. These segments supervise standard TAL detectors without modifying their inference procedures. Cross-subject experiments on four inertial-sensing
benchmarks demonstrate improved pseudo-boundary quality over adapted point-supervised baselines, compatibility with different TAL detectors, and robustness to point sampling. Code is available at https://github.com/joeeeeyin/PSEE.
Read the full abstract ▾
All authors Jiaxi Yin, Ge Wang, Han Ding, Fei Wang
Computer Vision
CompAdapt: Adaptable Composite Motion Modeling for Physics-Consistent Text-to-Video Generation
Haoran Qin, Renlong Wu, Tianyu Huang et al. (+3 more)
While
diffusion-based text-to-
video (T2V) models have demonstrated impressive capability in generating realistic and temporally coherent
videos, they often fail to respect fundamental physical dynamics. Although recent physics-constrained methods incorporate explicit dynamics priors to improve physical plausibility, they remain limited to simple single-type motions, depend on manually specified parameters, and struggle to generalize to unseen physical laws. In this work, we propose CompAdapt, a physics-consistent T2V framework for adaptable generation across complex real-world scenarios. It extends neural dynamics modeling beyond single-type motions to encompass composite physical behaviors, including coupled motions, multi-stage transitions, and multi-object collisions. Furthermore, CompAdapt translates natural
language prompts into structured physical semantics, enabling end-to-end specification of motion types, temporal relations, and initial physical parameters. To generalize to novel physical environments, CompAdapt introduces dynamics-aware prior matching, achieving one-shot adaptation without retraining the core dynamics module. In addition, a physics-aware latent feature fusion module improves visual fidelity under fast and complex motion. Experiments on physics-focused T2V
benchmarks demonstrate that CompAdapt improves physical consistency over both general T2V models and physics-constrained baselines, while preserving high visual quality and adaptability to unseen dynamics. The project page is available at https://makapic.github.io/CompAdapt/ .
Read the full abstract ▾
All authors Haoran Qin, Renlong Wu, Tianyu Huang, Yukang Ding, Hui Li, Wangmeng Zuo
Computer Vision
ME-Dex 1.0: Bringing Heterogeneous Tactile Sensing into World Action Modeling
Xuancheng Zhang, Xuetao Liu, Qianying Tang et al. (+8 more)
World Action Models bring the predictive capabilities of
video models into
robot action generation, providing a rich foundation for modeling future visual states. Tactile sensing complements this foundation with direct measurements of physical interaction. Some existing methods use tactile features as conditioning inputs without jointly predicting future tactile states, visual observations, and actions. Our key insight is that tactile signals, like
video, provide observations of the evolving world state and should be modeled as future observations alongside
video. We present ME-Dex-1.0 (MachEmbodied-Dex-1.0), a unified World Action Tactile Model for joint visual, tactile, and action learning. ME-Dex-1.0 adopts a Mixture-of-
Transformers architecture comprising a
Video Expert, a Tactile Expert, and an Action Expert, all trained with flow matching. We use shared
attention connects the experts in intermediate layers, allowing action generation to draw on learned representations of visual and tactile dynamics during joint denoising. To support multi-source heterogeneous tactile inputs, a Canonical Hand Model and a Unified Tactile Autoencoder map tactile observations from different embodiments and sensing layouts into shared spatial and latent spaces. To address the limited availability of paired visual, tactile, and action data, we develop the Agentic Tactile Data Engine, an
agent-based data production platform. It supplements RoboTwin and DexJoCo with tactile data recorded directly from force sensors during trajectory replay in simulation. Experiments on the RoboTwin, DexJoCo, and ManiFeel simulation platforms, together with real
robot evaluations, demonstrate improved manipulation performance using both grippers and dexterous hands equipped with tactile sensing.
Read the full abstract ▾
All authors Xuancheng Zhang, Xuetao Liu, Qianying Tang, Jizhe Wang, Runsheng Wang, Zhijing Cheng, Bochen Lin, Haoran Wen, Ming Li, Kun Zhan, Yu Liu
Computer Vision
Think Locally, Refine Globally for Memory-Efficient 3D Reconstruction
Jingke Zhou, Chenhang Ma, Zhizhou Zhong et al. (+6 more)
We propose LoG-VGGT, a
memory-efficient framework for long-sequence 3D reconstruction that balances local temporal modeling with global camera consistency. Instead of relying on full global
attention, our method introduces cross-window
attention at a small subset of
transformer blocks, enabling effective information propagation across adjacent temporal windows while keeping
memory usage bounded. To mitigate long-term pose drift, we further design a global camera consistency refinement module, where camera
tokens interact with compact register
tokens via cross-
attention to enforce scene-level constraints across the entire sequence. This design enables joint optimization of camera representations and significantly improves long-horizon pose stability without incurring the high cost of sequence-wide
attention. Extensive experiments demonstrate that LoG-VGGT achieves improved depth accuracy and robust camera pose estimation across multiple long-sequence
benchmarks, while delivering competitive streaming reconstruction performance.
Read the full abstract ▾
All authors Jingke Zhou, Chenhang Ma, Zhizhou Zhong, Mingkai Liu, Zhuang Zhou, Yicheng ji, Binghua Su, Bo Cai, Xianliang Huang
Computer Vision
P$^3$-SAM: SAM with Perceptual Parallel Prompt for Few-Shot Strip Steel Surface Defect Segmentation
Qian Xu, Hang Xiong, Anpeng Wang et al. (+3 more)
Few-shot semantic
segmentation (FSS) of strip steel surface defects (S$^3$D) has posed significant challenges distinct from natural scenes. Unlike natural images, S$^3$D task exhibits unique characteristics including low local contrast, uneven illumination, and complex fine-grained texture patterns. Although recent methods based on Segment Anything Model (SAM) have shown promise in FSS on natural images by leveraging SAM's powerful pre-trained representations, these unique industrial characteristics of S$^3$D images lead to performance drop when directly applying SAM to industrial defect scenarios. In this paper, we propose a novel Perceptual Parallel
Prompt (P$^3$) framework that empowers SAM, creating the P$^3$-SAM model to address these challenges through two core strategies. First, we develop a Perceptual-Optimized Encoding (POE) strategy that enhances local contrast and preserves critical texture details for S$^3$D
segmentation. Second, we introduce the Parallel
Prompt Generator (PPG) strategy that simultaneously generates both semantic and spatial
prompts, enabling comprehensive guidance for SAM's decoder across varying images. Extensive experiments on three few-shot S$^3$D
benchmarks demonstrate that P$^3$-SAM achieves state-of-the-art performance, with particularly notable improvements of 12.00% in mIoU on Surface Defects-4i
dataset.
Read the full abstract ▾
All authors Qian Xu, Hang Xiong, Anpeng Wang, Sam Kwong, Cong Zhang, Runmin Cong
Computer Vision
When Online Adaptation Hurts: Parameter-Frozen Test-Time Ensembling for Continual Medical Image Segmentation
Ruijie Huang
Medical image segmenters often get worse when sites, scanner vendors, or protocols change. Continual test-time adaptation (CTTA) addresses this problem without target labels, but it can be impossible to update a model on a non-stationary stream and can lead to a lot of errors. We examine a more reasonable and meaningful alternative: parameter-frozen inference enhancement(PIE). We use a source-trained segmenter that learns about anatomy-preserving scale and flip views, maps their predictions back to the native location, and averages the probabilities. We do not modify the weights of the model or the normalization statistics. On a cardiac MRI stream from M&Ms, which is trained on vendor A and evaluated sequentially on vendors B, C, and D, PIE has 0.7786 mean Dice, compared to 0.7680 for source-only inference and 0.7388–0.7416 for five other online-adaptation baselines. The controlled ablations show that performance saturates at 28 views, and confidence weighting, class-prior correction, connected-component filtering, morphological refinement, and inter-slice smoothing have no effect or cause negative transfer. Qualitative results on cardiac MRI and fundus images are also consistent with the frozen ensemble keeping thinner and nested anatomical structures. These results provide a strong, stable baseline for medical CTTA and expose an important failure mode: adaptation and handcrafted refinement can be less reliable than carefully designed inference.
Read the full abstract ▾
Computer Vision
Quantization-Aware Kalman Estimation for Diffusion Sampling
Qitan Shi, Cheng Jin, Jiawei Zhang et al. (+1 more)
Quantization offers a practical path to deploying
diffusion models with reduced
memory and computation, but aggressive compression can cause quantized outputs to deviate substantially from their full-precision counterparts. Sampling-stage correction methods seek to compensate for such deviations during sampling, but existing approaches rely primarily on local information and underexploit trajectory history, limiting their ability to correct errors that propagate across timesteps. In this work, we formulate sampling with a quantized denoiser as an online estimation problem, using the history of quantized denoiser outputs to recover the underlying full-precision outputs required by the sampler. We propose QuAKE, a Quantization-Aware Kalman
Estimator that combines a smooth trajectory prior with a conditional Gaussian observation model. At each sampling step, QuAKE recursively updates the posterior over the output window in closed form and feeds its posterior mean to the sampler. QuAKE is a lightweight plug-and-play corrector that requires no modification to the quantized network and naturally supports arbitrary high-order multistep ODE samplers. Experiments across W4A4-quantized text-to-image
diffusion models show that QuAKE consistently outperforms existing methods in reducing the distributional discrepancy from full-precision sampling.
Read the full abstract ▾
All authors Qitan Shi, Cheng Jin, Jiawei Zhang, Yuantao Gu
Computer Vision
Zhibo Zhang, Qijie Wang and Zengqiang Yan
Surgical instrument
segmentation (SIS) plays a critical role in robotic assistance and
surgical workflow analysis. However, most existing SIS methods formulate
segmentation as a category-driven localization problem, limiting their ability to capture procedural context and task-dependent semantics in
surgical workflows. We introduce
Reasoning-Aware
Surgical Instrument
Segmentation (RA-SIS), a task formulation that frames
segmentation as query-conditioned inference under
surgical context. To
benchmark this setting, we construct SurgRS, a
surgical reasoning segmentation dataset consisting of 41,000 image-text pairs, which aligns instance-level masks with structured query-answer supervision to enable semantic grounding at the pixel level. Based on SurgRS, we propose
Surgical Instrument
Reasoning and
Segmentation Assistant (SIRA), a
multimodal framework that disentangles target-level and query-level semantics and integrates them with visual features through query-anchored dual
alignment. By aligning query semantics with spatial features and
segmentation prompts, SIRA enhances semantic-visual consistency in mask prediction. Extensive experiments on SurgRS demonstrate improvements over existing
reasoning-aware baselines. Code is available at https://github.com/linxir226/SIRA.
Read the full abstract ▾
Computer Vision
A Scene Language Model for Open-Vocabulary Scene Mapping
Adam Lilja, Fabio Hübel, Siming He et al. (+6 more)
Open-vocabulary 3D scene mapping aims to build a persistent representation of the objects in an environment. Existing systems typically rely on engineered mapping pipelines to associate observations, merge information across views, and maintain a consistent scene representation over time. Many additionally store feature-rich object representations, such as
embeddings or image crops, increasing the size and complexity of the persistent
memory. We introduce SceneLM, a Scene-
Language Model that directly maintains a textual scene map. The full scene is represented as a structured text list of objects, which serves as the model's only persistent
memory. For each input image, the model reads the current scene state and updates the map by adding, editing, and removing objects. To learn this behavior, we introduce supervision tasks for iterative scene map maintenance together with an automatic annotation pipeline that generates training data from images without human labels. We evaluate SceneLM on both a
language-grounded
retrieval benchmark and a localization
benchmark. Across both
benchmarks, the model produces a scene map that achieves competitive performance with complete mapping systems built from dedicated perception and
geometric modules while producing a scene representation that is 6-12x more compact. We further show that SceneLM can be run online on an edge device through experiments on a quadruped. These results show that a persistent open-vocabulary 3D scene map can be maintained directly by a single
vision-
language model using only a lightweight text representation. Training and inference code is available on https://goldengait.github.io/scenelm/.
Read the full abstract ▾
All authors Adam Lilja, Fabio Hübel, Siming He, Junsheng Fu, Claire Tomlin, Lars Hammarstrand, Jitendra Malik, Jonas Frey, Marco Pavone
Computer Vision · Cross-listed from cs.CL
Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction
Qi Chen, Yunfei Chu, Haolin He et al. (+15 more)
Natural audio-visual interaction is emerging as an important interface for AI assistants, allowing users to communicate through
speech and
vision rather than carefully composed text
prompts. However, existing
benchmarks of interactive capabilities still focus primarily on response quality, leaving a more fundamental question underexplored: can a model correctly infer the user's underlying demand from complex
multimodal interaction? Real-world user demands are often underspecified in
speech and must be inferred from
multimodal cues and dialogue history. This inference is further complicated by ambiguous or disfluent expression and noisy acoustic environments. Conversely, request-like
speech may not constitute a demand to the assistant, leading to false triggers. We establish Omni Demand Understanding (ODU) as a distinct
multimodal contextual inference problem: given an interaction stream, a model must detect whether a user demand is present and infer intent from
multimodal and conversational context. ODU evaluates this capability along five dimensions, covering both single-turn and multi-turn interactions. We construct ODU-Bench using a challenge-driven taxonomy, taxonomy-guided agentic
video generation, and human-recorded interactions, followed by media-grounded annotation and human verification. We evaluate 14 native MLLMs. Even the strongest, Gemini 3.1 Pro, recovers only 44.7% of key information that must be inferred from visual, acoustic, or conversational context. Moreover, 11 of the 14 models exhibit false-trigger rates above 50% on non-demand scenarios. These results reveal a systematic capability gap in current MLLMs' ability to infer contextual user demands. We hope ODU can establish the evaluation of a previously underexplored yet essential capability in
multimodal interaction: correctly understanding user demands before generating an appropriate response.
Read the full abstract ▾
All authors Qi Chen, Yunfei Chu, Haolin He, Yifan Yang, Zihan Liu, Yuxuan Wang, Ziyang Ma, Ruiyang Xu, Meng Gao, Yinsong Yan, Ling Wang, Hui Wang, Wen Huang, Yiheng Chen, Guanrou Yang, Qiuqiang Kong, Jin Xu, Xie Chen
Computer Vision · Cross-listed from eess.IV
WS-NeRF: A Mamba-Driven World-State-Aware Adaptive Deblurring Neural Radiance Field
Hang Jiang, Jinghao Wang, Yiming Zhang et al. (+3 more)
Neural Radiance Fields (NeRF) have attracted extensive
attention in recent years due to their strong capability for high-quality 3D reconstruction and novel view synthesis from multi-view images. Existing methods usually rely on high-quality sharp inputs, while real-world image acquisition is highly susceptible to blur degradation, which severely affects the reconstruction quality of NeRF. In this paper, we propose a novel Mamba-driven world-state-aware adaptive deblurring neural radiance field, termed WS-NeRF, to address image degradation and 3D inconsistency. We formulate the alternating optimization of radiance fields as a dynamic
evolution process with temporal
memory, and jointly exploit comprehensive multi-dimensional world states and a mixture-of-experts mechanism to dynamically adjust the confidence of deblurring priors. Experimental results show that WS-NeRF significantly improves blurry radiance field reconstruction quality, achieving better performance on PSNR, SSIM, and LPIPS, while exhibiting more stable iterative recovery behavior.
Read the full abstract ▾
All authors Hang Jiang, Jinghao Wang, Yiming Zhang, Xinhong Wang, Luwei Ran, Yinfeng Yu
Computer Vision
AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents
Seoyeon An, Hyeonseo Jang, Minsu Kim et al. (+3 more)
Comprehensive
video understanding is crucial for advancing artificial intelligence toward the intricate dynamics of the physical world. While recent advances in
Multimodal Large
Language Models (MLLMs) have demonstrated remarkable capabilities in
video understanding, existing
benchmarks remain confined to simple scene-level queries or global summaries that require only single-step inference. Real-world
video understanding involves more challenging tasks that require multi-hop
multimodal reasoning, and there is a critical absence of
video benchmarks equipped to rigorously evaluate these agentic capabilities. To bridge this gap, we introduce AgentVidBench, a multi-hop
video question answering
benchmark focused on evaluating the spatial, temporal, and causal
reasoning capabilities of MLLM
agents. Beyond standard question-answer pairs, AgentVidBench provides step-by-step solution traces to support trajectory evaluation that assesses whether
agents explicitly acquire the evidence needed to justify their answers. Experiments with 12 proprietary and open-source MLLMs show that single-turn performance remains limited on AgentVidBench, while integrating these models into state-of-the-art agentic workflows generally improves performance with respect to both accuracy and trajectory scores. We further present a simple yet effective agentic strategy that serves as a competitive baseline on AgentVidBench, establishing our
benchmark as a holistic testbed for future research on agentic
video understanding. Code and
datasets are available at https://github.com/krafton-ai/agentvidbench and https://huggingface.co/
datasets/agentvidbench/agentvidbench.
Read the full abstract ▾
All authors Seoyeon An, Hyeonseo Jang, Minsu Kim, Chanho Lee, Younghan Park, Kangwook Lee