Computer Vision
AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
Cheng Yang, Yifan Wu, Yutao Huang et al. (+18 more)
GUI
agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step workflows. However, the diversity of available trajectories is constrained by the applications, interface states, and workflows accessible in the underlying environments. Expanding this coverage requires deploying increasingly diverse and complex software, with specialized applications imposing additional installation, configuration, and runtime costs. We introduce AutoGUIWorld, a data generation framework that combines the visual priors of image generators with the task knowledge of a planner to synthesize GUI interaction trajectories without deploying or running the corresponding software environments. AutoGUIWorld samples initial GUI scenes from structured specifications of operating-system context, visual appearance, and interface state, and generates tasks conditioned on those scenes. A planner then specifies atomic actions and their intended visual consequences, while an image generator iteratively edits the current screenshot to produce subsequent observations. Action grounding and transition-level quality filtering yield 79,266 spatially annotated step-level training samples across Ubuntu, Windows, macOS, and Chrome. Fine-tuning Qwen3.5-35B-A3B on AutoGUIWorld trajectories improves the mean task score on OSWorld from 33.0% to 40.8% and the task success rate on ScienceBoard from 14.0% to 32.2%. These results show that generated trajectories improve GUI-
agent performance on real desktop and scientific tasks.
Read the full abstract ▾
All authors Cheng Yang, Yifan Wu, Yutao Huang, Zhaohua Zhang, Beiduo Chen, Muxi Chen, Chenchen Zhao, Hexuan Deng, Haolin Yang, Geyuan Zhu, Sa Zhu, Jianhuan Zhuo, Qiuyong Xiao, Jianhao Ruan, Yiran Peng, Jiayi Zhang, Tian Ye, Xinlei Yu, Tianwen Jiang, Jihong Zhang, Yuyu Luo
Computer Vision
EgoFound3R: End-to-End Egocentric Hand Reconstruction in World Space with Point-Wise Interaction Attributes
Hongming Fu, Jingcheng Shi, Wenjia Wang et al. (+2 more)
Egocentric
video has become a primary source of supervision for embodied models, and its value rests on recovering hand motion in world coordinates, which camera motion and hand occlusion make difficult. Existing reconstruction pipelines typically separate hand and scene estimation, leave interaction attributes to separate task-specific models, and invoke several models per
video, so no prior reconstruction model estimates these attributes and throughput becomes a practical constraint on large-scale annotation. We therefore introduce EgoFound3R, a unified end-to-end model that estimates world-space hand geometry in a metric scale shared with the scene, and predicts point-wise interaction attributes, including visibility, contact, and distance. The model integrates three designs: (i) structured hand
prompts that transfer pretrained
geometric priors to world-space hand reconstruction; (ii) an explicit hand representation that decodes hand geometry and interaction attributes; and (iii) a shared-parameter multi-rate design that lowers inference cost. Together, these designs predict hand geometry and point-wise attributes in one pass. On OakInk-v2, TACO, and HOI4D, EgoFound3R reduces the mean per-joint position error (MPJPE) by 43.2%, 22.4%, and 11.6% over previous methods and predicts point-wise contact and distance alongside the geometry in the same pass, while attaining approximately 6x higher throughput.
Read the full abstract ▾
All authors Hongming Fu, Jingcheng Shi, Wenjia Wang, Binhua Zuo, Bo Zhao
Computer Vision
Resolving Mixed Single-Photon LiDAR Returns for Foreground-View and Hidden Scene Reconstruction
Ziting Wen, Runrong Deng, Zili Zhang et al. (+5 more)
Partially transmissive screens and protective covers are common in robotic inspection, but they create mixed
LiDAR returns from both the foreground material and the scene behind it. Conventional peak-based
LiDAR usually discards weak hidden returns, while single-
photon LiDAR records time-resolved histograms that preserve attenuated and overlapping echoes. However, existing transient reconstruction methods typically fit a single scene representation to the measured waveform. Under occlusion, weak or nearby foreground–hidden echoes can form a broad peak or subtle shoulder. Because such waveforms can also be explained by a displaced single surface or a thick density distribution, accurate transient fitting does not necessarily imply correct geometry. We propose a state-aware framework for foreground-view and hidden scene reconstruction from occluded single-
photon histograms. For each ray, we estimate local echo evidence, identifying no reliable surface evidence, single-return evidence, or two returns. The inferred echo state routes supervision for a two-head neural field: all rays constrain waveform reconstruction, while reliable anchors provide geometry localization. We also introduce a real paired single-
photon LiDAR occlusion
dataset with occluded and clean captures at fixed poses. Experiments on a real
dataset show improved hidden scene depth and point-cloud accuracy over baselines. Our results demonstrate single-
photon layered reconstruction as a practical route for 3D perception through partially transmissive occluders.
Read the full abstract ▾
All authors Ziting Wen, Runrong Deng, Zili Zhang, Haitao Zheng, Yuecong Xu, Xiaoqiang Ren, Guodong Shi, Kemi Ding
Computer Vision
Semantic RGB–Depth Based Surgical Skill Assessment in Microscopic Stereo Videos
Jecia Z. Y. Mao, Sue M. Cho, Francis X. Creighton et al. (+3 more)
Objective assessment of microsurgical technical skill is essential for competency-based training and quality assurance, yet existing
video-based approaches predominantly rely on RGB images and therefore overlook the 3D spatial relationships that characterize instrument-anatomy interactions. Although stereo operating microscopes provide complementary depth information, conventional stereo matching
algorithms can produce sparse and unreliable depth estimates under high-magnification imaging conditions, limiting their use for automated skill assessment. This work presents a semantic RGB-Depth framework for surgical skill assessment from microscopic stereo
videos. A regression-based depth fusion method combines sparse metric stereo depth with dense monocular depth estimates to generate a dense
geometric representation of the surgical scene. This representation is integrated with semantically decomposed RGB streams corresponding to individual surgical instruments and surrounding anatomy. A hierarchical
attention architecture jointly encodes these streams to capture discriminative patterns of instrument use and instrument-anatomy interaction across surgeons at different training levels. The framework was evaluated on 33 ex vivo transoral microlaryngeal procedures performed by six surgeons, comprising attending surgeons and surgical residents, using leave-one-surgeon-out cross-validation. The proposed semantic RGB-Depth model achieved an F1 score of 0.938 for skill-level classification, compared with 0.696 for semantic RGB and 0.929 for semantic depth. These results suggest that
geometric information can improve automated surgical skill assessment from microscopic stereo
videos. The learned spatial, temporal, and semantic
attention patterns also support qualitative examination of the scene regions,
video segments, and semantic streams emphasized by the model.
Read the full abstract ▾
All authors Jecia Z. Y. Mao, Sue M. Cho, Francis X. Creighton, Deepa Galaiya, Russell H. Taylor, Manish Sahu
Computer Vision
iSEE: Object Permanence Through Self-Supervision
Pramish Paudel, Ajad Chhatkuli, Luc Van Gool et al. (+1 more)
Object permanence, keeping track of an object's identity and position while it is occluded, is central to
video representations that track, predict and plan. Trackers that achieve it learn from boxes, track identities and visibility labels. On the other hand, self-supervised object-centric methods discover objects without labels: through slot
attention, it represents a
video as slots that bind to objects and follow them across frames. However, these slots are lost under occlusion, making the desired permanence impossible.
Reasoning permanence is a hard problem because it requires to detect when an object becomes occluded, re-identify when object reappears, and keep the object's hidden position continuous, using reapperance as the only learning cue. To address this, we propose iSEE, a novel framework that offers all three aforementioned requirements, without any labels whatsoever. We built iSEE using the following three proposed components: (i) Object evidence modelling: a slot's
attention, compared with its own past, reveals when its object is hidden. (ii) Appearance-position separation: two slot streams let the appearance be held for re-identification while the position keeps changing. (iii) Permanence from reappearance: a walker follows the hidden object's position, trained only on where the object reappears. On LA-CATER static, iSEE returns a reappearing object to its own slot after 86% of occlusions, against 32% for SlotContrast, and localises it while hidden within 4.1 mAP of the label-trained SoTA RAM. The two streams also allow downstream planning, with the position stream as the action of a world model. Project page: https://insait-institute.github.io/iSEE/
Read the full abstract ▾
All authors Pramish Paudel, Ajad Chhatkuli, Luc Van Gool, Danda Pani Paudel
Computer Vision
FlashBack: Knowing When to Remember in Streaming Vision-Language Models
Yi Chen, MingMing Yu, Rui-Qi Wang et al. (+5 more)
Streaming
vision-
language models must process continuously growing
video streams under a bounded compute budget, creating a persistent tension between real-time perception and long-term
memory. Retrieving historical information provides a natural remedy, yet historical recall is not uniformly beneficial: unnecessary history may introduce irrelevant context into current
reasoning and interfere with native real-time perception. Effective streaming
memory should therefore address not only what to remember, but also when and how to access it. To this end, we introduce FlashBack, a training-free framework for selective, multi-level
memory in streaming
vision-
language models. Before retrieving history, FlashBack draws on the semantic understanding of the frozen streaming VLM to infer whether a query calls for historical evidence. This assessment determines whether inference remains on the Native trajectory or invokes an isolated Recall trajectory. The Recall trajectory combines recent context with retrieved long-term
memory through a query-local Side-KV pathway, preserving local temporal continuity without modifying the persistent Native state. We instantiate FlashBack on StreamingVLM and Mage-VL-4B and evaluate it on OVO-Bench and StreamingBench. The results show improvements on several long-horizon and
memory-dependent tasks while largely preserving real-time perception, with performance competitive with strong training-based streaming methods despite requiring no additional training. Our code will be announced later.
Read the full abstract ▾
All authors Yi Chen, MingMing Yu, Rui-Qi Wang, Boran Wang, Xiaohang Cao, Chu Tang, Jingmin Chen, Jie Gu
Computer Vision
Color Independent Word Segmentation From Transcribed Bangla Passages
Faias Satter, Noor Masrur and Sk. Md. Masudul Ahsan
An optical character recognition(OCR) system can scan paper and extract text, making people's jobs easier. While numerous OCR systems are accessible in the software sector, finding a dependable equivalent solution for Bangla is tough. When it comes to handwritten texts, the case is even more rare. The first fundamental step to any OCR is to segment words from text images. If this stage fails, the total OCR's performance will be poor no matter how promising the later stages perform. This research aims to segment words in a handwritten Bangla text image. This research can be implemented on any smartphone-captured image, irrespective of the color and type of paper and ink. Furthermore, as smartphone-captured images can create shadow interferences, the custom
dataset built for this research is created in such a way that every possible obstacle that can be faced is included. For 7374 words, a total of 7278 bounding boxes are generated, which have recall of 90.60%, precision of 91.80%, and F1-score of 91.20%. The system can be further improved with nested operations on bounding boxes containing several words or by adjusting the adaptive thresholding and dilation filter sizes to a more precise level.
Read the full abstract ▾
Computer Vision
Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models
Yuliang Cai, Mohammad Rostami and Jesse Thomason
Despite the strong performance of
Vision-
Language Models (VLMs) on a wide range of visual question answering (VQA) tasks, these models consistently struggle to understand negation and produce incorrect answers when questions involve negated clauses. To address this limitation, we propose Skeleton-and-Strategy Prompting (SSP), a training-free, in-context learning method that improves VLM negation understanding capabilities without any parameter updates. Given a negation question, our method first abstracts the underlying question structure into a skeleton, retrieves a small set of same-skeleton questions from a lightweight question pool, then
prompts the VLM to analyze their shared negation pattern and synthesize a single-sentence answering strategy. The skeleton and strategy are prepended to the test sample to guide the model correctly tackle the negation problems. Experiments on multiple negation VQA
benchmarks show that SSP achieves state-of-the-art performance on negation-focused VQA tasks while remaining computationally efficient.
Read the full abstract ▾
Computer Vision
CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment
Kunyang Li, Hai Nguyen, Joshua Lowe et al. (+6 more)
Cardiovascular
magnetic resonance (CMR), including cine imaging, is a reference standard for the noninvasive assessment of cardiac morphology and ventricular function. Cine CMR interpretation integrates qualitative visual assessment with quantitative measurements of ventricular volumes, ejection fraction, myocardial mass, wall thickness, and regional wall motion. Current medical
vision-
language models (VLMs) cannot reliably derive quantitative measurements from multidimensional cine images without analysis tools. We present CineMR, a tool-augmented VLM that invokes cardiac image-analysis tools and integrates their outputs into interleaved
reasoning for quantitative CMR assessment. We also construct a multi-cohort visual question answering
benchmark covering quantitative metric extraction, multiclass diagnosis, and differential diagnosis, together with tools for segmentation, phase selection, volumetry, morphometry, and regional wall motion analysis. CineMR is trained with supervised fine-tuning (SFT) on tool-interaction traces followed by Group Relative Policy Optimization (GRPO) with conditional tool-use rewards. On the multi-cohort cine CMR
benchmark, CineMR achieves 35.9% pass@1 and 58.9% pass@4, compared with 1.5% pass@1 for the Qwen3-VL-8B backbone and 0.0% and 7.0% pass@1 for LLaVA-Med v1.5 and MedGemma-4B, respectively. Correct tool invocation reaches 99.8% after GRPO, up from 78.9% after SFT. Live tool outputs improve ventricular measurement accuracy by 20.4–23.7% over direct model predictions, and removing all tools reduces pass@1 from 35.9% to 27.9%. These results highlight the importance of reliable tool use for quantitative cine CMR
reasoning and support CineMR as a promising approach for assistive cardiac image assessment. Code,
benchmark resources, and model weights are available at https://github.com/AI-MIND-Lab/CineMR.
Read the full abstract ▾
All authors Kunyang Li, Hai Nguyen, Joshua Lowe, Chenguang Zhao, Peace C. Madueme, Mehdi Hedjazi Moghari, Mubarak Shah, Pegah Khosravi, Yuzhang Shang
Computer Vision
PhysicsLENS: Diagnosing Physical Property Blindness in Video Generation Models
Isaiah Milkey, Som Sagar, Aditya Taparia et al. (+3 more)
Reliable
video world models could provide scalable predictive environments for
robot learning, planning, and evaluation. However, generated
robot videos can violate physical principles and complete tasks through physically implausible behavior, limiting their reliability for
robot learning and planning. Current
video-generation
benchmarks exclude physics that are inherently hidden by visuals (e.g., weight, viscosity, friction). Due to this,
video models are evaluated on the fidelity of physics, not the underlying accuracy of physics. We introduce PhysicsLENS, a
dataset and
benchmark for evaluating plausibility of physical properties grounded in robotics. PhysicsLENS uses matched scenario pairs that hold the same conditioning frame and task, while varying underlying physics in the scene description. Scenarios are curated from public
robot video sources and annotated across seven physical domains: collision,
gravity, momentum, friction, deformation, fluid, and causality. We evaluate across four
video generation models, producing over 400 human-annotated labels. Results show that plausible-looking
videos often ignore the stated property (34 of 47), and that stating the property lowers plausibility only slightly and not significantly.
Read the full abstract ▾
All authors Isaiah Milkey, Som Sagar, Aditya Taparia, Xinyuan Liu, Jiqing Wen, Ransalu Senanayake
Computer Vision
OptimusMesh: Compact Autoregressive Mesh Generation from Point Clouds via Sparse Latent Pivots
Mazhar Iqbal, Xuanmeng Sha, Naoya Chiba et al. (+2 more)
Generating compact and geometrically faithful 3D meshes directly from point clouds remains a fundamental challenge. Point clouds are unordered and sparse, whereas meshes exhibit irregular structure and varying
topology. As a result, many existing approaches rely on implicit representations followed by surface extraction or reconstruction. Although effective, these pipelines can produce dense or over-smoothed meshes, often requiring computationally expensive post-processing and simplification. We present OptimusMesh, a framework for direct compact triangle mesh generation from point clouds using sparse latent pivot conditioning. Our key idea is to compress $2{,}048$ oriented input points into only $16$ sparse latent pivots, reducing the
geometric conditioning set by $128\times$. These pivots provide a compact structural representation shared across a two-stage autoregressive framework that first generates mesh vertices and then predicts triangular faces conditioned on the generated vertices and the same pivots. Compared with the evaluated recent point-cloud-conditioned autoregressive methods, which use $257$ decoder-conditioning
tokens, OptimusMesh uses only $16$, yielding a $16.1\times$ shorter conditioning sequence. Experiments show that OptimusMesh produces the most compact outputs among the compared recent autoregressive methods, using $25.7\%$–$94.1\%$ fewer faces while maintaining competitive
geometric fidelity and distributional quality.
Read the full abstract ▾
All authors Mazhar Iqbal, Xuanmeng Sha, Naoya Chiba, Yuki Uranishi, Tomohiro Mashita
Computer Vision
The RSNA Intracranial Aneurysm (RSNA-ICA) Dataset
Maria Correia de Verdier, Rachit Saluja, Jason Sho et al. (+37 more)
Intracranial aneurysm rupture is associated with substantial morbidity and mortality, yet aneurysm detection remains challenging, particularly for small lesions and on routine non-angiographic imaging examinations. To support the development and evaluation of artificial intelligence (AI)
algorithms for intracranial aneurysm detection and localization, the Radiological Society of North America (RSNA), in collaboration with the American Society of Neuroradiology (ASNR), the Society of Neurointerventional Surgery (SNIS), and the European Society of Neuroradiology (ESNR), curated the RSNA Intracranial Aneurysm (RSNA-ICA)
Dataset. Developed for the 2025 RSNA Intracranial Aneurysm Detection Challenge, RSNA-ICA is a large, publicly available, expert-annotated
dataset comprising 7202 CTA, MRA, and MRI series from 4278 adult patients collected across 21 institutions in 12 countries spanning five continents. The
dataset includes 2566 CTA, 2166 MRA, and 2470 MRI series from patients with and without intracranial saccular aneurysms, providing substantial geographic and imaging diversity. Expert annotations indicate both aneurysm presence and location, and 178 series additionally include three-dimensional segmentations of challenge-defined vascular locations. RSNA-ICA was used to develop and evaluate
algorithms in the 2025 RSNA Intracranial Aneurysm Detection Challenge. Of the 7202 image series, 5041 are publicly available through MIRA, while the remainder were used for challenge public and private test sets. The
dataset is freely available to the research community for noncommercial use and provides a comprehensive resource for advancing AI-based aneurysm detection across both angiographic and routine neuroimaging examinations.
Read the full abstract ▾
All authors Maria Correia de Verdier, Rachit Saluja, Jason Sho, Maryam Vabarizad, Rennie Yung-Chieh Chen, Uyen N. T. Nguyen, Mona Alrehaili, Layal Aweidah, Deniz Bulja, Wesley C. Chan, Hernan Chaves, Madhavi Duvvuri, Huseyin Ekin Ergin, Undrakh-Erdene Erdenebold, Ekim Gumeler, Mohamed Sobhi Jabal, Chin-Chi Kuo, Fatima Mubarak, Sevde Nur Emir, Scott Riley K. Ong, Johanna Ortiz, Almudena Pérez-Lara, Andreas M. Rauschecker, Shayan Sirat Maheen Anwar, Charit Tippareddy, Tam Tran, Sorawis Visrutaratna, John Mongan, Adam E. Flanders, Robyn Ball, Greg Zaharchuk, Peter D. Chang, Felipe Kitamura, Errol Colak, Luciano Prevedello, Tyler Richards, Data Contributor Group, Dataset Annotator Group, Evan Calabrese, Jeffrey D. Rudie
Computer Vision
Open Vocabulary Word Recognition From Transcribed Bangla Texts
Faias Satter and Sk. Md. Masudul Ahsan
An optical character recognition (OCR) can scan a paper and extract text using technology, making people's jobs easier. While various OCR systems are available in the software industry, finding a reliable equivalent solution for Bangla takes much work. When it comes to handwritten texts, the situation is much more unusual. Recognizing words from word images is the most critical stage in any OCR process. It is the second stage after segmenting words from text pictures. If this stage fails, the overall performance of the OCR will be poor, regardless of how well the other phases perform. This study aims to recognize words using deep learning in a handwritten Bangla word image. Three object detection models, SSD with MobileNetV2, Faster R-CNN with InceptionResNetV2, and an ensemble model of these two, have been used to train and test handwritten word images. A modified Non-Maximum Suppression has been introduced to enhance the effectiveness of the models' results. A customized
dataset of 9841 handwritten Bangla word images has been compiled, featuring diverse handwriting styles from various individuals. All three models' performances have been checked against the test
dataset, and the ensemble model has been the most impressive, with an F1-score of 92.61%. Also, at the word level, the ensemble model correctly recognizes 96.12% of the words to some extent. The system can be further improved by introducing a post-processing phase to correct errors generated by the system.
Read the full abstract ▾
Computer Vision
Affine-Aligned Atlas for Canonical Gaussian Construction in Video Representation
Masaya Takabe, Hiroshi Watanabe, Sujun Hong et al. (+1 more)
Gaussian splatting has recently emerged as an efficient representation for images and
videos due to its explicit structure and fast rendering capability. Existing Gaussian-based
video representations often decompose a
video into canonical Gaussians and temporal deformation. However, when a
video contains large global motion such as camera movement, the canonical representation may become misaligned with individual frames, increasing the burden on the temporal deformation model. In this paper, we propose an affine-atlas canonical Gaussian representation, which constructs canonical Gaussians in a larger affine-aligned atlas space. Frame-wise affine transforms absorb global motion before canonical Gaussian construction, reducing the gap between the canonical representation and target frames. Since the proposed method only modifies the canonical construction stage, it can be integrated into existing canonical-Gaussian-based methods with negligible additional parameter cost. Experiments show that our method improves reconstruction quality especially for sequences with large camera motion.
Read the full abstract ▾
All authors Masaya Takabe, Hiroshi Watanabe, Sujun Hong, Tomohiro Ikai
Computer Vision
MVDG: Efficient Multi-view 3D Disambiguation on Unconstrained Real-World Images
Hanyuan Xiao, Gonglin Chen, Haolin Xiong et al. (+3 more)
Illusory matches between distinct yet visually similar 3D surfaces–doppelgangers–remain a fundamental obstacle for large-scale, in-the-wild 3D reconstruction and visual localization. Prior work mitigates this issue with pairwise classifiers, but this design limits multi-view contextual
reasoning and incurs O(n^2) inference complexity for downstream structure-from-motion (SfM). We present MVDG, a scalable multi-view disambiguation framework built on the 3D foundation model VGGT, which jointly reasons over an arbitrary number of multiview images. By incorporating 3D-aware multi-view features, our method reduces dependence on pairwise comparisons by encoding and decoding views in a single pass. We further observe that direct multi-view fine-tuning of VGGT can be unstable under noisy supervision; motivated by label ambiguity in Doppelgangers, we construct a pseudo-pairwise training set from AerialMegaDepth and show that fine-tuning on sampled subsets yields stable optimization and strong generalization to held-out scenes. Finally, because full SfM evaluation (even with faster pipelines such as GLOMAP) remains expensive, we process a pseudo-pairwise
dataset for efficient validation; we derive a predictive relationship between regular SfM metrics and the classification accuracy on this pseudo-pairwise test. Experiments show that our method achieves comparable pairwise accuracy while improving both SfM accuracy and inference speed over baselines.
Read the full abstract ▾
All authors Hanyuan Xiao, Gonglin Chen, Haolin Xiong, Wenbin Teng, Haiwei Chen, Yajie Zhao
Computer Vision · Cross-listed from cs.LG
Dataset Identity, Not Novelty: The Source of an Inflated OOD Detection Gain
Donghoon Lee and Shinjin Kang
A post-hoc out-of-distribution (OOD) detector reads the activations of a trained classifier and returns a score. It fits that score on in-distribution data, and the
benchmarks that evaluate it supply a second piece of OOD data for the fitting itself. Some detectors tune a constant on it. Others fit a direction in feature space or train a flexible combiner and report the number that it reaches as the gain that is still available. Every such fit is validated on held-out samples of the same OOD
dataset. That check rules out memorizing individual images. It says nothing about a fit that has instead learned which
dataset it is looking at, and a direction that recognizes one OOD
dataset rather than novelty passes it perfectly. The detector that a practitioner installs meets OOD data from a source that nobody fitted it on, so the difference decides what the reported number is worth. We measure it by holding out the whole OOD
dataset rather than a sample of it, and we call that gap the inflation. We read it across a range of combiners on ImageNet and CIFAR-100 backbones. Most of the gain that the usual protocol reports turns out to be
dataset identity rather than novelty. The size of the fit does not move what survives, so the effect is not ordinary overfitting. The share depends instead on whether the input exposes class identity, and two controls that vary that property alone separate the inflation on every backbone of both
benchmarks. A closed form accounts for the effect and computes it from the fitting rows, so a practitioner can tell which fits will inflate without running the hold-out protocol. One of these fits survives, namely the single constant that the field already picks on a designated validation
dataset. Anything above it reports a gain that the hold-out protocol does not return, and on one
benchmark what survives falls while what is reported climbs.
Read the full abstract ▾
Computer Vision
Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana et al. (+10 more)
Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilities is crucial, existing
benchmarks focus mainly on single short actions or step-by-step instructions. This leaves multi-step physical
reasoning underexplored, especially in egocentric
video generation that requires planning to simulate proper execution to accomplish high-level goals by carrying out multiple real-world manipulations. We introduce Ego2Act, a goal-directed
benchmark featuring 2,640
videos from 110 real-world tasks across day-to-day settings, varying object clutter and multi-step complexity. Given an initial scene image and a high-level goal, Ego2Act evaluates whether
video generation models can produce realistic egocentric
videos of a hand manipulating objects to carry out the task. To support scalable evaluation, we also introduce Ego2ActJudge, a reference-free evaluation pipeline that achieves better task completion and physics plausibility evaluation
alignment with human consensus compared to relevant baselines. Our findings reveal that models' generated simulations often skip or partially execute steps, leaving later steps missing dependent states, which leads to unfulfilled goal. Furthermore, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and persistent world modeling. We hope Ego2Act provides a rigorous testbed for advancing
video models toward physically plausible, goal-directed simulation.
Read the full abstract ▾
All authors Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana, Qinrong Cui, Erland Hilman Fuadi, Zayd M. K. Zuhri, Nanda Ryaas Absar, Ahmed Elshabrawy, Wilfried Ariel Mulyawan, Shoubin Yu, Yue Zhang, Mohit Bansal, Alham Fikri Aji
Computer Vision
Overcoming Kernel Redundancy for Scaling Logic Gate Networks
Sejin Park, Hongjae Lee, Changwoo Han et al. (+1 more)
Differentiable logic gate networks, which operate using only logic gates, have recently attracted
attention as an efficient alternative to conventional neural networks. However, despite their efficiency, the scaling behavior of logic gate networks remains underexplored. By contrast, scaling model capacity is a central design principle in deep neural networks and typically leads to improved performance. This discrepancy raises a key question: Can similar scaling benefits also be achieved in logic gate networks? In this work, we focus on width as a primary scaling axis and conduct a systematic analysis of its behavior in logic gate networks. We observe that naive width scaling often introduces redundancy among logic kernels, limiting the effective use of additional kernels and leading to performance saturation. To address this limitation, we propose a dynamic logic kernel framework that reorganizes kernel utilization by promoting specialization across kernel groups. This enables the network to better utilize increased width via input-dependent kernel routing, while ensuring that both routing and computation are implemented entirely with gate-level Boolean operations at inference time. We further find that kernel redundancy is most pronounced at the first gate level, motivating an early-stage dynamic logic kernel strategy that concentrates adaptation at this level. Experimental results demonstrate that our approach improves kernel utilization and increases kernel diversity, leading to higher accuracy with improved parameter efficiency.
Read the full abstract ▾
All authors Sejin Park, Hongjae Lee, Changwoo Han, Seung-Won Jung
Computer Vision
HierGF: Hierarchical Gaussian Fields via Geometry-perception Message Passing for Sparse-view 3D Reconstruction
Bi'an Du, Zhimin Zhang, Daizong Liu et al. (+2 more)
Sparse view 3D reconstruction is an important and common scenario in multimedia applications, such as augmented reality/virtual reality (AR/VR) content creation, cultural heritage digitization, and certain robotic applications, where only a limited number of randomly captured views may be available. However, sparse views contain only limited 3D information, posing two major challenges:1) too few images are available for matching, making it difficult to build multi-view consistency; 2) insufficient view coverage leads to a lack of information in under-sampled regions, resulting in missing parts of object structure. Existing methods mostly still rely on limited reprojection errors and regularization terms, which are prone to overfitting to a single view and inconsistent appearances across views. In geometrically under-sampled regions, they often rely on heuristic density control, lacking reliable guidance and often resulting in blurring and structural holes.To address these issues, this paper proposes Hierarchical Gaussian Fields (HierGF), which revisits sparse-view reconstruction from a hierarchical geometry-perception perspective and converts limited observations into reliable self-generated supervision beyond fixed priors and heuristic density control. In particular, we transform coarse 3D
geometric information and additional 2D generative priors into structured pseudo-supervision through a two-stage geometry-perception backbone network, thereby enhancing multi-view consistency with very few input views. In addition, we introduce a learnable confidence network to guide gradients toward cross-view consistent content, and a geometrically consistent densification module to improve the reconstruction of multi-view
alignment and under-sampled regions.
Read the full abstract ▾
All authors Bi'an Du, Zhimin Zhang, Daizong Liu, Baoquan Chen, Wei Hu
Computer Vision
Towards Subject Consistency over Dynamic Subject Sets in Video Generation
Tongcheng Zhang, Jun Zhu and Jianfei Chen
We argue that as
video generation extends to longer durations, subject consistency should be evaluated over dynamic subject sets. We therefore introduce DynSC-Eval, an evaluation framework that dynamically tracks eligible subjects throughout their visible lifespans and measures local continuity and global identity preservation using six complementary object-level metrics, with explicit detection of inconsistency events. To validate its effectiveness, we design synthetic experiments that actively inject inconsistency events, demonstrating both the sensitivity of DynSC-Eval and the limitations of existing metrics. Evaluations of diverse models on 5s, 15s, and 60s
video generation further reveal substantial subject consistency differences that are obscured by conventional metrics. Beyond evaluation, we construct rewards from DynSC-Eval and apply DiffusionNFT post-training in an autonomous-driving testbed. On 5s generation, our approach reduces the six inconsistency metrics by an average of 13.82% for Wan-2.1-1.3B and 5.66% for SANA-2B, with improvements also observed on the I2V model ReSim. Qualitative comparisons further demonstrate the effectiveness of our method. We then extend generation to 10s and 30s through curriculum learning and show that consistency optimization remains effective while largely preserving other capabilities.
Read the full abstract ▾
Computer Vision
Bootstrapping Video Interaction Generation with Synthetic State Transitions
Jiho Jang, Jinyoung Kim, Nojun Kwak et al. (+1 more)
While recent
video generative models can synthesize high-fidelity
videos, they struggle to portray plausible physical interactions and the resulting state transitions, a critical bottleneck for applications in robotics and VR/AR. To address this, we introduce a framework to generate a scalable synthetic
dataset of controllable interactions. Our pipeline leverages a structured taxonomy and state-of-the-art image editing models to create explicit `start' and `end' state images, which serve as visual anchors for the interaction. To generate a seamless
video utilizing these anchors, we propose State-Guided Sampling (SGS), a novel sampling technique that mitigates artifacts common in naive conditional generation. Furthermore, we develop and validate a new automated evaluation system that aligns with human judgments to ensure data quality. Experiments show that fine-tuning a base model on our
dataset significantly enhances its ability to generate plausible interactions.
Read the full abstract ▾
All authors Jiho Jang, Jinyoung Kim, Nojun Kwak, Kyungjune Kim
Computer Vision
Towards Automatic Video Annotation with ASH: Zero-Shot Open-Vocabulary Multi-Object Tracking and Segmentation
Arash Rocky and Q. M. Jonathan Wu
Memory-
attention-based
Video Instance Segmentation (VIS) methods have demonstrated strong zero-shot tracking capability, yet their substantial
memory requirements confine them to short
video clips and their single-
prompt inference design makes multi-category open-vocabulary tracking computationally prohibitive. This work introduces two contributions toward fully automated tracking annotation of arbitrary
video. The Generalized Presence
Token (GPT) reformulates SAM3's inference pipeline to process N text
prompts simultaneously via virtual
prompt batching, reducing image encoding cost from O(N) to O(1) with no modifications to any learned component. The Annotation and Segmentation Handler (ASH) extends any
memory-
attention VIS tracker to sequences of arbitrary length through overlapping temporal chunks with IoU-based inter-chunk identity matching, requiring no
dataset-specific training. Instantiated on SAM3, the resulting pipeline – SAM3-ASH – achieves state-of-the-art HOTA on MOTS20 under fully zero-shot conditions and remains competitive with trained specialists across seven additional
benchmarks, while peak GPU
memory consumption stays below 25 GB, establishing a practical baseline for scalable, training-free automated
video annotation.
Read the full abstract ▾
Computer Vision
FutureWorlds: Learning Robotic World Models from Alternative Futures
Hao Wu, Shengju Qian, Weiyan Wang et al. (+5 more)
Robotic world models predict action-conditioned future scenes, providing a foundation for understanding action outcomes. However, turning alternative predictions into useful learning signals remains challenging: similar candidates limit informative quality comparisons, while diverging trajectories require persistent maintenance of their individual histories. We introduce FutureWorlds, a framework that unifies candidate construction, history maintenance, and learning from relative quality. Built on a
multimodal discrete autoregressive model, FutureWorlds uses diverse beam search during reinforcement learning to construct candidate futures that balance confidence and diversity. Candidate-specific bounded
memory preserves scene states and ensures that generation and policy scoring use matching histories. We further propose MemSPO (
Memory-Conditioned Search-Guided Policy Optimization), which converts
video trajectory rewards into group-relative advantages to optimize the world model. On RT-1, BridgeV2, and RoboCasa, FutureWorlds reduces LPIPS for 32-frame predictions by 14.78%, 20.84%, and 9.12%, respectively, relative to the strongest baseline on each
dataset. Under fixed evaluation configurations, only 200 MemSPO updates further improve generation quality and support continued prediction beyond the training horizon.
Memory ablations, decoding sensitivity analysis, and optical-flow evaluation show that these gains extend beyond visual quality to more accurate motion prediction and more consistent object states. Project page and code: https://github.com/Alexander-wu/FutureWorlds.
Read the full abstract ▾
All authors Hao Wu, Shengju Qian, Weiyan Wang, Fan Xu, Fan Zhang, Yuanpeng He, Qingsong Wen, Yuxuan Liang
Computer Vision
VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstruction
Junyi Wu, Fanqing Kong, Leyang Chen et al. (+2 more)
Feed-forward 3D
vision models such as VGGT have achieved remarkable progress, unifying camera estimation and dense scene reconstruction in a single pass. However, their quadratic global
attention makes long image sequences expensive, while existing sparse methods may favor highly attended yet value-redundant regions. To address these limitations, we introduce VASC, a training-free sparse
attention method combining value-aware block selection and execution-aware cross-layer
memory. Our value-aware block selection integrates pooled query–key relevance with neighboring value contrast, reducing redundancy while preserving query-relevant and distinctive content. Cross-layer
memory tracks unserved demand across layers and updates this state according to actual execution, enabling previously underserved blocks to compete under a fixed computation budget. Experiments on 7Scenes and NeuralRGB-D with VGGT and $\pi^3$ demonstrate improved pose estimation and reconstruction quality compared with FasterVGGT, together with up to $2.29\times$ faster inference than dense VGGT. Code is available at https://github.com/kosakayamahoo-design/VASC.
Read the full abstract ▾
All authors Junyi Wu, Fanqing Kong, Leyang Chen, Shaoqiu Zhang, Yulun Zhang
Computer Vision
Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioning
Gunwoo Lee, Yoori Oh and Yoseob Han
Video-to-
speech synthesis aims to generate natural-sounding
speech from silent talking-face
videos while ensuring phonetic accuracy. A fundamental challenge in this task is the inherent one-to-many mapping problem, where visual dynamics often lack sufficient information to uniquely determine the corresponding utterance. To address this, we propose Watch Your
Speech (WYS), a
video-to-
speech synthesis framework that incorporates textual conditioning as an explicit linguistic cue to mitigate visual ambiguity. Our framework features an
attention-based
embedding fusion module that synergistically integrates textual context with
video sequences, coupled with a conditional flow matching objective for high-fidelity
speech generation. Extensive experiments on the LRS2 and LRS3
datasets demonstrate that WYS achieves superior performance, establishing new state-of-the-art results in audio-visual synchronization (LSE-C/D) while maintaining highly competitive textual accuracy (WER). Subjective evaluations further confirm that our model generates
speech with near-human naturalness, validating the effectiveness of textual conditioning in content-controlled
video-to-
speech synthesis. Project page: https://github.com/gunwoo5034/Watch-your-
Speech
Read the full abstract ▾
Computer Vision
VIEScore2: Unified Image Evaluation with Spatially Grounded Explanations
Xianda Du, Max Ku, Weiming Ren et al. (+5 more)
Existing synthetic image evaluators typically provide only a scalar quality score and do not identify the image regions that support it. We introduce VIEScore2, a unified evaluator for image generation and editing tasks with optional conditioning images. VIEScore2 represents an image as an N x N grid and jointly predicts quality scores and defect locations in a single model pass. Its text-native grid representation provides a common interface for heterogeneous spatial supervision and enables directly verifiable post-training objectives. We train on 38K examples spanning score-only, localization-only, and joint supervision across generation and editing tasks. Starting from supervised fine-tuning, we further apply GRPO to improve defect localization using rewards that combine
cell-level Dice overlap, score accuracy, and output-format validity. A parameter-free parser converts the structured predictions into readable explanations. On the primary suite, VIEScore2 achieves an overall-score SRCC of 0.601, compared with 0.491 for Gemini-3-Flash, the strongest zero-shot general-purpose VLM baseline under matched inputs. For defect localization, VIEScore2 outperforms both general-purpose VLMs and specialized spatial evaluators on three of six
benchmarks in per-image grid IoU and ranks among the top three on five, including
datasets beyond its training sources.
Read the full abstract ▾
All authors Xianda Du, Max Ku, Weiming Ren, Zhi Rui Tam, Chunlin Ren, Ping Nie, Min-Hung Chen, Wenhu Chen
Computer Vision · Cross-listed from cs.RO
NarrativeFlow: Flow-Based Vision-Language-Action Model Using Robot Velocity Fields
Shota Kobayashi, Koki Seno, Daichi Yashima et al. (+1 more)
We focus on
language-conditioned flow-based manipulation, where
robot flows (
robot velocity fields) serve as embodiment-agnostic, motion-centric representations for leveraging data collected from multiple
robot platforms. This task is crucial because
language-conditioned manipulation is essential for practical robotic systems, yet scaling
robot foundation models remains limited by the labor-intensive collection of embodiment-specific data. Existing methods either coarsely approximate
robot flows with sparse keypoint displacements, or cannot handle
language-conditioned manipulation. To address this limitation, we propose NarrativeFlow, which models
robot flows as continuous velocity fields using a flow-matching formulation conditioned on
language. Accordingly, NarrativeFlow generates
robot flows that are physically consistent with real-world manipulation. To validate NarrativeFlow, we have conducted experiments on standard
datasets for
language-conditioned manipulation. The experimental results show that NarrativeFlow outperforms representative baseline methods on standard evaluation metrics. Furthermore, through real-world experiments, we show that NarrativeFlow achieves higher success rates than baseline methods across multiple manipulation tasks. The project page is available at https://shota0520.github.io/NarrativeFlow-project-page/
Read the full abstract ▾
All authors Shota Kobayashi, Koki Seno, Daichi Yashima, Komei Sugiura
Computer Vision
Concept Driven Domain Adaptation: Finding an Abstract Needle in a Haystack
Haiming Zhao, Tai Wang, Kun Zhang et al. (+2 more)
Science teachers frequently search for documentary excerpts not by describing what appears on screen, but by querying the abstract concepts they intend to teach. This use case exposes a limitation of existing
language-based
video moment
retrieval methods, which typically assume that queries describe observable events, whereas instructional search requires retrieving concrete visual phenomena that instantiate an underlying scientific principle. We study this setting as concept-to-example
video retrieval, an abstract-needle-in-a-haystack problem where compact curriculum concepts must be grounded in temporally sparse documentary evidence. To bridge this abstraction gap, we propose Concept-Driven Domain Adaptation (CDDA), a three-stage framework for adapting two-tower
vision-
language models to concept-level
retrieval. CDDA treats concepts as intermediate semantic anchors: it first structures the textual
embedding space with textbook and teacher-handbook example-concept pairs, then transfers this concept-aware geometry to documentary visuals under a frozen visual encoder, and finally jointly adapts both encoders with sparse visual concept supervision. From a
geometric perspective, this staged
alignment reduces text-concept and
vision-concept angular gaps, thereby encouraging concept-level adaptation while preserving the pretrained model's concrete image description
alignment. On a curated middle-school physics
retrieval benchmark, CDDA achieves stronger pedagogically oriented concept
retrieval than several competitive
multimodal baselines, including Qwen3-VL-
Embedding-2B, while maintaining concrete image-text matching after adaptation.
Read the full abstract ▾
All authors Haiming Zhao, Tai Wang, Kun Zhang, Xicheng Peng, Zhiyang Li
Computer Vision
RelationVGGT: Visual Geometry Transformers for 3D Spatial Relation Segmentation
Minsu Kim, Jaesung Choe, Jiwoo Lee et al. (+2 more)
Recent advances in 3D reconstruction have progressed from per-scene optimization to feed-forward inference, and semantic scene understanding has followed suit – yet existing methods remain confined to object-centric perception, neglecting spatial relations between objects. We formulate 3D spatial relation segmentation in a feed-forward, pose-free multi-view setting: given a visually specified subject and a relational text query, the model segments the target across views without receiving its category name. To this end, we propose RelationVGGT, a novel feed-forward framework that integrates semantic features from a visual foundation model with geometry-aware representations from a 3D geometry foundation model and leverages a relation
transformer for subject-conditioned, cross-view relation prediction – requiring neither per-scene optimization nor known camera poses. We additionally provide a fully automated annotation pipeline built on ScanNet++ with VLMs and LLMs, enabling scalable training data generation for this new task.
Read the full abstract ▾
All authors Minsu Kim, Jaesung Choe, Jiwoo Lee, Yu-Chiang Frank Wang, Seon Joo Kim
Computer Vision
Video-Index: A Curated Meta-Benchmark for Video Understanding
Enxin Song, Yinuo Xu, Shusheng Yang et al. (+2 more)
A
video benchmark should reward the capability it claims to measure, yet models can exploit answer options, question text, or partial visual evidence. We introduce the attack pyramid, five levels of shortcut attacks with increasing access to each item, and audit 115
video benchmarks with it. On 35
benchmarks, attackers that never see a frame approach full-
video accuracy. On 51
benchmarks with temporal probes, shuffled frames keep a median 96% of full-
video accuracy. Near-duplicate questions make up at least half the items in 63
benchmarks. We screen 505,518 question-answer pairs from 112 of them into an audited pool.
Agents turn evaluation requests into specifications, and a deterministic selector with a red-team gate composes reproducible
benchmarks. We release
Video-Index, the 210 hardest verified items under these attacks in each of four capability groups, 840 items from 76 sources. With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and
agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy. Blog: https://www.enxinsong.com/blog/
video-index/ GitHub: https://github.com/Espere-1119-Song/
Video-Index Hugging Face: https://huggingface.co/
datasets/
Video-Index/
Video-Index
Read the full abstract ▾
All authors Enxin Song, Yinuo Xu, Shusheng Yang, Wenhao Chai, Jiatao Gu