Computer Vision
Polarization-Based Eye Tracking with Personalized Siamese Architectures
Beyza Kalkanli, Tom Bu, Mahsa Shakeri et al. (+4 more)
Head-mounted devices integrated with eye tracking promise a solution for natural human-computer interaction. However, they typically require per-user calibration for optimal performance due to inter-person variability. A differential personalization approach using Siamese architectures learns relative gaze displacements and reconstructs absolute gaze from a small set of calibration frames. In this paper, we
benchmark Siamese personalization on polarization-enabled eye tracking. For benchmarking, we use a 338-subject
dataset captured with a polarization-sensitive camera and 850 nm illumination. We achieve performance comparable to linear calibration with 10-fold fewer samples. Using polarization inputs for Siamese personalization reduces gaze error by up to 12% compared to near-infrared (NIR)-based inputs. Combining Siamese personalization with linear calibration yields further improvements of up to 13% over a linearly calibrated baseline. These results establish Siamese personalization as a practical approach enabling accurate eye tracking.
Read the full abstract ▾
All authors Beyza Kalkanli, Tom Bu, Mahsa Shakeri, Alexander Fix, Dave Stronks, Dmitri Model, Mantas Žurauskas
Computer Vision
PAN Team Institute of Foundation Models: Qiyue Gao, Kun Zhou, Jiannan Xiang et al. (+14 more)
World models (WMs) are intended to serve as internal simulators of the real world that enable
agents to understand, anticipate, and act upon complex environments. Existing WM
benchmarks remain narrowly focused on next-state prediction and visual fidelity, overlooking the richer simulation capabilities required for intelligent behavior. To address this gap, we introduce WR-Arena, a comprehensive
benchmark for evaluating WMs along three fundamental dimensions of next world simulation: (i) Action Simulation Fidelity, the ability to interpret and follow semantically meaningful, multi-step instructions and generate diverse counterfactual rollouts; (ii) Long-horizon Forecast, the ability to sustain accurate, coherent, and physically plausible simulations across extended interactions; and (iii) Simulative
Reasoning and Planning, the ability to support goal-directed
reasoning by simulating, comparing, and selecting among alternative futures in both structured and open-ended environments. We build a task taxonomy and curate diverse
datasets designed to probe these capabilities, moving beyond single-turn and perceptual evaluations. Through extensive experiments with state-of-the-art WMs, our results expose a substantial gap between current models and human-level hypothetical
reasoning, and establish WR-Arena as both a diagnostic tool and a guideline for advancing next-generation world models capable of robust understanding,
forecasting, and purposeful action. The code is available at https://github.com/MBZUAI-IFM/WR-Arena.
Read the full abstract ▾
All authors PAN Team Institute of Foundation Models: Qiyue Gao, Kun Zhou, Jiannan Xiang, Zihan Liu, Dequan Yang, Junrong Chen, Arif Ahmad, Cong Zeng, Ganesh Bannur, Xinqi Huang, Zheqi Liu, Yi Gu, Yichi Yang, Guangyi Liu, Zhiting Hu, Zhengzhong Liu, Eric Xing
Computer Vision
Automated Quality Assessment of Blind Sweep Obstetric Ultrasound for Improved Diagnosis
Prasiddha Bhandari, Kanchan Poudel, Nishant Luitel et al. (+6 more)
Blind Sweep Obstetric Ultrasound (BSOU) enables scalable fetal imaging in low-resource settings by allowing minimally trained operators to acquire standardized sweep
videos for automated Artificial Intelligence(AI) interpretation. However, the reliability of such AI systems depends critically on the quality of the acquired sweeps, and little is known about how deviations from the intended protocol affect downstream predictions. In this work, we present a systematic evaluation of BSOU quality and its impact on three key AI tasks: sweep-tag classification, fetal presentation classification, and placenta-location classification. We simulate plausible acquisition deviations, including reversed sweep direction, probe inversion, and incomplete sweeps, to quantify model robustness, and we develop automated quality-assessment models capable of detecting these perturbations. To approximate real-world deployment, we simulate a feedback loop in which flagged sweeps are re-acquired, showing that such correction improves downstream task performance. Our findings highlight the sensitivity of BSOU-based AI models to acquisition variability and demonstrate that automated quality assessment can play a central role in building reliable, scalable AI-assisted prenatal ultrasound workflows, particularly in low-resource environments.
Read the full abstract ▾
All authors Prasiddha Bhandari, Kanchan Poudel, Nishant Luitel, Bishram Acharya, Angelina Ghimire, Tyler Wellman, Kilian Koepsell, Pradeep Raj Regmi, Bishesh Khanal
Computer Vision
Speech-Synchronized Whiteboard Generation via VLM-Driven Structured Drawing Representations
Suraj Prasad and Pinak Mahapatra
Creating whiteboard-style educational
videos demands precise coordination between freehand illustrations and spoken narration, yet no existing method addresses this
multimodal synchronization problem with structured, reproducible drawing representations. We present the first
dataset of 24 paired Excalidraw demonstrations with narrated audio, where every drawing element carries millisecond-precision creation timestamps spanning 8 STEM domains. Using this data, we study whether a
vision-
language model (Qwen2-VL-7B), fine-tuned via LoRA, can predict full stroke sequences synchronized to
speech from only 24 demonstrations. Our topic-stratified five-fold evaluation reveals that timestamp conditioning significantly improves temporal
alignment over ablated baselines, while the model generalizes across unseen STEM topics. We discuss transferability to real classroom settings and release our
dataset and code to support future research in automated educational content generation.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Learning to Recorrupt: Noise Distribution Agnostic Self-Supervised Image Denoising
Brayan Monroy, Jorge Bacca and Julián Tachella
Self-supervised image denoising methods have traditionally relied on architectural constraints, pseudo-pair constructions, or specialized loss functions to avoid the trivial identity mapping. Among these, approaches such as Noisier2Noise or R2R create training pairs by adding synthetic noise to noisy images. While effective, these recorruption-based approaches require precise knowledge of the noise distribution, which is often unavailable. We present Learning to Recorrupt (L2R), a self-supervised framework that does not require exact knowledge of the noise distribution. Our method introduces a learnable recorruptor jointly optimized with the denoiser through a min–max saddle-point objective. The proposed method achieves state-of-the-art performance among methods without prior knowledge of the noise distribution across unconventional and heavy-tailed noise distributions, such as log-gamma and Laplace, as well as spatially correlated noise, while obtaining competitive results on real-world denoising
benchmarks characterized by signal-dependent noise.
Read the full abstract ▾
Computer Vision
Seeing Through Smoke: Surgical Desmoking for Improved Visual Perception
Jingpei Lu, Fengyi Jiang, Xiaorui Zhang et al. (+2 more)
Minimally invasive and
robot-assisted surgery relies heavily on endoscopic imaging, yet
surgical smoke produced by electrocautery and vessel-sealing instruments can severely degrade visual perception and hinder
vision-based functionalities. We present a
transformer-based
surgical desmoking model with a physics-inspired desmoking head that jointly predicts smoke-free image and corresponding smoke map. To address the scarcity of paired smoky-to-smoke-free training data, we develop a synthetic data generation pipeline that blends artificial smoke patterns with real endoscopic images, yielding over 80,000 paired samples for supervised training. We further curate, to our knowledge, the largest paired
surgical smoke
dataset to date, comprising 5,817 image pairs captured with the da Vinci robotic
surgical system, enabling benchmarking on high-resolution endoscopic images. Extensive experiments on both a public
benchmark and our
dataset demonstrate state-of-the-art performance in image reconstruction compared to existing dehazing and desmoking approaches. We also assess the impact of desmoking on downstream stereo depth estimation and instrument
segmentation, highlighting both the potential benefits and current limitations of digital smoke removal methods.
Read the full abstract ▾
All authors Jingpei Lu, Fengyi Jiang, Xiaorui Zhang, Lingbo Jin, Omid Mohareri
Computer Vision
GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks
Saelyne Yang, Jaesang Yu, Yi-Hao Peng et al. (+4 more)
Graphical User Interface (GUI)
agents have the potential to assist users in interacting with complex software (e.g., PowerPoint, Photoshop). While prior research has primarily focused on automating user actions through clicks and keystrokes, this paradigm overlooks human intention, where users value the ability to explore, iterate, and refine their ideas while maintaining agency. To move beyond automation and toward collaboration, GUI
agents must understand what users are doing and why. We introduce GUIDE (GUI User Intent Detection Evaluation), a
benchmark that evaluates AI models on their ability to perceive user behavior, infer intent, and provide assistance in open-ended GUI tasks. GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations, across 10 software. GUIDE defines three tasks - (i) Behavior State Detection, (ii) Intent Prediction, and (iii) Help Prediction that test a model's ability to recognize behavior state, reason about goals, and decide when and how to help. Evaluations across eight state-of-the-art
multimodal models reveal that all models struggled, achieving only 44.6% and 55.0% accuracy on behavior state and help prediction. However, providing user context significantly improved the performance, raising help prediction by up to 50.2pp, highlighting the critical role of structured user understanding in effective assistance. Our
dataset is available at https://guide-bench.github.io.
Read the full abstract ▾
All authors Saelyne Yang, Jaesang Yu, Yi-Hao Peng, Kevin Qinghong Lin, Jae Won Cho, Yale Song, Juho Kim
Computer Vision
Dynamic LIBRAS Gesture Recognition via CNN over Spatiotemporal Matrix Representation
Jasmine Moreira
This paper proposes a method for dynamic hand gesture recognition based on the composition of two models: the MediaPipe Hand Landmarker, responsible for extracting 21 skeletal keypoints of the hand, and a convolutional neural network (CNN) trained to classify gestures from a spatiotemporal
matrix representation of dimensions 90 by 21 of those keypoints. The method is applied to the recognition of LIBRAS (Brazilian Sign
Language) gestures for device control in a home automation system, covering 11 classes of static and dynamic gestures. For real-time inference, a sliding window with temporal frame triplication is used, enabling continuous recognition without recurrent networks. Tests achieved 95% accuracy under low-light conditions and 92% under normal lighting. The results indicate that the approach is effective, although systematic experiments with greater user diversity are needed for a more thorough evaluation of generalization.
Read the full abstract ▾
Computer Vision
GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understanding
Trong Thang Pham, Hien Nguyen and Ngan Le
Current
multimodal large
language models (MLLMs) cannot effectively utilize eye-gaze information for
video understanding, even when gaze cues are supplied via visual overlays or text descriptions. We introduce GazeQwen, a parameter efficient approach that equips an open-source MLLM with gaze awareness through hidden-state modulation. At its core is a compact gaze resampler (~1-5 M trainable parameters) that encodes V-JEPA 2.1
video features together with fixation-derived positional encodings and produces additive residuals injected into selected LLM decoder layers via forward hooks. An optional second training stage adds low-rank adapters (LoRA) to the LLM for tighter integration. Evaluated on all 10 tasks of the StreamGaze
benchmark, GazeQwen reaches 63.9% accuracy, a +16.1 point gain over the same Qwen2.5-VL-7B backbone with gaze as visual
prompts and +10.5 points over GPT-4o, the highest score among all open-source and proprietary models tested. These results suggest that learning where to inject gaze within an LLM is more effective than scaling model size or engineering better
prompts. All code and checkpoints are available at https://github.com/phamtrongthang123/gazeqwen .
Read the full abstract ▾
Computer Vision
Fus3D: Decoding Consolidated 3D Geometry from Feed-forward Geometry Transformer Latents
Laura Fink, Linus Franke, George Kopanas et al. (+2 more)
We propose a feed-forward method for dense Signed Distance Field (SDF) regression from unstructured image collections in less than three seconds, without camera calibration or post-hoc fusion. Our key insight is that the intermediate feature space of pretrained multi-view feed-forward geometry
transformers already encodes a powerful joint world representation; yet, existing pipelines discard it, routing features through per-view prediction heads before assembling 3D geometry post-hoc, which discards valuable completeness information and accumulates inaccuracies. We instead perform 3D extraction directly from geometry
transformer features via learned volumetric extraction: voxelized canonical
embeddings that progressively absorb multi-view geometry information through interleaved cross- and self-
attention into a structured volumetric latent grid. A simple convolutional decoder then maps this grid to a dense SDF. We additionally propose a scalable, validity-aware supervision scheme directly using SDFs derived from depth maps or 3D assets, tackling practical issues like non-watertight meshes. Our approach yields complete and well-defined distance values across sparse- and dense-view settings and demonstrates geometrically plausible completions. Code and further material can be found at https://lorafib.github.io/fus3d.
Read the full abstract ▾
All authors Laura Fink, Linus Franke, George Kopanas, Marc Stamminger, Peter Hedman
Computer Vision
ViGoR-Bench: How Far Are Visual Generative Models From Zero-Shot Visual Reasoners?
Haonan Han, Jiancheng Huang, Xiaopeng Sun et al. (+7 more)
Beneath the stunning visual fidelity of modern AIGC models lies a "logical desert", where systems fail tasks that require physical, causal, or complex spatial
reasoning. Current evaluations largely rely on superficial metrics or fragmented
benchmarks, creating a “performance mirage” that overlooks the generative process. To address this, we introduce ViGoR
Vision-G}nerative
Reasoning-centric
Benchmark), a unified framework designed to dismantle this mirage. ViGoR distinguishes itself through four key innovations: 1) holistic cross-modal coverage bridging Image-to-Image and
Video tasks; 2) a dual-track mechanism evaluating both intermediate processes and final results; 3) an evidence-grounded automated judge ensuring high human
alignment; and 4) granular diagnostic analysis that decomposes performance into fine-grained cognitive dimensions. Experiments on over 20 leading models reveal that even state-of-the-art systems harbor significant
reasoning deficits, establishing ViGoR as a critical “stress test” for the next generation of intelligent
vision models. The demo have been available at https://vincenthancoder.github.io/ViGoR-Bench/
Read the full abstract ▾
All authors Haonan Han, Jiancheng Huang, Xiaopeng Sun, Junyan He, Rui Yang, Jie Hu, Xiaojiang Peng, Lin Ma, Xiaoming Wei, Xiu Li
Computer Vision
Geo$^\textbf{2}$: Geometry-Guided Cross-view Geo-Localization and Image Synthesis
Yancheng Zhang, Xiaohan Zhang, Guangyu Sun et al. (+3 more)
Cross-view geo-spatial learning consists of two important tasks: Cross-View Geo-Localization (CVGL) and Cross-View Image Synthesis (CVIS), both of which rely on establishing
geometric correspondences between ground and aerial views. Recent
Geometric Foundation Models (GFMs) have demonstrated strong capabilities in extracting generalizable 3D
geometric features from images, but their potential in cross-view geo-spatial tasks remains underexplored. In this work, we present Geo^2, a unified framework that leverages
Geometric priors from GFMs (e.g., VGGT) to jointly perform geo-spatial tasks, CVGL and bidirectional CVIS. Despite the 3D reconstruction ability of GFMs, directly applying them to CVGL and CVIS remains challenging due to the large viewpoint gap between ground and aerial imagery. We propose GeoMap, which embeds ground and aerial features into a shared 3D-aware latent space, effectively reducing cross-view discrepancies for localization. This shared latent space naturally bridges cross-view image synthesis in both directions. To exploit this, we propose GeoFlow, a flow-matching model conditioned on geometry-aware latent
embeddings. We further introduce a consistency loss to enforce latent
alignment between the two synthesis directions, ensuring bidirectional coherence. Extensive experiments on standard
benchmarks, including CVUSA, CVACT, and VIGOR, demonstrate that Geo^2 achieves state-of-the-art performance in both localization and synthesis, highlighting the effectiveness of 3D
geometric priors for cross-view geo-spatial learning.
Read the full abstract ▾
All authors Yancheng Zhang, Xiaohan Zhang, Guangyu Sun, Zonglin Lyu, Safwan Wshah, Chen Chen
Computer Vision
Do All Vision Transformers Need Registers? A Cross-Architectural Reassessment
Spiros Baxevanakis, Platon Karageorgis, Ioannis Dravilas et al. (+1 more)
Training
Vision Transformers (ViTs) presents significant challenges, one of which is the emergence of artifacts in
attention maps, hindering their interpretability. Darcet et al. (2024) investigated this phenomenon and attributed it to the need of ViTs to store global information beyond the [CLS]
token. They proposed a novel solution involving the addition of empty input
tokens, named registers, which successfully eliminate artifacts and improve the clarity of
attention maps. In this work, we reproduce the findings of Darcet et al. (2024) and evaluate the generalizability of their claims across multiple models, including DINO, DINOv2, OpenCLIP, and DeiT3. While we confirm the validity of several of their key claims, our results reveal that some claims do not extend universally to other models. Additionally, we explore the impact of model size, extending their findings to smaller models. Finally, we untie terminology inconsistencies found in the original paper and explain their impact when generalizing to a wider range of models.
Read the full abstract ▾
All authors Spiros Baxevanakis, Platon Karageorgis, Ioannis Dravilas, Konrad Szewczyk
Computer Vision
LEMON: a foundation model for nuclear morphology in Computational Pathology
Loïc Chadoutaud, Alice Blondel, Hana Feki et al. (+3 more)
Computational pathology relies on effective representation learning to support cancer research and precision medicine. Although self-supervised learning has driven major progress at the patch and whole-slide image levels, representation learning at the single-
cell level remains comparatively underexplored, despite its importance for characterizing
cell types and cellular phenotypes. We introduce LEMON (Learning
Embeddings from Morphology Of
Nuclei), a self-supervised foundation model for scalable single-
cell image representation learning. Trained on millions of
cell images from diverse tissues and cancer types, LEMON learns robust and versatile morphological representations that support large-scale single-
cell analyses in pathology. We evaluate LEMON on five
benchmark datasets across a range of prediction tasks and show that it provides strong performance, highlighting its potential as a new paradigm for
cell-level computational pathology. Model weights are available at https://huggingface.co/aliceblondel/LEMON.
Read the full abstract ▾
All authors Loïc Chadoutaud, Alice Blondel, Hana Feki, Jacqueline Fontugne, Emmanuel Barillot, Thomas Walter
Computer Vision
End-to-end Feature Alignment: A Simple CNN with Intrinsic Class Attribution
Parniyan Farvardin and David Chapman
We present Feature-Align CNN (FA-CNN), a prototype CNN architecture with intrinsic class attribution through end-to-end feature
alignment. Our intuition is that the use of unordered operations such as Linear and Conv2D layers cause unnecessary shuffling and mixing of semantic concepts, thereby making raw feature maps difficult to understand. We introduce two new order preserving layers, the dampened skip connection, and the global average pooling classifier head. These layers force the model to maintain an end-to-end feature
alignment from the raw input pixels all the way to final class logits. This end-to-end
alignment enhances the interpretability of the model by allowing the raw feature maps to intrinsically exhibit class attribution. We prove theoretically that FA-CNN penultimate feature maps are identical to Grad-CAM saliency maps. Moreover, we prove that these feature maps slowly morph layer-by-layer over network depth, showing the
evolution of features through network depth toward penultimate class activations. FA-CNN performs well on
benchmark image classification
datasets. Moreover, we compare the averaged FA-CNN raw feature maps against Grad-CAM and permutation methods in a percent pixels removed interpretability task. We conclude this work with a discussion and future, including limitations and extensions toward hybrid models.
Read the full abstract ▾
Computer Vision
ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions
Zikai Wang, Zhilu Zhang, Yiqing Wang et al. (+2 more)
Existing hand-object interactions (HOI) methods are largely limited to rigid objects, while 4D reconstruction methods of articulated objects generally require pre-scanning the object or even multi-view
videos. It remains an unexplored but significant challenge to reconstruct 4D human-articulated-object interactions from a single
monocular RGB
video. Fortunately, recent advancements in foundation models present a new opportunity to address this highly ill-posed problem. To this end, we introduce ArtHOI, an optimization-based framework that integrates and refines priors from multiple foundation models. Our key contribution is a suite of novel methodologies designed to resolve the inherent inaccuracies and physical unreality of these priors. In particular, we introduce an Adaptive Sampling Refinement (ASR) method to optimize object's metric scale and pose for grounding its normalized mesh in world space. Furthermore, we propose a
Multimodal Large
Language Model (MLLM) guided hand-object
alignment method, utilizing contact
reasoning information as constraints of hand-object mesh composition optimization. To facilitate a comprehensive evaluation, we also contribute two new
datasets, ArtHOI-RGBD and ArtHOI-Wild. Extensive experiments validate the robustness and effectiveness of our ArtHOI across diverse objects and interactions. Project: https://arthoi-reconstruction.github.io.
Read the full abstract ▾
All authors Zikai Wang, Zhilu Zhang, Yiqing Wang, Hui Li, Wangmeng Zuo
Computer Vision
Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis
Yuan Zhang, Sihao Dou, Kai Hu et al. (+4 more)
Endoscopic
video analysis is essential for early gastrointestinal screening but remains hindered by limited high-quality annotations. While self-supervised
video pre-training shows promise, existing methods developed for natural
videos prioritize dense spatio-temporal modeling and exhibit motion bias, overlooking the static, structured semantics critical to clinical decision-making. To address this challenge, we propose Focus-to-Perceive Representation Learning (FPRL), a cognition-inspired hierarchical framework that emulates clinical examination. FPRL first focuses on intra-frame lesion-centric regions to learn static semantics, and then perceives their
evolution across frames to model contextual semantics. To achieve this, FPRL employs a hierarchical semantic modeling mechanism that explicitly distinguishes and collaboratively learns both types of semantics. Specifically, it begins by capturing static semantics via teacher-prior adaptive masking (TPAM) combined with multi-view sparse sampling. This approach mitigates redundant temporal dependencies and enables the model to concentrate on lesion-related local semantics. Following this, contextual semantics are derived through cross-view masked feature completion (CVMFC) and
attention-guided temporal prediction (AGTP). These processes establish cross-view correspondences and effectively model structured inter-frame
evolution, thereby reinforcing temporal semantic continuity while preserving global contextual integrity. Extensive experiments on 11 endoscopic
video datasets show that FPRL achieves superior performance across diverse downstream tasks, demonstrating its effectiveness in endoscopic
video representation learning. The code is available at https://github.com/MLMIP/FPRL.
Read the full abstract ▾
All authors Yuan Zhang, Sihao Dou, Kai Hu, Shuhua Deng, Chunhong Cao, Fen Xiao, Xieping Gao
Computer Vision
Evaluating Synthetic Images as Effective Substitutes for Experimental Data in Surface Roughness Classification
Binwei Chen, Huachao Leng, Chi Yeung Mang et al. (+5 more)
Hard coatings play a critical role in industry, with ceramic materials offering outstanding hardness and thermal stability for applications that demand superior mechanical performance. However, deploying artificial intelligence (AI) for surface roughness classification is often constrained by the need for large labeled
datasets and costly high-resolution imaging equipment. In this study, we explore the use of synthetic images, generated with Stable
Diffusion XL, as an efficient alternative or supplement to experimentally acquired data for classifying ceramic surface roughness. We show that augmenting authentic
datasets with generative images yields test accuracies comparable to those obtained using exclusively experimental images, demonstrating that synthetic images effectively reproduce the structural features necessary for classification. We further assess method robustness by systematically varying key training hyperparameters (epoch count, batch size, and learning rate), and identify configurations that preserve performance while reducing data requirements. Our results indicate that generative AI can substantially improve data efficiency and reliability in materials-image classification workflows, offering a practical route to lower experimental cost, accelerate model development, and expand AI applicability in materials engineering.
Read the full abstract ▾
All authors Binwei Chen, Huachao Leng, Chi Yeung Mang, Tsz Wai Cheung, Yanhua Chen, Wai Keung Anthony Loh, Chi Ho Wong, Chak Yin Tang
Computer Vision
A Survey of OCR Evaluation Methods and Metrics and the Invisibility of Historical Documents
Fitsum Sileshi Beyene and Christopher L. Dancy
Optical character recognition (OCR) and document understanding systems increasingly rely on large
vision and
vision-
language models, yet evaluation remains centered on modern, Western, and institutional documents. This emphasis masks system behavior in historical and marginalized archives, where layout, typography, and material degradation shape interpretation. This study examines how OCR and document understanding systems are evaluated, with particular
attention to Black historical newspapers. We review OCR and document understanding papers, as well as
benchmark datasets, which are published between 2006 and 2025 using the PRISMA framework. We look into how the studies report training data,
benchmark design, and evaluation metrics for
vision transformer and
multimodal OCR systems. During the review, we found that Black newspapers and other community-produced historical documents rarely appear in reported training data or evaluation
benchmarks. Most evaluations emphasize character accuracy and task success on modern layouts. They rarely capture structural failures common in historical newspapers, including column collapse, typographic errors, and hallucinated text. To put these findings into perspective, we use previous empirical studies and archival statistics from significant Black press collections to show how evaluation gaps lead to structural invisibility and representational harm. We propose that these gaps occur due to organizational (meso) and institutional (macro) behaviors and structure, shaped by
benchmark incentives and data governance decisions.
Read the full abstract ▾
Computer Vision
ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytelling
Yawen Luo, Xiaoyu Shi, Junhao Zhuang et al. (+5 more)
Multi-shot
video generation is crucial for long narrative storytelling, yet current bidirectional architectures suffer from limited interactivity and high latency. We propose ShotStream, a novel causal multi-shot architecture that enables interactive storytelling and efficient on-the-fly frame generation. By reformulating the task as next-shot generation conditioned on historical context, ShotStream allows users to dynamically instruct ongoing narratives via streaming
prompts. We achieve this by first fine-tuning a text-to-
video model into a bidirectional next-shot generator, which is then distilled into a causal student via Distribution Matching
Distillation. To overcome the challenges of inter-shot consistency and error accumulation inherent in autoregressive generation, we introduce two key innovations. First, a dual-cache
memory mechanism preserves visual coherence: a global context cache retains conditional frames for inter-shot consistency, while a local context cache holds generated frames within the current shot for intra-shot consistency. And a RoPE discontinuity indicator is employed to explicitly distinguish the two caches to eliminate ambiguity. Second, to mitigate error accumulation, we propose a two-stage
distillation strategy. This begins with intra-shot self-forcing conditioned on ground-truth historical shots and progressively extends to inter-shot self-forcing using self-generated histories, effectively bridging the train-test gap. Extensive experiments demonstrate that ShotStream generates coherent multi-shot
videos with sub-second latency, achieving 16 FPS on a single GPU. It matches or exceeds the quality of slower bidirectional models, paving the way for real-time interactive storytelling. Training and inference code, as well as the models, are available on our
Read the full abstract ▾
All authors Yawen Luo, Xiaoyu Shi, Junhao Zhuang, Yutian Chen, Quande Liu, Xintao Wang, Pengfei Wan, Tianfan Xue
Computer Vision
Less Gaussians, Texture More: 4K Feed-Forward Textured Splatting
Yixing Lao, Xuyang Bai, Xiaoyang Wu et al. (+7 more)
Existing feed-forward 3D Gaussian Splatting methods predict pixel-aligned primitives, leading to a quadratic growth in primitive count as resolution increases. This fundamentally limits their scalability, making high-resolution synthesis such as 4K intractable. We introduce LGTM (Less Gaussians, Texture More), a feed-forward framework that overcomes this resolution scaling barrier. By predicting compact Gaussian primitives coupled with per-primitive textures, LGTM decouples
geometric complexity from rendering resolution. This approach enables high-fidelity 4K novel view synthesis without per-scene optimization, a capability previously out of reach for feed-forward methods, all while using significantly fewer Gaussian primitives. Project page: https://yxlao.github.io/lgtm/
Read the full abstract ▾
All authors Yixing Lao, Xuyang Bai, Xiaoyang Wu, Nuoyuan Yan, Zixin Luo, Tian Fang, Jean-Daniel Nahmias, Yanghai Tsin, Shiwei Li, Hengshuang Zhao
Computer Vision
MuRF: Unlocking the Multi-Scale Potential of Vision Foundation Models
Bocheng Zou, Mu Cai, Mark Stanley et al. (+2 more)
Vision Foundation Models (VFMs) have become the cornerstone of modern computer
vision, offering robust representations across a wide array of tasks. While recent advances allow these models to handle varying input sizes during training, inference typically remains restricted to a single, fixed scale. This prevalent single-scale paradigm overlooks a fundamental property of visual perception: varying resolutions offer complementary inductive biases, where low-resolution views excel at global semantic recognition and high-resolution views are essential for fine-grained refinement. In this work, we propose Multi-Resolution Fusion (MuRF), a simple yet universally effective strategy to harness this synergy at inference time. Instead of relying on a single view, MuRF constructs a unified representation by processing an image at multiple resolutions through a frozen VFM and fusing the resulting features. The universality of MuRF is its most compelling attribute. It is not tied to a specific architecture, serving instead as a fundamental, training-free enhancement to visual representation. We empirically validate this by applying MuRF to a broad spectrum of critical computer
vision tasks across multiple distinct VFM families - primarily DINOv2, but also demonstrating successful generalization to contrastive models like SigLIP2.
Read the full abstract ▾
All authors Bocheng Zou, Mu Cai, Mark Stanley, Dingfu Lu, Yong Jae Lee
Computer Vision
RefAlign: Representation Alignment for Reference-to-Video Generation
Lei Wang, YuXin Song, Ge Wu et al. (+5 more)
Reference-to-
video (R2V) generation is a controllable
video synthesis paradigm that constrains the generation process using both text
prompts and reference images, enabling applications such as personalized advertising and virtual try-on. In practice, existing R2V methods typically introduce additional high-level semantic or cross-modal features alongside the VAE latent representation of the reference image and jointly feed them into the
diffusion Transformer (DiT). These auxiliary representations provide semantic guidance and act as implicit
alignment signals, which can partially alleviate pixel-level information leakage in the VAE latent space. However, they may still struggle to address copy–paste artifacts and multi-subject confusion caused by modality mismatch across heterogeneous encoder features. In this paper, we propose RefAlign, a representation
alignment framework that explicitly aligns DiT reference-branch features to the semantic space of a visual foundation model (VFM). The core of RefAlign is a reference
alignment loss that pulls the reference features and VFM features of the same subject closer to improve identity consistency, while pushing apart the corresponding features of different subjects to enhance semantic discriminability. This simple yet effective strategy is applied only during training, incurring no inference-time overhead, and achieves a better balance between text controllability and reference fidelity. Extensive experiments on the OpenS2V-Eval
benchmark demonstrate that RefAlign outperforms current state-of-the-art methods in TotalScore, validating the effectiveness of explicit reference
alignment for R2V tasks.
Read the full abstract ▾
All authors Lei Wang, YuXin Song, Ge Wu, Haocheng Feng, Hang Zhou, Jingdong Wang, Yaxing Wang, Jian Yang
Computer Vision
Vega: Learning to Drive with Natural Language Instructions
Sicheng Zuo, Yuxuan Li, Wenzhao Zheng et al. (+3 more)
Vision-
language-action models have reshaped autonomous driving to incorporate
languages into the decision-making process. However, most existing pipelines only utilize the
language modality for scene descriptions or
reasoning and lack the flexibility to follow diverse user instructions for personalized driving. To address this, we first construct a large-scale driving
dataset (InstructScene) containing around 100,000 scenes annotated with diverse driving instructions with the corresponding trajectories. We then propose a unified
Vision-
Language-World-Action model, Vega, for instruction-based generation and planning. We employ the autoregressive paradigm to process visual inputs (
vision) and
language instructions (
language) and the
diffusion paradigm to generate future predictions (world modeling) and trajectories (action). We perform joint
attention to enable interactions between the modalities and use individual projection layers for different modalities for more capabilities. Extensive experiments demonstrate that our method not only achieves superior planning performance but also exhibits strong instruction-following abilities, paving the way for more intelligent and personalized driving systems.
Read the full abstract ▾
All authors Sicheng Zuo, Yuxuan Li, Wenzhao Zheng, Zheng Zhu, Jie Zhou, Jiwen Lu
Computer Vision · Cross-listed from cs.RO
Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Driving
Zehao Wang, Huaide Jiang, Shuaiwu Dong et al. (+3 more)
Human driving behavior is inherently personal, which is shaped by long-term habits and influenced by short-term intentions. Individuals differ in how they accelerate, brake, merge, yield, and overtake across diverse situations. However, existing end-to-end autonomous driving systems either optimize for generic objectives or rely on fixed driving modes, lacking the ability to adapt to individual preferences or interpret natural
language intent. To address this gap, we propose Drive My Way (DMW), a personalized
Vision-
Language-Action (VLA) driving framework that aligns with users' long-term driving habits and adapts to real-time user instructions. DMW learns a user
embedding from our personalized driving
dataset collected across multiple real drivers and conditions the policy on this
embedding during planning, while natural
language instructions provide additional short-term guidance. Closed-loop evaluation on the Bench2Drive
benchmark demonstrates that DMW improves style instruction adaptation, and user studies show that its generated behaviors are recognizable as each driver's own style, highlighting personalization as a key capability for human-centered autonomous driving. Our data and code are available at https://dmw-cvpr.github.io/.
Read the full abstract ▾
All authors Zehao Wang, Huaide Jiang, Shuaiwu Dong, Yuping Wang, Hang Qiu, Jiachen Li
Computer Vision
MegaFlow: Zero-Shot Large Displacement Optical Flow
Dingxi Zhang, Fangjinhua Wang, Marc Pollefeys et al. (+1 more)
Accurate estimation of large displacement optical flow remains a critical challenge. Existing methods typically rely on iterative local search or/and domain-specific fine-tuning, which severely limits their performance in large displacement and zero-shot generalization scenarios. To overcome this, we introduce MegaFlow, a simple yet powerful model for zero-shot large displacement optical flow. Rather than relying on highly complex, task-specific architectural designs, MegaFlow adapts powerful pre-trained
vision priors to produce temporally consistent motion fields. In particular, we formulate flow estimation as a global matching problem by leveraging pre-trained global
Vision Transformer features, which naturally capture large displacements. This is followed by a few lightweight iterative refinements to further improve the sub-pixel accuracy. Extensive experiments demonstrate that MegaFlow achieves state-of-the-art zero-shot performance across multiple optical flow
benchmarks. Moreover, our model also delivers highly competitive zero-shot performance on long-range point tracking
benchmarks, demonstrating its robust transferability and suggesting a unified paradigm for generalizable motion estimation. Our project page is at: https://kristen-z.github.io/projects/megaflow.
Read the full abstract ▾
All authors Dingxi Zhang, Fangjinhua Wang, Marc Pollefeys, Haofei Xu
Computer Vision
PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow
Xincheng Shuai, Song Tang, Yutong Huang et al. (+2 more)
Graphic design is a creative and innovative process that plays a crucial role in applications such as e-commerce and advertising. However, developing an automated design system that can faithfully translate user intentions into editable design files remains an open challenge. Although recent studies have leveraged powerful text-to-image models and MLLMs to assist graphic design, they typically simplify professional workflows, resulting in limited flexibility and intuitiveness. To address these limitations, we propose PSDesigner, an automated graphic design system that emulates the creative workflow of human designers. Building upon multiple specialized components, PSDesigner collects theme-related assets based on user instructions, and autonomously infers and executes tool calls to manipulate design files, such as integrating new assets or refining inferior elements. To endow the system with strong tool-use capabilities, we construct a design
dataset, CreativePSD, which contains a large amount of high-quality PSD design files annotated with operation traces across a wide range of design scenarios and artistic styles, enabling models to learn expert design procedures. Extensive experiments demonstrate that PSDesigner outperforms existing methods across diverse graphic design tasks, empowering non-specialists to conveniently create production-quality designs.
Read the full abstract ▾
All authors Xincheng Shuai, Song Tang, Yutong Huang, Henghui Ding, Dacheng Tao
Computer Vision
How good was my shot? Quantifying Player Skill Level in Table Tennis
Akihiro Kubota, Tomoya Hasegawa, Ryo Kawahara et al. (+1 more)
Gauging an individual's skill level is crucial, as it inherently shapes their behavior. Quantifying skill, however, is challenging because it is latent to the observed actions. To explore skill understanding in human behavior, we focus on dyadic sports – specifically table tennis – where skill manifests not just in complex movements, but in the subtle nuances of execution conditioned on game context. Our key idea is to learn a generative model of each player's tactical racket strokes and jointly embed them in a common latent space that encodes individual characteristics, including those pertaining to skill levels. By training these player models on a large-scale
dataset of 3D-reconstructed professional matches and conditioning them on comprehensive game context – including player positioning and opponent behaviors – the models capture individual tactical identities within their latent space. We probe this learned player space and find that it reflects distinct play styles and attributes that collectively represent skill. By training a simple relative ranking network on these
embeddings, we demonstrate that both relative and absolute skill predictions can be achieved. These results demonstrate that the learned player space effectively quantifies skill levels, providing a foundation for automated skill assessment in complex, interactive behaviors.
Read the full abstract ▾
All authors Akihiro Kubota, Tomoya Hasegawa, Ryo Kawahara, Ko Nishino
Computer Vision
Unleashing Guidance Without Classifiers for Human-Object Interaction Animation
Ziyin Wang, Sirui Xu, Chuan Guo et al. (+5 more)
Generating realistic human-object interaction (HOI) animations remains challenging because it requires jointly modeling dynamic human actions and diverse object geometries. Prior
diffusion-based approaches often rely on hand-crafted contact priors or human-imposed kinematic constraints to improve contact quality. We propose LIGHT, a data-driven alternative in which guidance emerges from the denoising pace itself, reducing dependence on manually designed priors. Building on
diffusion forcing, we factor the representation into modality-specific components and assign individualized noise levels with asynchronous denoising schedules. In this paradigm, cleaner components guide noisier ones through cross-
attention, yielding guidance without auxiliary classifiers. We find that this data-driven guidance is inherently contact-aware, and can be enhanced when training is augmented with a broad spectrum of synthetic object geometries, encouraging invariance of contact semantics to
geometric diversity. Extensive experiments show that pace-induced guidance more effectively mirrors the benefits of contact priors than conventional classifier-free guidance, while achieving higher contact fidelity, more realistic HOI generation, and stronger generalization to unseen objects and tasks.
Read the full abstract ▾
All authors Ziyin Wang, Sirui Xu, Chuan Guo, Bing Zhou, Jiangshan Gong, Jian Wang, Yu-Xiong Wang, Liang-Yan Gui
Computer Vision
SlotVTG: Object-Centric Adapter for Generalizable Video Temporal Grounding
Jiwook Han, Geo Ahn, Youngrae Kim et al. (+1 more)
Multimodal Large
Language Models (MLLMs) have shown strong performance on
Video Temporal Grounding (VTG). However, their coarse recognition capabilities are insufficient for fine-grained temporal understanding, making task-specific fine-tuning indispensable. This fine-tuning causes models to memorize
dataset-specific shortcuts rather than faithfully grounding in the actual visual content, leading to poor Out-of-Domain (OOD) generalization. Object-centric learning offers a promising remedy by decomposing scenes into entity-level representations, but existing approaches require re-running the entire multi-stage training pipeline from scratch. We propose SlotVTG, a framework that steers MLLMs toward object-centric, input-grounded visual
reasoning at minimal cost. SlotVTG introduces a lightweight slot adapter that decomposes visual
tokens into abstract slots via slot
attention and reconstructs the original sequence, where objectness priors from a self-supervised
vision model encourage semantically coherent slot formation. Cross-domain evaluation on standard VTG
benchmarks demonstrates that our approach significantly improves OOD robustness while maintaining competitive In-Domain (ID) performance with minimal overhead.
Read the full abstract ▾
All authors Jiwook Han, Geo Ahn, Youngrae Kim, Jinwoo Choi