Computer Vision
Echo-Memory: A Controlled Study of Memory in Action World Models
Wayne King, Zeyue Xue, Yuxuan Bian et al. (+13 more)
We present Echo-
Memory, a controlled study of
memory mechanisms in action-conditioned world models. These models generate multi-segment
videos from a first frame, text
prompt, and camera-action sequence, but their central failure is often
memory rather than local image synthesis: after the camera leaves and returns, the scene or salient object may silently change. Existing
memory designs are hard to compare because gains are entangled with backbone, training,
retrieval, and evaluation differences. Echo-
Memory fixes the action-to-
video interface and varies only how history is stored and read by the generator. Under a shared
video diffusion backbone, optimizer, camera-action representation, sampler, and evaluation pipeline, we compare raw context, compression-based
memory, spatial summaries with different read-out paths, and state-space recurrence. This matched
matrix separates four otherwise conflated axes: capacity, compression, read-out, and recurrence. We also evaluate
memory through a three-branch protocol: replay quality, in-domain loop revisit, and open-domain return probes. The branches routinely disagree, showing that replay fidelity is not a sufficient proxy for remembering a world. Three findings follow. Raw context is a strong capacity baseline and improves open-domain return far more than it improves replay metrics. Compactness is not a free substitute for capacity: aggressive spatial and hybrid-compression
memories lose the salient evidence needed for return. Finally, block-wise state-space recurrence is the strongest open-domain return mechanism in our
matrix, showing that the structure of implicit
memory matters as much as the decision to use it. These results provide a compact protocol for studying
memory in action world models beyond isolated replay metrics.
Read the full abstract ▾
All authors Wayne King, Zeyue Xue, Yuxuan Bian, Jie Huang, Haoran Li, Yaowei Li, Yaofeng Su, Yuming Li, Haoyu Wang, Shiyi Zhang, Songchun Zhang, Yuwei Niu, Sihan Xu, Junhao Zhuang, Haoyang Huang, Nan Duan
Computer Vision
Beyond Spherical Harmonics: Rethinking Appearance Models for Radiance Reconstruction
Ewa Miazga, Jorge Condor and Piotr Didyk
View-dependent appearance modeling remains a challenging problem in novel-view synthesis and reconstruction. Accurately representing complex angular effects often requires substantial
memory and computational resources. For new learning-based methods, a common approach is to rely on SH. However, capturing high-frequency phenomena such as specular reflections demands high-order expansions, which increase
memory usage and computational cost. Consequently, most methods employ low-order SH, which limits the ability to model complex view-dependent effects, resulting in overly smooth or diffuse representations. To address these limitations, we systematically evaluate a wide range of spherical functions in the context of scene reconstruction. Some of them are introduced to graphics and computer
vision for the first time in this paper. Based on the insights from the experiment, we develop a novel spherical formulation, the Normalized Anisotropic Spherical Gabor function that enables efficient modeling and learning of high-frequency appearance effects while maintaining compact representation. Compared to existing approaches, our function achieves higher-quality reconstruction of view-dependent phenomena such as glints, while being up to five times more
memory-efficient and more efficient to evaluate. We validate its performance in radiance-field reconstruction tasks.
Read the full abstract ▾
Computer Vision
End-to-End Optimization of Incoherent Imaging for Classification Under Detector-Limited Readout
Archer Wang, Joshua Chen, Sachin Vaidya et al. (+1 more)
End-to-end co-optimization of optical front-ends (e.g. metasurfaces) and neural network back-ends has been widely applied to imaging tasks, yet a formalism characterizing when and why such systems outperform conventional lens-based imaging is largely lacking. This paper focuses on object classification, a central imaging task, and asks when end-to-end optimization of a phase mask for incoherent imaging improves performance over a conventional focusing lens. We find that these gains arise primarily under constrained detector readout and are limited under full detector readout. In the latter setting, we prove that no incoherent phase mask exceeds the ideal-channel mutual information between detector measurements and class labels; a conventional focusing lens approaches this ceiling, and joint optimization yields no empirical gain. When detector readout is constrained – by coarse spatial sampling or a limited number of measurements – optimized optics can substantially improve classification by increasing class separability in the detector measurements. These gains are largest under low detector noise and shrink as noise grows, because the optics shape the signal before it reaches the detector but cannot remove noise added afterward. The advantage also depends on the spectral structure of the task: co-design helps most when class-discriminative content is concentrated at lower spatial frequencies than within-class variation. We develop a theoretical framework formalizing these distinctions and test its predictions on synthetic data and standard
benchmarks (MNIST, FashionMNIST, SVHN).
Read the full abstract ▾
All authors Archer Wang, Joshua Chen, Sachin Vaidya, Marin Soljačić
Computer Vision
POTATR: A Lightweight Image-to-Graph Model for Page-Level Table Extraction
Brandon Smock, Libin Liang, Maxim Sokolov et al. (+4 more)
Large-scale document processing requires contextually aware table extraction (TE) that is both accurate and efficient. Yet current approaches require billions of parameters, hundreds of autoregressive steps, or costly API inference. Motivated by this, we introduce the Page-Object Table
Transformer (POTATR), a lightweight 29M parameter image-to-graph model that extends the Table
Transformer (TATR) for contextualized page-level TE. POTATR outperforms all models tested on the PubTables-v2 Single Pages
benchmark – including frontier MLLMs – achieving $\textrm{GriTS}_\textrm{Con}$ of 0.964 while running over 130$\times$ faster at 300$\times$ lower cost. Further, POTATR's output is spatially grounded: every recognized element has a bounding box, enabling visual verification and
geometric text assignment. As a result, POTATR performs unified page-level TE while composing with other models, enabling extension to scanned documents via external OCR and to full-document TE via techniques like cross-page merging. Code and models are available at https://github.com/kensho-technologies/POTATR.
Read the full abstract ▾
All authors Brandon Smock, Libin Liang, Maxim Sokolov, Amrit Ramesh, Valerie Faucon-Morin, Tayyibah Khanam, Maury Courtland
Computer Vision
SemDINO: Foundation Prior-Guided Cross-Temporal Semantic Alignment Network for Remote Sensing Change Detection
Xinyu Tong, Meihua Zhou, Jinxiao Sun et al. (+3 more)
Semantic change detection (SCD) in remote sensing aims to identify land-cover transitions between bi-temporal observations while suppressing pseudo-changes caused by illumination variations, seasonal differences, and registration errors. Although
Vision Foundation Models (VFMs) provide transferable semantic priors, their application to SCD remains challenging due to the mismatch between foundation-model representations and task-specific spatial features, as well as temporal-order sensitivity. To address these issues, this paper proposes SemDINO, a foundation prior-guided framework that integrates transferable
vision foundation model priors with hierarchical convolutional representations for cross-temporal semantic
reasoning. Specifically, a Gated Pyramid Fusion (PyFu) module is developed to adaptively combine foundation-model semantics with CNN spatial details while reducing domain noise. A Multi-scale Temporal Bi-directional
Transformer (M-TBTT) is introduced to achieve symmetric cross-temporal feature interaction and alleviate temporal-order bias. Furthermore, a Feature Change Enhancement (FeaCE) flow is designed to refine aligned representations and distinguish genuine semantic transitions from pseudo variations. Finally, a multi-branch decoupled prediction head jointly generates change masks, bi-temporal semantic maps, and edge constraints. Extensive experiments across five
benchmark datasets demonstrate that SemDINO consistently outperforms state-of-the-art methods on both semantic and binary change detection tasks. The results validate the effectiveness of
alignment-oriented representation learning for robust remote sensing change analysis.
Read the full abstract ▾
All authors Xinyu Tong, Meihua Zhou, Jinxiao Sun, Zaiyan Zhang, Hongruixuan Chen, Lei Wang
Computer Vision
Hybrid Robustness Verification for Spatio-Temporal Neural Networks
Sherwin Varghese, Matthew Wicker and Alessio Lomuscio
With AI increasingly deployed in safety-critical systems, providing formal robustness guarantees for the underlying models is essential. Existing verification methods either rely on overly conservative approximations or incur prohibitive computational costs. For example, the use of lp-norm perturbations in
video settings encodes the belief that the adversary can inject noise in every
video frame. In practice,
adversarial perturbations exhibit structured spatial and temporal correlations, constrained to lower-dimensional, semantically meaningful subspaces. In this work, we study robustness verification of 3D CNNs processing
video and volumetric inputs, targeting applications in action recognition (UCF-101), autonomous driving (Udacity), and medical imaging (MedMNIST) exploiting realistic assumptions on
adversarial strength by modelling them as spatio-temporal constraints - where the attacker can modify either a subset of frames or patches within a set of consecutive frames. We demonstrate that modelling realistic constraints enables tighter approximations. We introduce Spatio-Temporal Bound Propagation (STBP), a verification framework that computes an exact closed-form characterization of the first convolutional layer and propagates certified bounds through subsequent layers using scalable approximations. Computing the exact closed form provides the tightest bounds for the first convolutional layer. Thus, we utilise approximation methods in the remainder of the network. To spur further progress in this field, we propose ST-Bench, a verification
benchmark for autonomous driving and activity recognition, to systematically evaluate verifiable robustness. Compared to existing verification-based approaches, STBP provides stronger robustness guarantees with significantly improved scalability, achieving 1.7x higher certified robust accuracy under identical perturbation budgets.
Read the full abstract ▾
Computer Vision
HDSL: A Hierarchical Domain-Specific Language for Structured 3D Indoor Scene Generation and Localized Editing with LLM Agents
Letian Li, Chao Shen, Shuzhao Xie et al. (+6 more)
Text-driven indoor scene generation and editing require an intermediate representation that
language models can both produce and revise. Existing LLM-based systems often rely on scene graphs or global constraint lists, which are compact but underspecify local geometry and make instruction-based edits difficult to localize. We frame this problem as structured program generation and local program repair, and propose Hierarchical Descriptive Scene
Language (HDSL), an XML/CSS-style domain-specific
language for structured 3D indoor scenes. HDSL represents rooms, regions, objects, and support surfaces as a tree with local coordinates, making complex scenes easier to plan recursively and easier to retrieve for editing. Our pipeline uses LLM
agents to generate HDSL subtrees with bounded verification, grounds non-virtual nodes through
multimodal asset
retrieval, and applies force-directed layout optimization to repair boundary and collision errors. For editing, Hierarchical
Retrieval-Augmented Generation retrieves the relevant subtree, asks the LLM to rewrite only that local context, and merges the result back through a deterministic three-way merge. In our reproduced
benchmark, HDSL improves average object coverage, text-scene
alignment, and generation time over full text-to-scene baselines while remaining competitive with recent layout-only reproductions on geometry metrics; for editing, HRAG reduces
token use by $5.22\times$ and runtime by $6.19\times$, produces valid DSL for all eight paired edits, and better preserves unrelated scene objects.
Read the full abstract ▾
All authors Letian Li, Chao Shen, Shuzhao Xie, Chenghao Gu, ZhengXiao He, Yu Meng, Xin Yang, Wenyuan Jiang, Zhi Wang
Computer Vision · Cross-listed from cs.LG
Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles
Xiao Li, Yixuan Jia, Zekai Zhang et al. (+6 more)
Diffusion models have demonstrated remarkable generative capabilities and have also emerged as powerful self-supervised representation learners, yet the connection between these two abilities remains less explored. Drawing inspiration from self-supervised learning (SSL), we introduce a framework for jointly evaluating the representation and generation capabilities of
diffusion models. Specifically, we decompose features into invariant and residual components and derive the Invariant Contamination Ratio (ICR), a Fisher-based metric that quantifies how residual variation contaminates invariant signal in feature space. We use this framework to analyze both discriminative and generative behavior of
diffusion models. On the representation side, we find that invariance peaks at intermediate noise levels, which also yield the best downstream classification performance. On the generative side, we study how training transitions from genuine generalization to memorization in data-limited regimes, and show that ICR serves as a sensitive training-time indicator of early learning: increasing residual energy along Fisher directions marks the onset of memorization, detectable from training features alone without external evaluators or held-out test sets. Overall, our results show that
diffusion models can be monitored from a self-supervised perspective through the geometry of their learned representations.
Read the full abstract ▾
All authors Xiao Li, Yixuan Jia, Zekai Zhang, Xiang Li, Lianghe Shi, Jinxin Zhou, Zhihui Zhu, Liyue Shen, Qing Qu
Computer Vision
Cranio-Diff: Diffusion-based Cross-domain Craniofacial Reconstruction with 2D X-ray Skull Guidance and Structural Identity Constraints
Ravi Shankar Prasad, Naresh Gurjar, Shashank Baghel et al. (+2 more)
The identification of unknown human remains is a persistent humanitarian and forensic challenge. In forensic investigations, it becomes even more challenging when common biometric DNA identifications, such as soft-tissue or hair strands are not present, as they degrade with time. To narrow down the process of identification, few attempts have been made to identify an individual by reconstructing a face from the given skeletal remains ( i.e., skull) using generative AI models, such as CycleGAN, Pix2Pix, and
diffusion models. However, they fail to effectively capture semantic information in craniofacial reconstruction, due to a mismatch in the
alignment of structural information across modalities. To address this issue, we propose Cranio-Diff, a
diffusion-based framework for craniofacial reconstruction from 2D X-ray skull images. The proposed approach integrates skull-conditioned structural guidance through ControlNet with biometric text
prompts to generate a face. The proposed Cranio-Diff method is evaluated on skull-face
dataset obtained from X-ray scans of 120 subjects in lateral and frontal views. To enable controlled evaluation, each face image is artificially synthesized across three age groups (25, 45, 65) and three BMI variations 10%, yielding 4320 paired samples. This
dataset is also curated across different facial modalities ( RGB and sketch) with different appearance conditions (hair and without hair). To the best of our knowledge, this is the only X-ray-face
dataset with this magnitude. Extensive experiments showed that the proposed method achieves improved performance over the recent existing approaches in both generated image quality and
retrieval task. Finally, to evaluate the performance of our proposed method, we have evaluated the quality of the generated image using FID, SSIM, LPIPS score.
Read the full abstract ▾
All authors Ravi Shankar Prasad, Naresh Gurjar, Shashank Baghel, Chirag, Dinesh Singh
Computer Vision
GenEyePose: Patient-Free, Knowledge-Based Saccadic Eye Movement Modeling for Digital Neurophysiologic Biomarker Development
Tianyu Lin, Jooyoung Ryu, Puvada Sreevarsha et al. (+12 more)
Eye movements, including saccades, are widely regarded as highly sensitive and objective biomarkers of neurophysiologic states. Detecting saccadic signatures in neurologic diseases offers a rapid, portable alternative to
brain imaging, avoiding access and cost barriers. Currently, there are no robust AI-enabled
video-oculographic solutions (e.g., digital biomarkers) for screening, triaging, or localizing
brain abnormalities due to
privacy issues and scarce
datasets. In this work, we propose the first fully synthetic, patient-free,
multimodal eye movement generation pipeline for generalizable saccade analysis. Using this synthetic
dataset, we trained a deep learning classifier to distinguish between normal and abnormal (hypometria and hypermetria) saccadic accuracies and evaluated its performance on real-world clinical data. The model achieved an AUROC of 0.76 and a sensitivity of 0.71, showing that the synthetic data has strong potential to generalize for clinical applications, including as a screening tool in at-home and emergency room settings or a tool for precise neuroanatomic localization.
Read the full abstract ▾
All authors Tianyu Lin, Jooyoung Ryu, Puvada Sreevarsha, Rahul Srinivasaragavan, Riya Satavlekar, Susan Kim, Nidhi Soley, Yujie Yan, Ishan Vatsaraj, Carl Harris, Aimon Rahman, Vishal Patel, Joseph Greenstein, Casey Taylor, Kemar E. Green
Computer Vision
SoccerNet 2026 Player-Centric Ball-Action Spotting:Retraining and Post-Processing Extensions to the FOOTPASS Baselines
Parthsarthi Rawat
We describe our system for the SoccerNet 2026 Player-Centric Ball-Action Spotting Challenge, which requires predicting who performs which action and when, across eight classes in broadcast soccer. Building on the three FOOTPASS baselines [1] (TAAD, TAAD+GNN, and TAAD+DST), we contribute four extensions: (1) gradient check pointing to enable full-backbone fine-tuning on a single GPU; (2) fusion of GNN logits into the DST encoder, combining graph-based tactical context with per-player visual features; (3) square-root frequency class weighting to address the 213:1 pass-to-tackle imbalance in the training data; and (4) a post processing pipeline comprising per-class logit gating, temporal frame refinement, jersey re-assignment, and a two-model ensemble. Our system achieves 0.548 Macro F1 on the test set and 0.446 on the challenge set (server evaluation).
Read the full abstract ▾
Computer Vision
Visual Prompting Meets Feature Reconstruction-Based Anomaly Detection with Dual-Teacher Supervision
Mateo Diaz-Bone, Daniel Caraballo, Florian Scheidegger et al. (+11 more)
Recent Anomaly Detection methods achieve perfect detection and
segmentation scores on well-established
datasets, such as MVTec. However, many of these methods face challenges when foundational assumptions - such as consistent object scale, viewpoint, background, illumination, and centered placement - are violated. Those variations that occur render anomaly detection methods unusable in many real-world scenarios. To address these limitations, we introduce three key contributions: (1) a visual prompting pipeline that isolates objects using foreground-background masking; (2) a mechanism for unfreezing the teacher in student-teacher models to improve domain adaptability; and (3) a data augmentation strategy leveraging
diffusion-generated synthetic images to enhance anomaly detection performance. We achieve a 3.5 percentage point improvement over the previous state-of-the-art on the challenging AeBAD
dataset by using the Masked Multiscale Reconstruction (MMR) model as our backbone.
Read the full abstract ▾
All authors Mateo Diaz-Bone, Daniel Caraballo, Florian Scheidegger, Thomas Frick, Mattia Rigotti, Andrea Bartezzaghi, Roy Assaf, Niccolo Avogaro, Yagmur G. Cinar, Brown Ebouky, Filip M. Janicki, Piotr S. Kluska, Cezary Skura, Cristiano Malossi
Computer Vision
How Do Video Foundation Models Encode Intuitive Physics? Probing Across Pretraining Paradigms
Samuele Punzo, Niccolò Caselli, Ippokratis Pantelidis et al. (+3 more)
Do
video foundation models encode a usable sense of how the physical world works, and does intuitive physics emerge naturally from large-scale
video pretraining? In this work, we study whether pretrained
video foundation models encode intuitive-physics information in their frozen representations, and how its accessibility varies across pretraining paradigms, layers, backbone variants, and probe types. Using frozen-feature probing on IntPhys2 and Minimal
Video Pairs (MVP), we compare predictive joint-
embedding models (V-JEPA), masked reconstruction models (VideoMAE), and
diffusion-based
video generators (LTX-
Video). We find that across linear, MLP, and temporal attentive probes, physics-relevant information is generally weakest in early layers and most accessible at intermediate to late depth. On MVP, temporal attentive probes substantially outperform linear readouts, with V-JEPA achieving 95.0% pair consistency. On IntPhys2, linear probes recover more signal, and VideoMAE attains the strongest temporal attentive result at 73.9% violation-of-expectation accuracy. On both
benchmarks, attentive probe performance on LTX consistently underperforms compared to the other two pretraining paradigms. Together, these results suggest that intuitive-physics is broadly decodable from
video foundation models, but it is not represented as a single, uniform capability. Its accessibility, and especially its temporal grounding, depends much more on the
benchmark and readout than on a simple hierarchy of pretraining objectives or model scale.
Read the full abstract ▾
All authors Samuele Punzo, Niccolò Caselli, Ippokratis Pantelidis, Francesco Massafra, Salvatore Lo Sardo, Mohammadreza Salehi
Computer Vision · Cross-listed from cs.CL
Where Does the Answer Come From? Benchmarking View-Level Visual Evidence Identification in Multi-View MLLMs for Autonomous Driving
Yimu Wang, Yee Man Choi, Barry Zhang et al. (+3 more)
Multimodal large
language models (MLLMs) achieve strong results on visual
reasoning benchmarks, but answer accuracy alone does not indicate whether a model relied on the correct visual evidence. This gap is particularly important in multi-view driving scenes used for autonomous driving, where a model can produce a plausible answer while grounding it in the wrong camera view. We introduce a multi-view visual question answering
benchmark for evaluating evidence-source identification: given six synchronized NuScenes views and a question, the model must identify the supporting camera view and answer the question. The
benchmark contains 122 conflict-centric question-answer pairs from 73 scenes, spanning causality, counterfactual
reasoning, and intent prediction. View labels are proposed by an automatic conflict-mining pipeline and manually verified by annotators. We evaluate three settings: camera-view selection, oracle QA given the golden view, and joint prediction in which the model selects a view and answers in one pass. Answers are evaluated in both multiple-choice and free-form formats, using exact match for structured predictions and an LLM judge for free-form responses. By explicitly separating visual-source identification from answer correctness, the
benchmark exposes grounding failures that answer-only evaluation misses.
Read the full abstract ▾
All authors Yimu Wang, Yee Man Choi, Barry Zhang, Mozhgan Nasr Azadani, Sean Sedwards, Krzysztof Czarnecki
Computer Vision
Jie Zhang, Qilang Ye, Hao Zhou et al. (+2 more)
The dominant paradigm in
video retrieval relies on
embedding-based full-corpus scanning, which suffers from inherent computational inefficiency and the semantic asymmetry between information-dense
videos and sparse textual queries. To bridge this gap, we introduce MAVIS, a novel multi-
agent framework that rethinks
retrieval as cooperative
reasoning rather than brute-force search. MAVIS first bridges the granularity mismatch by parsing raw
videos into a Structured Semantic Library, enabling explicit attribute-level indexing. During
retrieval, a planner decomposes complex user intents into atomic sub-tasks, dispatching specialized
agents to independently nominate candidates. Crucially, MAVIS employs a Logic-aware Debate mechanism with a strict veto protocol, where
agents collaboratively prune logical mismatches to identify a compact set of “controversial” candidates for fine-grained verification. This agentic workflow effectively bypasses the inefficiency of full-library traversal. Extensive experiments on MSR-VTT, MSVD, and ActivityNet demonstrate that MAVIS achieves competitive performance without task-specific fine-tuning, offering a scalable and interpretable alternative to traditional dual-encoder approaches.
Read the full abstract ▾
All authors Jie Zhang, Qilang Ye, Hao Zhou, Haochen Liang, Fei Luo
Computer Vision
CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation
Yuheng Chen, Teng Hu, Yuji Wang et al. (+7 more)
The fidelity and structural diversity of training
datasets fundamentally determine the capabilities of
video generation models. While commercial systems showremarkableabilitytogeneratecinematicnarratives, the progress of open-source models remains limited by the scarcity of high-quality training data. To bridge this gap, we introduce CineDance-1M, a large-scale, open research Text-to-Audio-
Video (T2AV)
dataset designed specifically for multi-shot, long-form joint audio-
video generation. Averaging 92.8 seconds and 24.2 continuous shots per
video, it provides configurable, structured annotations for both audio and
video modalities. This exceptional quality is achieved through a rigorous three-stage curation pipeline: i) diverse sourcing and comprehensive cleansing, ii) film-theory-inspired narrative parsing, and iii) hierarchical dual-modal captioning. For a comprehensive assessment, we propose CineBench, featuring a diverse
prompt suite and a six-dimensional, human-aligned metric system tailored for complex narrative audio-
video evaluation. Furthermore, we adapt LTX-2.3 into CineDance, which demonstrates exceptional single-modality quality alongside precise audio-
video alignment and robust subject and environment consistency, effectively validating our curation strategy and the high quality of CineDance-1M. We anticipate that this work will serve as a solid foundation for accelerating future research in multi-shot, long-form joint audio-
video generation. Our project page is available at https://aliothchen.github.io/projects/CineDance/.
Read the full abstract ▾
All authors Yuheng Chen, Teng Hu, Yuji Wang, Qingdong He, Zhucun Xue, Qianyu Zhou, Jason Li, Lizhuang Ma, Jiangning Zhang, Dacheng Tao
Computer Vision
ATN3D: Density-Aware LiDAR-Radar Early 3D Object Detection Under Extreme Sparsity
Debojyoti Biswas and Xianbiao Hu
3D object detection is the backbone of perception for automated vehicles (AV) and broader intelligent transportation systems applications. Long-range detection is challenging because sensing evidence is sparse; yet this “long-range” scenario is routine in traffic. Although >30m is often labeled long-range in computer
vision, on roadways it affords only approx. 1-2s for perception and decision-making. Under such extreme sparsity, two core challenges arise. First, early
multimodal fusion tends to discard sparsity information and inject noise from empty or falsely occupied
cells, degrading long-range recall. Second, context-agnostic uniform channel supervision favors dense and near-range samples, leaving far and small objects under-optimized, delaying the earliest detection of distant objects. We propose “Ask The Neighbor” (ATN3D), a
LiDAR-Radar framework tailored for sparse-range conditions. ATN3D introduces (i) Density-aware early fusion with cross-modal gating that conditions fusion on per-voxel density/sparsity and Radar evidence, (ii) Occupancy-gated neighborhood aggregation with circular kernels to aggregate only from credible
cells, (iii) Evidence-conditioned channel self-
attention to adapt channel weights with weather/range, and (iv) a Range-aware loss that re-balances classification and localization by distance, aligning training with distance-stratified evaluation. On the VoD
benchmark across clear and foggy conditions, ATN3D surpasses strong baselines: +3.55% mAP in clear weather and +8.41% mAP under simulated heavy fog; for >30m objects, gains are +3.33% (clear) and +2.09% (heavy fog). These results indicate earlier and more reliable long-range detections under sparse sensing in on-road traffic.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
DexPIE: Stable Dexterous Policy Improvement from Real-World Experience
Ruizhe Liao, Wenrui Chen, Liangji Zeng et al. (+4 more)
Dexterous manipulation presents substantial challenges for imitation learning due to its high-dimensional action space and complex contact-rich dynamics. Policies trained purely from demonstrations often suffer from compounding errors during deployment and require large amounts of expert data to achieve reliable performance. To move beyond the limitations of demonstration data, in this work, we propose DexPIE, a post-training framework for dexterous policy improvement from experience collected through real-world deployment. First, DexPIE enables effective exploration coverage through a dexterous-hand-adapted intervention system and multi-stage DAgger-style data collection across initial and intermediate task stages. Meanwhile, we enhance consistency between training and inference to reduce the distribution shift between rollouts and demonstration data, better aligning rollout behavior with demonstrations, allowing the critic to learn a value function induced by a more consistent underlying policy. Together, these components provide reliable supervision for policy evaluation. Finally, DexPIE improves the policy through conditioning on a continuous optimality indicator, allowing the policy to leverage the quality of data in a more fine-grained manner. Across three challenging real-world dexterous manipulation tasks, DexPIE achieves a 37.3% improvement in success rate over the demonstration-based reference policy, outperforming all baseline methods and demonstrating stronger robustness. The source code and
dataset will be made publicly available.
Read the full abstract ▾
All authors Ruizhe Liao, Wenrui Chen, Liangji Zeng, Haoran Lin, Fan Yang, Kailun Yang, Yaonan Wang
Computer Vision
TUDSR: Twice Upsampling-Diffusion for Higher Super-Resolution
Zhiqiang Wu, Yitong Dong and Xian Wei
Diffusion-based generative models have achieved remarkable success in real-world image super-resolution (SR). With tiled
diffusion techniques, these models can produce high-resolution images that exceed their native-supported resolution. However, the quality of such high-resolution (e.g $2048^2$) outputs often remains extremely poor, primarily due to two factors we consider: the image upsampling ratio (e.g $\times8$) exceeding the model's native-supported upsampling ratio (e.g $\times4$), and the model's native-supported resolution. In practice, training a native high-resolution model requires larger architectures, which incur significant computational overhead and GPU
memory costs, making it hard on limited-resource equipment. Thus, we present TUDSR, a Twice Upsampling-
Diffusion framework for higher SR. The TUDSR framework mainly consists of two stages: the first involves training at $R$-resolution, and the second introduces a looped chunk-based training strategy at $NR$-resolution. Each stage adapts a one-step GAN architecture comprising a generator and a discriminator. Based on SD2.1-base, we develop TUDSR-S, which achieves state-of-the-art performance across multiple
benchmarks. Extensive experiments further demonstrate that TUDSR-S generates high-quality images at the resolutions of $1024^2$ and even $2048^2$, significantly outperforming existing approaches. Code is available at https://github.com/wuer5/TUDSR.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Efficient Minimal Solvers for Relative Pose Estimation in Autonomous Driving Applications
Tao Li, Liang Liu, Jianli Han et al. (+1 more)
With the advancement of visual sensing systems, computer
vision is playing an increasingly important role in autonomous driving and
robot navigation. Relative pose estimation in multi-camera systems is essential for accurate vehicle localization and environment perception, demanding high real-time performance and robustness. Existing methods, however, often involve high computational costs and rely heavily on abundant feature matches, limiting their applicability in time-sensitive driving scenarios. To address these limitations, this paper introduces a unified framework for efficient relative pose estimation, built upon a novel translation parameterization and first-order rotation approximation. Within this framework, we propose three efficient minimal solvers specifically designed for autonomous vehicles. The first solver integrates the vertical direction prior from Inertial Measurement Units (IMUs), the second utilizes the rotation axis direction prior during steering maneuvers, and the third is designed for planar motion - a realistic assumption for ground vehicles operating on structured roads. By reducing both the minimal number of point correspondences and the algebraic complexity, our methods enable faster hypothesis generation within RANSAC-based pipelines, improving suitability for real-time systems. Extensive experiments on synthetic
datasets and the KITTI autonomous driving
benchmark demonstrate that the proposed solvers achieve a favorable balance between speed and accuracy compared to existing state-of-the-art
algorithms.
Read the full abstract ▾
All authors Tao Li, Liang Liu, Jianli Han, Weimin Lv
Computer Vision
Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?
Apratim Bhattacharyya, Shweta Mahajan, Sanjay Haresh et al. (+5 more)
Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online
videos. This opens the door to the use of
video (and
multimodal) large
language models (LLMs) as task guidance assistants. A crucial capability for the real-world success of a prospective task guidance assistant is it's ability to intervene proactively as soon as a mistake is apparent in order to guide the user. To evaluate this crucial capability, we introduce Ego-MC-Bench (Mistake Corrections), a
benchmark for evaluating reactive, step-by-step task guidance in realistic cooking scenarios. Extensive experiments show that Ego-MC-Bench is highly challenging for state-of-the-art
video LLMs. We argue that a key reason is the limited availability of training data for fine-tuning models on this task. Although there exists a wide range of cooking
video datasets, existing
datasets lack examples of mistakes along with appropriately timed interventions. To help address this data limitation, we also introduce Ego-CoMist, a counterfactual synthetic
dataset created by transforming non -interactive cooking
videos into supervised training examples showing proactive interventions. We show that fine-tuning on Ego-CoMist yields performance gains especially for smaller and more efficient
video LLMs that are well suited for delivering assistance on edge devices.
Read the full abstract ▾
All authors Apratim Bhattacharyya, Shweta Mahajan, Sanjay Haresh, Rajeev Yasarla, Reza Pourreza, Litian Liu, Risheek Garrepalli, Roland Memisevic
Computer Vision
A VideoMAE-v2 Approach to Zero-Shot Traffic Accident Anticipation
Siyuan Li, Xiaoyang Bi and Mengshi Qi
Traffic accident anticipation – predicting the likelihood of an imminent collision at every frame of a dashcam
video – is safety-critical yet difficult to scale, because collecting in-domain annotated accident footage for every deployment scenario is prohibitively expensive. We study this task under a zero-shot setting where no target-domain training data is available: the model must learn exclusively from a publicly available binary-labelled driving-accident
dataset and generalise to unseen dashcam footage. We propose a framework that bridges the gap between the frame-level temporal risk estimation task and coarsely labelled binary accident
datasets by coupling a VideoMAE-v2 backbone with a per-frame prediction head under a sliding-window protocol. Our method achieves 2nd place in the 2026 CVPR@AUTOPILOT Zero-Shot Accident Anticipation competition. Code is available at https://github.com/TimeSouth/zero-shot-taa-solution.
Read the full abstract ▾
Computer Vision
Adversarial Attack and Disturbance Detection by Hadamard-Coded Output Representations for Object Detection and Semantic Segmentation
Lucas Görnhardt, Timo Bartels, Niklas Schwarz et al. (+1 more)
Conventional one-hot encodings often yield poorly calibrated models, being overconfident under attack, and letting
entropy-based detection
algorithms fail. Previous image classification works have demonstrated that Hadamard-coded output representations can improve
adversarial robustness. However, attempts to integrate Hadamard codes into semantic
segmentation fall far behind state-of-the-art models in mean intersection-over-union performance. Regarding object detection, such output encodings have not yet been investigated at all. Further, no prior art addressed intrinsic codeword inconsistencies or actually exploited intrinsic codeword redundancy. Accordingly, we first derive a novel decoding procedure for Hadamard codewords towards optimal class-wise probabilities, solving the underlying optimization problem by using the projection onto the probability simplex. Second, our optimization delivers a measure of prediction inconsistency. Third, we are the first to show how to exploit these inconsistencies for
adversarial attack and disturbance detection. Fourth, we introduce HadamardNet, a framework employing Hadamard codes as output representations for semantic
segmentation and object detection models and tasks. We conduct a comprehensive evaluation both on disturbances and
adversarial attacks, achieving state-of-the-art perturbation detection performance for both tasks in only a single detection pass, while delivering equivalent or close-by reference performance on clean data. Code is available at https://github.com/ifnspaml/HadamardPerturbationDetection.
Read the full abstract ▾
All authors Lucas Görnhardt, Timo Bartels, Niklas Schwarz, Tim Fingscheidt
Computer Vision
SwiftVR: Real-Time One-Step Generative Video Restoration
Jiaqi Yan, Xiangyu Chen, Xinlin Zhong et al. (+5 more)
Real-time
video restoration (VR) for live streams requires high-resolution outputs under strict per-frame latency constraints. Existing one-step
diffusion-based VR models remain difficult to deploy on consumer-grade GPUs due to two main bottlenecks: quadratic spatial
attention at high resolutions and the latency-
memory overhead of large
video autoencoders. We present SwiftVR, a streaming one-step generative VR framework that reduces both bottlenecks under a causal chunk-wise protocol. For
attention, mask-free shifted-window self-
attention gathers each spatial window into a dense tensor via deterministic indexing, keeping all
attention calls on the dense scaled dot-product
attention path without masks, cyclic shifts, padding, or hardware-specific sparse kernels. Because SwiftVR uses only standard dense SDPA calls, the trained model transfers to consumer GPUs without retraining or custom kernels. For autoencoding, a lightweight Restoration-aware Autoencoder enables fast chunk-wise decoding while preserving reconstruction quality. On a single H100, SwiftVR sustains 31~FPS at 2560x1440 and 14~FPS at 3840x2160, whereas all compared
diffusion-based VR baselines exceed the
memory limit at 4K. On a consumer RTX~5090, SwiftVR reaches 26~FPS at 1920x1080. To our knowledge, SwiftVR is the first generative VR model to achieve real-time 1080p streaming on a consumer-grade GPU, while attaining strong no-reference perceptual quality with lower inference cost. Project is available at https://h-oliday.github.io/SwiftVR.
Read the full abstract ▾
All authors Jiaqi Yan, Xiangyu Chen, Xinlin Zhong, Haibin Huang, Chi Zhang, Jie Liu, Jiantao Zhou, Xuelong Li
Computer Vision
Securing Self-supervised Data Curation for Foundation Models Robustness
Sandeep Gupta and Roberto Passerone
Self-supervised data curation provides a pathway to scaling and improving the generalization capabilities of machine learning models. By leveraging self-supervised learning (SSL) for data curation, the demand for massive training
datasets required by foundation models can be effectively met. SSL greatly alleviates the costs associated with annotation and manual
dataset curation while minimizing the need for human oversight. However, the integrity of SSL-curated
datasets must be rigorously checked, as reliance on anonymous and unvetted external sources can substantially increase the risk of data poisoning. In this paper, we propose a Poisoned Data Detector (PDD), an active defense mechanism designed to ensure the integrity of SSL-curated
datasets prior to foundation model training. PDDs are designed using a combination of the pretrained ImageBind model and traditional classifiers, including Random Forest (RF), k-Nearest Neighbors (KNN), Naive Bayes (NB), and Support Vector Machines (SVM). We rigorously evaluated PDDs using 176,200 images from three diverse
datasets and three different
adversarial attacks encompassing both in-distribution and out-of-distribution scenarios. Notably, SVM-PDD achieves superior performance for both in-distribution (Set3-Set5) and out-of-distribution (TrueFace and 140K RealFace)
datasets. Our design demonstrates strong scalability and enables the rapid integration of new
adversarial attack detectors through an ensemble approach.
Read the full abstract ▾
Computer Vision
Prisma-World: Camera-Controllable Multi-Agent Video World Model
Huiqiang Sun, Zhan Peng, Size Wu et al. (+9 more)
Video world models have made rapid progress in generating controllable visual experiences, but most of them still simulate the world from a single observer. Extending such models to multiple
agents raises a central challenge: if each
agent's future state is generated independently, overlapping views may instantiate different versions of the same scene, leading to inconsistent objects, layouts, and appearances across
agents. Conventional camera conditioning controls individual trajectories, but it does not explicitly couple the generation of views that should agree under shared scene geometry. We introduce Prisma-World, a camera-controllable multi-
agent world model that formulates multi-
agent generation as a joint geometry-aware denoising process for cross-view consistency. Prisma-World processes all
agent videos within one full-
attention sequence, uses a multi-
agent RoPE design to distinguish
agent identities while preserving synchronized temporal coordinates, and injects relative camera geometry into
attention to bias overlapping viewpoints toward shared scene evidence. To further strengthen multi-view consistency and enhance global spatial perception, we augment our framework with an overlap-decaying curriculum training paradigm alongside minimap-conditioned structural guidance. To facilitate the training and evaluation of multi-
agent models, we introduce PrismaDataset, a large-scale UE5
dataset with panoramic acquisition across diverse scenes, composable multi-
agent view groups with flexible
agent counts and complex camera trajectories, and precise camera/action annotations for consistency training and evaluation. Experiments show that a single Prisma-World model can generate high-fidelity multi-
agent videos with flexible
agent numbers, camera controllability, improved cross-view consistency, and spatial grounding under minimap guidance.
Read the full abstract ▾
All authors Huiqiang Sun, Zhan Peng, Size Wu, Kun Wang, Kang Liao, Dianyi Wang, Xingyu Zeng, Sheng Jin, Yangguang Li, Zhiguo Cao, Ziwei Liu, Wei Li
Computer Vision
ContextShift: A Controlled Benchmark for Context Dependence in Object Detection
Dan Zlotnikov, Alex Lazarovich and Ohad Ben-Shahar
Modern object detectors achieve strong performance on standard
benchmarks, yet their robustness to contextual variation remains insufficiently understood. Prior evaluations largely rely on aggregate metrics such as AP on uncontrolled distribution shifts, which can obscure how performance degrades under context change. We introduce ContextShift, a controlled
benchmark that systematically manipulates object–context relationships while preserving object appearance. Built on COCO 2017, it isolates context as an independent variable through
geometric transformations and synthetic and natural background substitutions, including a continuous compatibility axis based on normalized pointwise mutual information (NPMI). Across diverse detector architectures, we observe a consistent degradation pattern: false negatives increase by up to 227% and prediction volume decreases by up to 44%, while false positives remain stable or decline. This suppression behavior is not captured by aggregate metrics such as AP, which can mask substantial recall loss and changes in prediction dynamics. Further analysis suggests that degradation is driven less by reduced confidence than by a reduced formation of valid detection candidates. Moreover, performance along the statistical compatibility axis is non-monotonic, peaking at intermediate NPMI and degrading toward both extremes, indicating that statistical co-occurrence does not correlate linearly with effective visual context. Finally, we show that context-aware augmentation improves robustness: every augmented variant outperforms the
dataset-only baseline on both original and manipulated test images, partially recovering performance lost to prediction-suppression failures by exposing models to object–context decoupling during training.
Read the full abstract ▾
Computer Vision
Optical Music Recognition for Real-World Manuscripts with Synthetic Data
Jiří Mayer, Martina Dvořáková, Vojtěch Dvořák et al. (+6 more)
Optical Music Recognition (OMR) has seen major progress in model design, with end-to-end methods now capable of recognising notation at all levels of complexity. However, the impact of this progress has been limited by the visual domains of available training
datasets, which are largely born-digital. Existing large collections of sheet music in libraries and other heritage institutions contain predominantly manuscripts, whose visual domains are highly diverse and different, so existing OMR systems fail when applied in the real world. These institutions are often resource-constrained, so large in-domain
datasets cannot be expected. We provide a first baseline on real-world manuscripts with complex piano notation in the resource-constrained scenario. Using fine-grained music notation graph (MuNG) annotations and the Smashcima synthesis tool, we then show that while some direct transcriptions of in-domain data remain essential, domain adaptation using synthetic musical manuscript images brings significant improvement. Furthermore, the symbols used do not need to be in-domain, so the expensive fine-grained annotation can be avoided. We thus bring OMR closer to one of its stated goals: preserving and promoting musical cultural heritage.
Read the full abstract ▾
All authors Jiří Mayer, Martina Dvořáková, Vojtěch Dvořák, Markéta Herzánová Vlková, Filip Bím, Pavel Pecina, Samuel Šomorjai, Petr Žabička, Jan Hajič jr
Computer Vision
Efficient Minimal Solvers for Visual-Inertial Relative Pose Estimation in Multi-Camera Systems
Tao Li, Zhenbao Yu, Banglei Guan et al. (+2 more)
Estimating the relative poses of multi-camera systems is a fundamental problem in computer
vision, with critical applications in autonomous vehicles, mobile devices, and unmanned aerial vehicles (
UAVs). However, existing solutions often suffer from high computational complexity or rely on an excessive number of point correspondences, limiting their real-world applicability. To address these limitations, we propose two efficient minimal solvers for estimating the relative poses of multi-camera systems using a novel parameterization. The first solver leverages the vertical direction prior provided by Inertial Measurement Units (IMUs), while the second utilizes the rotation axis direction prior from IMUs. Our methods require only four point correspondences and reduce the problem of multi-camera relative pose estimation to solving a univariate 6th-degree
polynomial, a significant improvement over existing approaches, which typically involve 8th-degree
polynomials. This reduction in computational complexity and correspondence requirements makes our solvers particularly effective when integrated into RANSAC frameworks, demonstrating strong potential for visual odometry applications. Through rigorous evaluations on synthetic data and the KITTI
benchmark, our methods achieved superior computational efficiency and competitive accuracy compared to state-of-the-art
algorithms.
Read the full abstract ▾
All authors Tao Li, Zhenbao Yu, Banglei Guan, Jianli Han, Weimin Lv
Computer Vision
Training-Free Generalized Few-Shot Segmentation through Open-Vocabulary Semantic Arbitration
Silas Kwabla Gah and Ebenezer Owusu
Generalized Few-Shot Semantic
Segmentation (GFSS) has traditionally been approached as a representation-learning problem, requiring task-specific adaptation to incorporate novel classes from limited support examples. Recent foundation models, however, already exhibit strong open-vocabulary recognition and
segmentation capabilities, raising a different question: can GFSS be solved through inference-time coordination of frozen semantic priors rather than parameter adaptation? We answer this question with Open-V, a training-free GFSS framework that combines Segment Anything (SAM3) Promptable Concept
Segmentation (PCS) with a K-shot CLIP support centroid through calibrated per-pixel semantic arbitration. OpenV introduces no trainable components and supports arbitrary semantic categories at inference time. Beyond
segmentation performance, our study contributes three broader findings. First, we show that support information can be incorporated through inference-time semantic grounding, and that its contribution increases as foundation-model text priors weaken on label-disjoint vocabularies. Second, we identify a reproducibility confound in foundationmodel
segmentation, demonstrating that preprocessing and evaluation-space mismatches can silently distort reported performance. Finally, we validate Open-V across PASCAL5i, COCO-20i, and ADE-OW, showing that training-free coordination of foundation-model priors generalizes across both conventional GFSS and open-vocabulary evaluation settings. On PASCAL-5i (1-shot), Open-V attains base/novel/harmonic mIoU of 78.4/77.5/77.9, without GFSS-specific training surpassing the strongest trained baseline by +17.7 HM.
Read the full abstract ▾