Computer Vision
Unsupervised spiking feature learning for event-based pedestrian crossing detection: approaching supervised accuracy without labelled training data
Henok Teklu, Mustafa Sakhai, Matej Mertik et al. (+1 more)
Event cameras are well suited to pedestrian crossing detection, and spiking neural networks (SNNs) can process their output natively, but current SNN detectors are trained with supervised backpropagation and therefore require costly frame-level crossing labels. We investigate crossing detection with no labels in feature learning and report the first unsupervised results on the recent DVS-PedX pedestrian-crossing
benchmark. A single spiking layer trained with winner-take-all spike-timing-dependent plasticity learns a dictionary from unlabelled event-frame patches; frames are encoded by cosine similarity to the learned filters with spatial pooling and read out by a linear classifier, the only supervised component. On the 24,454-frame test split the method attains 90.3% accuracy and an area under the receiver operating characteristic curve (AUROC) of 0.936, compared with 92.0% and 0.943 for a supervised spiking network trained end-to-end on the same frames; under adverse weather the AUROC is 0.913. The result is insensitive to the choice of plasticity rule but depends strongly on the readout protocol: with the classical
neuron-assignment readout the same network attains only 0.70 AUROC. On the
benchmark's real converted portion, a readout refit lifts performance from chance (0.53) to 0.67 AUROC, within the published supervised range. These findings indicate that the accuracy cost of removing labels from feature learning is small on this
benchmark, and that reported weaknesses of unsupervised spiking networks may be attributable to the readout protocol rather than to the learning rule.
Read the full abstract ▾
All authors Henok Teklu, Mustafa Sakhai, Matej Mertik, Maciej Wielgosz
Computer Vision
One-Step Is Optimal: Unconditional Rectified Flows are Noise2Noise Denoisers, and Multi-Step Integration Provably Hurts—A Benchmark and Task-Based Detectability Study on Low-Dose CT
Timothy Sereda and Debesh Jha
Iterative and generative denoisers are increasingly used under the assumption that multi-step refinement outperforms a single regression pass. We show the opposite for label-free denoising. An unconditional rectified flow trained on two noisy observations of the same signal, as in Noise2Noise, has a minimiser whose one-step readout is exactly the MMSE denoiser without requiring clean targets. In contrast, multi-step integration provably departs from the MMSE solution because the flow terminates at the noisy data distribution rather than the clean-signal distribution. This departure is exact in a tractable Gaussian model and is confirmed experimentally: one-step flow matches a direct regressor, whereas multi-step Euler integration progressively reduces fidelity. Counterintuitively, the degradation increases with training quality, as a better velocity field more faithfully transports samples toward the noisy terminal law. The key ingredient is therefore the decorrelated pairing, not the flow machinery: a one-step regressor trained on matched noisy pairs gives our best label-free result ($+1.99$,dB). We evaluate these findings on CTDenoiser, a controlled low-dose CT
benchmark spanning five architectures and supervised, similarity-based, blind-spot, and per-image methods. Among label-free approaches, only correlated-noise-aware Noise2Sim improves over the noisy baseline, while Noise2Void is flat-to-negative because CT noise violates its pixel-independence assumption. Finally, although supervised denoisers gain approximately $4$,dB PSNR, a channelized Hotelling observer shows reduced low-contrast lesion detectability, revealing clinically relevant degradation missed by PSNR and SSIM.
Read the full abstract ▾
Computer Vision
Query-aligned video frame selection for long video understanding
Md. Safayet Islam, Dilip Sarkar and Liang Liang
Multimodal large
language models (MLLMs) process
multimodal inputs by converting text, images, and
videos into
token sequences that are subsequently processed by a backbone
language model. While MLLMs have achieved excellent performance in understanding the content of individual images,
video understanding remains significantly more difficult because
videos contain large number of
video frames. MLLMs typically process only a subset of these frames, usually ranging from 8 to 64. MLLMs usually sample frames uniformly, regardless of their relevance to the question being answered. To address this limitation, several training-free, model-agnostic methods for selecting question-relevant frames have recently been proposed. In this work, we introduce a frame-selection method designed specifically for multiple-choice questions. We extend the query text by appending semantic cues derived from the answer choices and employ a direct query-frame
alignment scoring mechanism. To the best of our knowledge, our method is the first to directly utilize answer choices as inference-time cues for selecting frames relevant to answering a question. The method first constructs a compact candidate pool by subsampling
video frames at a fixed rate. The frames are then scored according to their maximum cosine similarity across all question-answer pairs to identify the most relevant frames for a given query. This approach preserves a fixed
token budget while improving the relevance of the visual evidence provided to the downstream MLLM. We evaluate the effectiveness of our frame-selection method on the MLVU,
Video-MME, and LongVideoBench
benchmarks using three MLLMs: LLaVA-Mini, Qwen2-VL, and LLaVA-
Video. Experimental results demonstrate that answer-aware frame selection generally outperforms uniform sampling and existing training-free frame-selection methods under the same frame budget.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Age-Adaptive Handwriting Reconstruction from an IMU-Based Digital Pen through Shared Representations and Domain-Specific Heads
Florent Imbert, Yann Soullard, Eric Anquetil et al. (+1 more)
Digital pens are widely used to capture handwriting on digital devices, enabling precise trace recording and enhancing human-computer interaction. However, most are bundled with tablets and lack cross-brand compatibility. Recent digital pens equipped with kinematic sensors have emerged, designed for use on any surface. This especially opens significant potential for supporting handwriting acquisition in classrooms. Handwriting reconstruction from such an IMU-equipped pen poses a challenge due to the significant variability in sensor signals between adults and children. Even when producing visually similar traces, variations in writing dynamics, motor control, pen holding, and user confidence introduce substantial discrepancies in the captured signals. Additionally, the high variability in children's handwriting requires collecting large amounts of data, which is not feasible to implement in a school environment at scale. Furthermore, models trained exclusively on adult data fail to generalize to children's handwriting, and conversely, models trained on children's data perform poorly on adult writers. This crosspopulation degradation highlights the need for a unified model that can be deployed directly on the pen, without any user-specific adaptation. To address this issue, we propose a cross-domain learning, using an original neural network architecture based on a Temporal Convolutional Network and multiple prediction heads. The model is designed to be robust across age groups by leveraging shared features while effectively handling variability induced by differences in graphomotor development. This approach aims to improve handwriting trace reconstruction from sensor data, where each domain benefits from additional data provided by the other domain.
Read the full abstract ▾
All authors Florent Imbert, Yann Soullard, Eric Anquetil, Hui Han
Computer Vision
Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges
Mahsa Mohammadi, Zeyu Fu and Sareh Rowlands
Action anticipation predicts future human actions from partial
video under incomplete context and temporal uncertainty. Recent systems introduce large
language models (LLMs),
vision-
language models (VLMs), or
language-derived semantics at different stages, but reported gains are difficult to interpret when task formulation, visual pretraining, supervision, decoder design, and evaluation code change simultaneously. The central contribution of this review is an evidence-aware design map that crosses task regime with the point at which
language-derived information intervenes. We characterise task regimes along six axes. These axes organise the literature into five broad task families: single-action, sequence, object-interaction, cross-view, and planning-oriented settings. C1-C3 locate interventions in context construction, goal/intention modelling, and future decoding, while C4 is treated as an adjacent, emerging grounding/executability extension. Unlike a generic processing pipeline, the map links each intervention to an appropriate counterfactual, failure diagnosis, and permissible evidence claim. Supporting contributions include a protocol-level audit of Ego4D-LTA and EPIC-KITCHENS-100, a multidimensional evidence profile, and the Backbone-Aware Comparison and Ablation Protocol (BCAP). The unresolved EK-100 record is treated as a reporting-comparability case study and is not used as a leaderboard. Evidence for LLM benefits, goal ambiguity, and horizon effects is therefore formulated as testable hypotheses requiring matched validation, not as causal conclusions. The accompanying package contains the coded evidence, source locators, protocol metadata, and versioned catalogue used in the review.
Read the full abstract ▾
Computer Vision
Learning Steadily: Accumulating Relative Point Margin Scores for Face Image Quality Assessment
Guray Ozgur, Tahar Chettaoui, Eduarda Caldeira et al. (+4 more)
Face Image Quality Assessment determines the suitability of captured face images for automated face recognition (FR), a critical capability for reliable biometric systems. Existing state-of-the-art FR-integrated FIQA methods suffer from temporal instability: as the feature space evolves during training, single-epoch quality estimates fluctuate, creating a moving target that undermines reliable quality prediction. We introduce CARPM-FIQA, a stabilization strategy for FR-integrated FIQA that accumulates relative point margin measurements, the ratio between intra-class compactness and inter-class separation, across the entire training trajectory rather than relying on single-epoch estimates. This cumulative averaging approach provides theoretically grounded advantages: reduced variance in quality estimates, improved mean squared error, and enhanced ranking stability with convergence guarantees as training progresses. Through controlled experiments on the SynFIQA
dataset with labeled quality groups, we demonstrate that cumulative averaging achieves superior discriminative ability, and ablation studies across different training configurations confirm consistent improvements. Evaluated against twelve FIQA methods on eight challenging
benchmarks with four FR models at two FMR thresholds, CARPM-FIQA places 4th (CARPM-FIQA(L)) and 6th (CARPM-FIQA(S)) of 17 compared methods by pAUC-EDC and AUC-EDC averaged across FR models and, after per-
benchmark normalization, across
benchmarks, staying within a few percent of the best method's normalized average for every FR model, providing a principled solution to training instability while maintaining the performance benefits of FR integration. More broadly, our work demonstrates that temporal aggregation strategies can stabilize training objectives in deep learning systems where target values inherently fluctuate due to evolving feature representations.
Read the full abstract ▾
All authors Guray Ozgur, Tahar Chettaoui, Eduarda Caldeira, Marco Huber, Jan Niklas Kolf, Naser Damer, Fadi Boutros
Computer Vision
ForensicZoom: Adaptive Visual Inspection with Multimodal LLMs for Industrial-Grade Face Forgery Detection
Hang Zhou, Yiming Tang, Kun Yu et al. (+3 more)
Reliable face forgery detection is critical to the security of online identity verification systems, where missed attacks compromise security and excessive false positives disrupt legitimate users. Specialized forensic detectors achieve strong detection performance but provide limited interpretability, while
multimodal large
language models (MLLMs) offer strong semantic understanding and interpretable
reasoning yet remain substantially weaker for face forgery detection. We argue that a key limitation lies in how visual evidence is acquired: subtle forensic artifacts may be poorly represented at standard resolution, while uniformly processing all cases at higher resolution is computationally inefficient. We therefore introduce ForensicZoom, an industrial-grade MLLM framework for adaptive visual inspection. ForensicZoom first equips a general-purpose MLLM with forensic-aware visual representations and aligns the
language model with these features. Its central mechanism, NEED_ZOOM, enables the model to autonomously request magnified views of suspicious regions when the initial evidence is insufficient, turning fixed-pass classification into adaptive multi-round forensic
reasoning. The zoom behavior is learned through reward shaping that balances detection accuracy with unnecessary visual inspection, concentrating additional computation on difficult cases. A final attribution optimization stage improves natural-
language forensic reports while preserving detection performance. On large-scale industrial identity verification data, ForensicZoom achieves over 97% TPR at 0.1% FPR, substantially outperforming both specialized detectors and existing MLLM-based methods while producing actionable forensic attributions. These results demonstrate that ForensicZoom can provide an effective path toward accurate, interpretable, and scalable MLLM-based face forgery detection.
Read the full abstract ▾
All authors Hang Zhou, Yiming Tang, Kun Yu, Qian Zhu, Minghao Li, Weigao Wen
Computer Vision
Cross-Dataset Transfer and Unknown-Class Detection in Imbalanced SAR Ship Classification
Ch Muhammad Awais, Marco Reggiannini, Davide Moroni et al. (+1 more)
Ship classification from Synthetic Aperture Radar (SAR) imagery is a critical computer
vision task, yet the robustness of models under deployment shifts remains unclear. While models are often trained on one
dataset and deployed on another, we lack a comprehensive understanding of their cross-
dataset generalization. To address this, we evaluate six pretrained models on two SAR ship
datasets in three settings: in-domain classification, cross-
dataset transfer, and unknown-class detection. For unknown detection, we hold out all classes one at a time. In-domain, SARDet100K gives the best balanced accuracy on both
datasets (73.4% on FUSARShip and 53.2% on OpenSARShip). In cross-
dataset transfer, we observe strong failures: some models show moderate accuracy but near-chance balanced accuracy (for example, 64.5% accuracy but 33.3% balanced accuracy for OpenSARShip to FUSARShip). In unknown detection, performance depends on the held-out class and
dataset, while MC-dropout variance is often close to random. These findings show that cross-
dataset generalization in SAR remains limited and that task-specific uncertainty scores are often more informative than MC-dropout variance for held-out-class detection, although their relative ranking depends on the
dataset and held-out class.
Read the full abstract ▾
All authors Ch Muhammad Awais, Marco Reggiannini, Davide Moroni, Giulio Del Corso
Computer Vision
Enhancing Foundation Models for Imbalanced SAR Ship Classification via Targeted Oversampling
Ch Muhammad Awais, Marco Reggiannini and Davide Moroni
Remote-sensing foundation models offer strong representations for SAR imagery, but their behavior under severe long-tail class imbalance is still not well characterized. We
benchmark DOFA and SAR-JEPA on the imbalanced OpenSARShip
dataset and compare them with ImageNet-pretrained baselines under a fixed, training-efficient protocol that keeps the backbone frozen. To mitigate imbalance without fine-tuning, we apply four oversampling methods in
embedding space exclusively to minority classes and train a lightweight classifier head on the augmented
embeddings. Across both foundation models, oversampling improves Macro-F1 and test accuracy relative to their respective baselines, with the largest Macro-F1 gains observed for DOFA using ADASYN (34.39 to 38.56) and for SAR-JEPA using SVM-SMOTE (25.89 to 32.30). We also report class-wise behavior, showing that aggregate improvements can coexist with persistent failures on specific rare classes. Code for
embedding extraction and reproducible multi-seed evaluation is provided to support rapid experimentation on free-tier hardware.
Read the full abstract ▾
Computer Vision
One Evaluation, Any Operating Point: Hypernetwork-Amortized MeanFlow for 3D MRI Reconstruction
Ruibo Wang
Generative priors reconstruct accelerated 3D MRI well but pay heavily at deployment: tens of network evaluations per volume, and protocol-specific hyperparameter tuning. A third hidden cost is the scanner's fixed sampling pattern. We treat the whole operating point as an input. A 3D MeanFlow patch network (a one-step flow model) is fine-tuned end-to-end through a warm-started, five-iteration differentiable conjugate-gradient projection. A small hypernetwork maps the operating point (data-consistency weight, acceleration, and the Cartesian sampling pattern itself) to the network's per-channel modulation. Three findings follow. (i) Learning the acquisition is worth more than any other operating point: on clinical knee data, the learned mask gains up to +2.34 dB over the protocol's variable-density mask. This gain requires the solver: with a feed-forward reconstructor the same learned mask hurts at 4x (-1.7 dB), but with the data-consistency projection it adds +4.6 dB. (ii) One evaluation is highly effective: it beats a 20-step patch-
diffusion prior by up to +3.1 dB on
brain and +2.9 dB on knee. Three to five evaluations extend the front to +6 dB while using a quarter of the prior's network calls. (iii) Fully sampled targets are optional: trained self-supervised on a split of acquired samples, the reconstructor matches its supervised twin at 4x on real data. Finally, we report what failed and why: subject-adaptive acquisition from measured energy, combining self-supervision with learned acquisition, and amortising the data-consistency weight.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
TTTIR: Unlocking Instance-Specific State Evolution via Test-Time Training for Image Restoration
Kaihang Zheng, Jun Li, Hang Guo et al. (+5 more)
Image restoration is inherently challenging due to the diverse and highly input-dependent nature of real-world degradations. While recent architectures like
Transformers and state-space models have advanced the field, they predominantly rely on static, globally shared parameters, which struggle to fully accommodate instance-specific degradation patterns. Test-Time Training (TTT) offers a promising paradigm for generating data-dependent operators, yet its standard self-supervised inner loop lacks the explicit guidance required to transition degraded features toward clean structures. To address this, we propose TTTIR, a novel framework that reformulates image restoration as an instance-specific state
evolution process. Specifically, we design Progressive State Sequence Generation (PSSG) to construct complementary spatial-frequency target states (defining what to recover), and State Transition
Evolution (STE) to adapt lightweight transition operators via a restoration-oriented TTT inner loop (determining how the features should evolve). Extensive experiments demonstrate that TTTIR consistently outperforms state-of-the-art models across multiple image restoration
benchmarks, achieving dynamic instance-specific recovery with favorable computational scalability. The code is available at https://github.com/Elysiaaaaaaaa/TTTIR.git.
Read the full abstract ▾
All authors Kaihang Zheng, Jun Li, Hang Guo, Hongyu Chi, Zimo Liu, Tao Dai, Jinpeng Wang, Yaowei Wang
Computer Vision
Yarn tracking of large-scale 3D textile reinforcements using topological material features
Hafsa El Herichi, Arturo Mendoza, Yanneck Wielhorski et al. (+2 more)
Automated
segmentation of CT images has become increasingly important to enhance the reliability of simulations through the generation of high fidelity numerical models. This study addresses the challenging task of semi-automatically tracking textile reinforcements in fan blade dry preforms using X-ray CT images captured at coarse resolutions (i.e., above 140 $\mu$m). Our approach offers a scalable, slice-based analysis conducted on planes orthogonal to the main yarn directions, applied to a large-scale real industrial component. This enables accurate identification and tracking of yarn paths while requiring minimal training. The method models three key yarn properties statistically: their typical cross-section shape, their continuity and movement in the 3D space, and their spatial relative arrangement with respect to neighboring yarns. These statistical properties are integrated into a tracking framework via a variational formulation that optimizes all yarn center positions in successive cross-section planes. The method tracks more than 3,000 warp yarns across 1,500 slices and achieves a tracking success rate above 90%. Overall, this work demonstrates a promising approach toward large-scale, automated textile reinforcement annotation, paving the way for more efficient material characterization in complex composite structures.
Read the full abstract ▾
All authors Hafsa El Herichi, Arturo Mendoza, Yanneck Wielhorski, Hugues Talbot, Stéphane Roux
Computer Vision
GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents
Boyu Qiao, Zixin Tang, Xiaoshuai Hao et al. (+1 more)
Coding-
agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing research has extensively evaluated the ability of coding
agents to generate programs from textual specifications. However, under black-box conditions where neither source code nor documentation is available, it remains underexplored whether an
agent can induce the rules solely through visual observation and active interaction and reproduce the target system as a verifiable executable system. To this end, we present GameReplica, a closed-loop evaluation framework for end-to-end black-box game replication that covers the full perception, exploration, induction, reproduction, and verification pipeline. GameReplica comprises 125 tasks spanning 25 games across 5 core mechanism families, with each game instantiated at five difficulty levels. The tasks require an
agent to access the target game only through screenshots and an action interface, induce the key visual elements and gameplay rules from pixel feedback and interaction outcomes, and generate a self-contained, runnable game replica that can be automatically verified by an external program. Experiments show that current coding
agents still face substantial challenges in end-to-end black-box replication: the best-performing model (Claude Opus 4.8) achieves an overall score of 71.6%, while the remaining models score only 4.0%–42.9%. Further analysis reveals a consistent pattern across all models: visual-fidelity scores are substantially higher than implementation- and rule-consistency scores, indicating that
agents replicate visual appearance more readily than game mechanics. The difficulty levels further amplify the performance gap: from L1 to L5, the overall score of weaker
agents drops sharply, whereas that of the best-performing
agent declines only slightly.
Read the full abstract ▾
All authors Boyu Qiao, Zixin Tang, Xiaoshuai Hao, Wenbo Li
Computer Vision
TAPe+ML: A Compact Structured Representation for Multi-Task Computer Vision
Sergey Kurinov and Alexey Upatov
We present TAPe+ML v3, a compact computer
vision system based on TAPe (Theory of Active Perception), a structured representation that encodes relations among perceptual elements before recognition. Instead of operating directly on pixel tensors, the system uses a shared TAPe representation and a modular recognition architecture for image classification, object detection, and instance
segmentation. TAPe+ML v3 combines background and contour processing, local object localization, prototype-based classification, and a coordinator for specialized submodels. Across the reported experiments, it uses fewer than 100,000 parameters. On COCO object detection, it obtains 84.7 mAP50 and 65.3 mAP50-95. On COCO instance
segmentation, it obtains 80.7 mask mAP50 and 58.4 mask mAP50-95. In classification experiments, it reaches 92 percent validation accuracy on Imagenette under an identical-training comparison with a raw-pixel baseline, and 89.9 percent Top-1 accuracy on ImageNet-Real. We also evaluate compactness in
video scene detection and adaptation under distribution shift in an industrial pilot. The results suggest that shifting part of the modeling burden from network parameters to a structured input representation can support compact multi-task
vision systems with reduced data,
memory, and compute requirements.
Read the full abstract ▾
Computer Vision
Rapid Loss of the Sierra Nevada's Largest Trees Driven by Fire
Fabien H. Wagner, Dan J. Dixon, Christopher W. Woodall et al. (+15 more)
Large trees disproportionately contribute to biomass storage, habitat structure, and ecosystem functioning. However, their distribution and health dynamics remain poorly quantified at a regional scale. Here, a deep learning model (U-Net-ID) and canopy height models derived from sub-meter aerial imagery from 2020 were used to delineate all individual trees with crown area $\geq$ 100 m$^2$ across the Sierra Nevada Floristic Province. The model was trained using more than 3.3 million synthetic tree crowns and achieved a median Intersection over Union (IoU) of 0.602 when validated against an independent
dataset of 20,273 crowns. A total of 6,515,705 large trees were mapped, occurring across approximately 78.7% of the Sierra Nevada Floristic Province. The spatial distribution of large trees showed associations with elevation, temperature, and precipitation. Using Sentinel-2 time series from 2020 to 2025, tree health dynamics were characterized by extracting spectral trajectories for each crown and applying BFAST breakpoint detection
algorithm combined with a disturbance classification framework to identify mortality, disturbance, and recovery trajectories of individual trees. Wildfires, estimated from CAL FIRE fire perimeters, were identified as the dominant driver of large-tree mortality, killing 10% of all large trees in the Sierra Nevada, with mortality strongly concentrated during the extreme 2020-2021 fire seasons.
Read the full abstract ▾
All authors Fabien H. Wagner, Dan J. Dixon, Christopher W. Woodall, Mayumi C. M. Hirye, Felipe Saad, Griffin Carter, Ricardo Dalagnol, Lorena Alves, Cynthia Creze, Stephen C. Hagen, Zhihua Liu, Christopher Mihiar, Adugna Mullissa, Le Bienfaiteur Sagang Takougoum, Bryan Shaddy, Yan Yang, Dafeng Zhang, Sassan Saatchi
Computer Vision
Audio for Sports Highlight Detection: A Comparative Empirical Study
Hao Xu, Meenakshi Sarkar, Vishnu Raj et al. (+1 more)
Sports highlight detection aims to identify the most exciting and meaningful moments from long sports
videos. While existing methods often emphasize visual or visual-
language representations, sports
videos contain rich audio cues, including commentator
speech, crowd reactions, whistles, ball impacts, and referee calls. In this work, we revisit the role of audio in sports highlight detection and ask a simple question: how far can audio alone go? We construct lightweight audio-only baselines using pretrained audio representations and compare them with visual-only and audio-visual methods on the SV-Highlights
benchmark. Surprisingly, our audio-only GRU baseline achieves strong performance and outperforms several existing audio-visual methods under our supervised evaluation setting. Furthermore, a simple audio-visual fusion baseline achieves the best performance across all metrics, indicating that audio and visual cues provide complementary information. To better understand the contribution of audio, we conduct source-separated analysis and show that vocal/commentary audio is more informative than background-only audio, while their combination performs best. We also analyze interpretable audio cues and find that highlight clips exhibit higher RMS loudness, peak loudness, and mid-frequency energy than non-highlight clips, although substantial distribution overlap indicates that loudness alone is insufficient. Our findings suggest that audio is an underexplored but highly informative modality for sports highlight detection and should be treated as a primary signal rather than merely an auxiliary cue.
Read the full abstract ▾
All authors Hao Xu, Meenakshi Sarkar, Vishnu Raj, David Gunawan
Computer Vision
Face-voice Association across LAnguages and Gender (FLAG) 2027 Challenge Evaluation Plan
Marta Moscati, Swapnil Khandoker, Muhammad Saad Saeed et al. (+8 more)
Face–voice association models may rely on
language or gender cues in the voice rather than on speaker-specific voice characteristics, which can lead to a performance deterioration when the model has to identify a
multilingual speaker or distinguis same-gender speakers. To investigate these issues, we introduce the Face-voice Association across
LAnguages and Gender (FLAG) 2027 Challenge. The challenge formulates face–voice association as a cross-modal verification task: given a voice, identify the speaker's face from a “gallery” of faces consisting of the speaker's face and a set of negative samples. Models are evaluated on identities not present in the training data (“unseen”) and both for
languages present or absent from the training data (“heard” and “unheard”). Two evaluation settings are used to test models' reliance on gender: a standard, unconstrained and a gender-constrained one, where the latter uses a same-gender gallery. The performance of existing, baseline models in these settings reveals that models performance degrades under
language shifts and in gender-constrained settings, highlighting the need to foster the development of models that capture identity-specific aspects beyond
language and gender. The challenge provides a
benchmark dataset, pretrained baseline models, and an evaluation framework to advance face–voice association.
Read the full abstract ▾
All authors Marta Moscati, Swapnil Khandoker, Muhammad Saad Saeed, Shah Nawaz, Fatima Noor, Rohan Kumar Das, Mubashir Noman, Junaid Mir, Muhammad Haroon Yousaf, Khalid Malik, Markus Schedl
Computer Vision
Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control
Mingyang Chen, Shengdong Chen, Xiaoxiao Fu et al. (+13 more)
We introduce Zing-0.5, a 5B autoregressive world model designed for playability: users can explore generated worlds, influence unfolding events, and respond to the resulting feedback through joint keyboard and online text control. Our approach brings together three technical contributions: (1) Unified action and text conditioning, combining magnitude-aware keyboard inputs with temporally aligned text instructions and jointly annotated
videos to learn
navigation and event control within the same sequence; (2) Event-scale supervision for incremental generation, using a segment-level teacher trained on connected multi-
prompt videos to supervise a block-level causal student through distribution-matching
distillation; and (3) Low-cost real-time interaction, combining four-step generation with context-preserving streaming to support 832 x 480 inference at 24 FPS at an estimated server rental cost of approximately USD 0.009 per stream-minute. Zing-0.5 achieves an overall score of 81.0 and a consistency score of 88.5 across 158 WBench
Navigation cases. A joint-control demonstration shows a text-directed event change during continued
navigation without restarting generation. We release the model weights, inference code, and Zing-SGLang serving implementation to support further work on playable generated worlds.
Read the full abstract ▾
All authors Mingyang Chen, Shengdong Chen, Xiaoxiao Fu, Bosheng Gong, Haoyuan Guo, Bowen Li, Jiawen Li, Kejun Li, Tianpeng Li, Yin Liu, Haoze Sun, Zeyang Tian, Meng Wang, Xinmiao Wu, Jiangqiao Yan, Zining Zhao
Computer Vision · Cross-listed from cs.AI
ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software
Kratika Bhagtani, Kusha Sridhar, Maziyar Baran Pouyan et al. (+2 more)
Computer-use
agents that operate through screenshots and simulated actions are advancing rapidly, yet their evaluation remains anchored to general desktop and web tasks. Enterprise Resource Planning systems run the finance, procurement, inventory, and customer operations of organizations worldwide, and pose distinct challenges for computer-use
agents: dense interfaces, coordinated multi-step interactions, and errors that alter persistent business records rather than surfacing on screen. Existing enterprise computer-use
benchmarks rely on proprietary platforms or on simulated approximations of such software. We introduce ERPBench, a
benchmark that evaluates screenshot-only
agents on a live and reproducible system and scores each task against ground-truth values in its database. Beyond the
benchmark, we present a production-grade harness that gates
agent actions behind human approval for safe deployment. Evaluating six closed and open-source
agents, we demonstrate that strong general performance does not transfer to enterprise reliability. Even when an
agent reaches the right form and saves it, the stored record is often wrong: some
agents save in up to 85% of runs but write the correct value in as few as 3%. We further characterize failure modes specific to enterprise workflows.
Read the full abstract ▾
All authors Kratika Bhagtani, Kusha Sridhar, Maziyar Baran Pouyan, Yuying Zhao, Eugene Siow
Computer Vision
Can VLMs Reliably Assess Sidewalk Accessibility Attributes from Pedestrian-Level Imagery?
Seung Jae Lieu, Diego Morra, Chiara Cadoni et al. (+3 more)
An important component of urban accessibility, particularly for wheelchair users and people with reduced mobility, is sidewalk compliance with measurable requirements. We test whether effective width, longitudinal slope, cross slope, and pavement condition can be assessed reliably from pedestrian-level imagery using
vision-
language models (VLMs). We present the first application of sampling-based conformal prediction (CP) for VLM-based accessibility assessment. We evaluate four VLMs on 514 sidewalk images from Seoul, South Korea, with field-measured ground truth. Conformal calibration attains the nominal 90% coverage for all models and attributes, but the calibrated regions differ in informativeness. Effective width yields the most informative estimates, with a mean interval half-width of about 1.0 m for the best model. Since every model overestimates width, asymmetric calibration shortens the intervals by up to 33% at unchanged coverage. Longitudinal slope is marginally informative, cross-slope intervals are too wide to resolve regulatory thresholds, and pavement-condition sets degenerate to all five grades (A-E) for three of the four models. Uncalibrated intervals from raw sampling dispersion cover only 17-47% of field-measured values at a nominal 90% level. Among the images with the most self-consistent responses, these intervals miss the field-measured value in up to 96% of cases. Response self-consistency is therefore not evidence of accuracy, and sampling dispersion cannot be interpreted as uncertainty until it has been calibrated against field-measured ground truth. No quantitative attribute reaches the precision required for general compliance assessment, but CP identifies from calibration data alone which attributes can support screening of segments far from the thresholds. We release the annotated pedestrian-level images and their corresponding field-measured attribute values.
Read the full abstract ▾
All authors Seung Jae Lieu, Diego Morra, Chiara Cadoni, Wonseop Song, Martina Mazzarello, Carlo Ratti
Computer Vision
Lumen: Parameter-Efficient Alignment of Pretrained Vision and Language Encoders for Zero-Shot Computational Pathology
Kiarash Tajbakhsh, Abdelrahman Faqieh, Michael Jopiti et al. (+9 more)
Pathology
vision-
language models are commonly built by pretraining or fine-tuning large encoders on paired image-caption data. We asked whether a pathology
vision-
language model can instead be assembled by parameter-efficient
alignment of frozen unimodal foundation models, leaving their pretrained representations untouched. Here we present Lumen, which aligns frozen Virchow2 and BioMedBERT backbones using rank-4 adapters and projection heads, training only 0.40% of the total parameters on the public QUILT-1M corpus. Across nine public zero-shot patch
benchmarks, Lumen achieved the highest mean chance-corrected balanced accuracy, 0.546 versus 0.461 for the strongest baseline (paired difference 0.086, 95% CI 0.042-0.136). On lymph-node metastasis detection, Lumen reached an AUROC of 0.964 (95% CI 0.956-0.971) on 4,214 held-out internal slides and 0.955 (95% CI 0.942-0.966) on 2,368 slides across nine external cohorts and six organs. At the internally calibrated threshold, it outperformed all
vision-
language baselines, with a balanced accuracy of 0.909 (95% CI 0.896-0.923) internally and 0.915 (95% CI 0.902-0.929) externally. Lumen performed competitively across the evaluations, with the exception of cross-modal
retrieval, where it ranked third behind CONCH and PathGen-L/14. Fully fine-tuning both encoders gave Lumen no consistent benefit over low-rank adaptation, although it improved
retrieval. Aligning frozen unimodal foundation models therefore yields strong and transferable performance at patch and slide level while training only a small fraction of the parameters.
Read the full abstract ▾
All authors Kiarash Tajbakhsh, Abdelrahman Faqieh, Michael Jopiti, Javier Garcia-Baroja, Philipp Zens, Branislav Zagrapan, Yuri Tolkach, Martin D. Berger, Aurel Perren, Bastian Dislich, Inti Zlobec, Amjad Khan
Computer Vision
Investigating Adversarial Robustness of Heterogeneous Cooperative Perception
Chenyi Wang, Yutong Liu, Qingzhao Zhang et al. (+1 more)
Heterogeneous cooperative perception (CP) enables connected vehicles with diverse sensor setups to share spatial awareness via compact feature maps, where receivers reconcile these maps using learned translation modules for fusion and inference. Prior attacks against CP in a homogeneous setting reveal that the data exchange introduces a critical attack surface: a single malicious
agent can transmit crafted features that erase real objects from a neighbor's fused scene. Yet, it is widely hypothesized that heterogeneity naturally defends against these attacks, as the attacker lacks knowledge of the victim's detector and the translation module scrambles
adversarial gradients. We demonstrate that this protection is largely an illusion. Using a matched-objective harness to standardize the perturbation budget, objective, and forward path, we show that properly tuned iterative attacks close or reverse the apparent robustness gap. However, these optimization-based attacks require ground-truth labels and iterative backpropagation, meaning they do not represent a practical field threat running in real-time. To bridge this gap, we introduce HetPoison, a learned generator that crafts a removal perturbation in a single, label-free forward pass. HetPoison transfers across major heterogeneous designs without requiring access to the victim's detector, matching or exceeding the effectiveness of expensive optimizer-based attacks. Since heterogeneity itself is not a defense, we propose HetShield, a lightweight trust layer that validates the spatiotemporal consistency across features, recovering 83–95% of the accuracy degraded by attacks, outperforming prior art.
Read the full abstract ▾
All authors Chenyi Wang, Yutong Liu, Qingzhao Zhang, Ming F. Li
Computer Vision
RoboVAD: A Large Cross-Domain Evaluation Benchmark for Anomaly Detection in Robotic Arm Manipulation Videos
Alexandru-Bogdan Dura, Sebastian Balmus and Radu Tudor Ionescu
Video anomaly detection (VAD) is an actively studied task, having wide applications in typical scenarios such as public surveillance and road traffic safety. The task is also relevant for robotic arm interactions, where it has several downstream applications, including learning better interaction and manipulation abilities, triggering recovery procedures when anomalies occur, etc. Despite its relevance, the exploration of anomaly detection in robotic arm manipulation
videos is limited by the low number of available resources. To this end, we introduce RoboVAD, a large-scale
benchmark for
video anomaly detection that comprises challenging cross-domain evaluation scenarios, where certain actions (tasks executed by a robotic arm) and anomaly types (mistakes that occur while performing certain tasks) remain unseen during training. RoboVAD is designed to
benchmark VAD methods in realistic scenarios, where robotic arms can perform unforeseen tasks, and thereby encounter new anomaly types. We train and evaluate several state-of-the-art VAD methods, including a novel method specifically adapted for robotic arm manipulation. While the proposed method outperforms many state-of-the-art competitors, all methods remain below a micro-averaged frame-level AUC threshold of 70% in the most challenging evaluation setup, confirming the difficulty of the proposed
benchmark. We publicly release our
dataset and code at https://zenodo.org/records/22754659.
Read the full abstract ▾
Computer Vision
CALIPER: Metric-Grounded Model-Free Recognition of Visually Similar Industrial Parts
Alankrit Gupta, Chenxi Tao and Seung-Kyum Choi
Fine-grained recognition of visually similar industrial parts is challenging when classes differ primarily in physical dimensions. Normalizing detected object crops to a fixed input size suppresses absolute scale, while CAD models and large class-specific
datasets may be unavailable in evolving industrial inventories. We present CALIPER, a model-free RGB-D framework that couples support-based appearance matching with metric size evidence. Each training class is onboarded from a single turntable RGB-D
video and one to two labeled real images; 3D reconstruction provides novel-view appearance support, while aligned depth yields a class-specific metric size profile. At inference, a coarse YOLOv8n-seg model localizes parts, and a frozen DINOv2 backbone with an episodically trained
embedding head performs fine-grained support matching. Margin-conditioned metric fusion activates probabilistic size evidence only for appearance-ambiguous decisions. New classes are enrolled from a small RGB-D support set without updating network parameters. We evaluate CALIPER on 18 visually similar industrial parts: 16 classes are used for training, while two screws are reserved for training-free enrollment. CALIPER achieves 88.2% closed-set accuracy with 99.8% localization recall and 85.7% overall accuracy after 10-shot enrollment of the two unseen screws. Metric fusion improves unseen-class accuracy by up to 37.4 percentage points without statistically significant degradation of the original inventory.
Robot-arm deployment identifies 17/18 parts without deployment-specific retraining.
Read the full abstract ▾
Computer Vision
GazeDiT: Gaze-Accurate Diffusion Image Generation for Eye Tracking via Spatial Conditioning
Dongze Wu, David Colmenares, Fengting Yang et al. (+4 more)
Diffusion models are increasingly used to generate synthetic training data, but precise label control remains difficult when the conditioning signal is low-dimensional and coarse. Text-conditioned images are judged by broad
prompt consistency, whereas supervised training requires precise correspondence between each image and its numerical label. This is challenging in eye tracking, where a 4D binocular gaze is expressed through subtle, spatially localized pupil and iris geometry. We introduce GazeDiT, a
diffusion model that generates images for a requested 4D gaze through an internally constructed spatial condition that grounds the global gaze label in this local geometry. During training, a frozen SegFormer extracts pupil/iris geometry from diverse real images, allowing the model to learn realistic appearance conditioned on that geometry. At inference, a physical eye renderer samples gaze-consistent geometries by varying anatomy and camera state, enabling diverse synthesis without a source image. GazeDiT achieves substantially lower tail gaze-label error than other
diffusion baselines, approaching the error of the same frozen gaze
estimator on real images. Its generated data also improves the downstream eye tracker, reducing gaze error on difficult cases from 3.05{\deg} to 2.80{\deg} in the smallest cohort.
Read the full abstract ▾
All authors Dongze Wu, David Colmenares, Fengting Yang, Jogendra Nath Kundu, Yao Xie, Ali Behrooz, Conny Lu
Computer Vision
Wind on Trees: Testing Physical Grounding in Dynamic 4D Gaussian Splatting
Weiying Chen and Edmond Lou
Monocular reconstruction of wind-driven vegetation is severely underconstrained: motion along the viewing direction is largely unobservable, a moving canopy offers few reliable correspondences, and nearly the entire scene is dynamic, providing little static reference. Directly-learned deformation fields in 4D Gaussian Splatting therefore optimize photometric consistency rather than recover the motion that produced it. We replace that field with a physically parameterized deformation prior: one damped harmonic oscillator per rigid part, driven by the observed wind and integrated by differentiable RK4, supervised photometrically alone. To test whether such a prior is physically grounded rather than merely well fit, we build a controlled synthetic testbed of three procedurally generated trees spanning an order of magnitude in skeleton complexity, whose per-part natural frequency follows from its own geometry and whose damping ratio is a fixed constant, both held out of training. On it, we measure held-out views, temporal extrapolation, zero-shot transfer to unseen wind speeds, and recovery of the physical parameters themselves. The prior costs appearance fidelity on in-distribution views and extrapolates markedly better outside the training window and the training wind, while parameter recovery is far weaker than it first appears: frequency recovery survives an untrained null control on only the sparsest of the three trees, and damping is not recovered at all.
Read the full abstract ▾
Computer Vision
AgenTeeth: A Model-Agnostic Framework for Suppressing Hallucination in Frozen Vision-Language Models on Dental X-Rays via Tool Evidence Injection
Ahmed Rafid, Fariya Ahmed, Rumman Adib et al. (+3 more)
Vision-
language models (VLMs) remain largely unreliable on panoramic dental radiographs and can rely on learned anatomical priors rather than evidence in the image. This is particularly problematic for tooth localization and spatial
reasoning, and fine-tuned dental VLMs can retain the same spatial biases. We present AgenTeeth, a model-agnostic, tool-augmented framework that grounds frozen VLMs using seven specialized dental
vision experts. A question-aware orchestrator selects the relevant tools, whose detections are mapped to FDI tooth numbers or anatomical regions and returned as structured findings together with annotated image overlays. A fresh synthesis call then answers the question using this evidence, without fine-tuning the underlying VLM. On MMOral-OPG-Bench, AgenTeeth improves four backbone VLMs by 12.9-23.0 percentage points over their baselines. Our strongest configuration reaches 65.66% on open-ended VQA, compared with 45.35% for OralGPT-Plus. The advantage also holds at matched scale: a frozen Qwen2.5-VL-7B-Instruct with AgenTeeth reaches 48.11%, exceeding OralGPT-Plus built on the same backbone after supervised fine-tuning and reinforcement learning for tool use. We release the framework, all seven expert models, and a dentist-annotated
dataset for alveolar bone-loss detection in panoramic radiographs.
Read the full abstract ▾
All authors Ahmed Rafid, Fariya Ahmed, Rumman Adib, Mehedi Ahamed, Ajwad Abrar, Tareque Mohmud Chowdhury
Computer Vision
Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Models
Akanksha Singh and Vinod K. Kurmi
Pre-trained
vision-
language models (VLMs) exhibit strong cross-domain recognition performance even without additional training. However, this robustness can also preserve undesirable domain-specific behavior, as domain-related and semantic information often remain entangled within the learned representation space, making selective domain unlearning challenging. Existing approaches typically address this problem through latent-space disentanglement and
prompt- or feature-level interventions, without directly attributing and attenuating individual patch-
token contributions. However, here we suggest that rather than uniformly suppressing the full representation, it may be more effective to exploit the spatial structure of
vision transformers to localize and suppress patch regions that contribute disproportionately to forget-domain prediction. Patches that strongly influence forget-domain prediction may not be equally important for semantic recognition, suggesting that forgetting should be guided according to the domain contribution of different visual regions. Specifically, we propose a two-stage patch-selective framework that first estimates patch-level domain sensitivity and then selectively attenuates patches whose contribution to forget-domain prediction is stronger than their semantic utility. We evaluate our framework on Office-Home, Mini DomainNet, and DomainNet. Experimental results demonstrate improved forgetting-retention tradeoffs compared to prior methods while improving retained-domain recognition by up to 3.8%. Additional evaluations under visually overlapping and unseen-domain settings further demonstrate improved robustness under distribution shift.
Read the full abstract ▾