Computer Vision
Generation of Heterogeneous PET Images from Uniform Organ Activity Maps Using a Pretrained Domain-Adapted Diffusion Model
Suya Li, Kaushik Dutta, Debojyoti Pal et al. (+2 more)
Synthetic PET images are valuable for quantitative imaging workflow development, scalable virtual imaging trials, and deep learning model training, but conventional physics-based simulation approaches are computationally intensive, limited in anatomical variability, and often fail to capture heterogeneous PET uptake. This study developed a pretrained domain-adapted
diffusion (PAD) model for anatomy-conditioned PET synthesis from uniform organ activity maps. PAD adopts a natural-image pretrained text-to-image decoder with an upstream conditioning encoder and a downstream PET-domain adapter. A two-phase training strategy was used, with the first phase learning coarse uptake distributions and the second refining local image details. Uniform organ activity maps were generated from CT-based
segmentations by assigning each organ its mean uptake from the paired PET image. Evaluation included quantitative accuracy, noise assessment, radiomic analysis, tumor
segmentation performance, and a human observer study. PAD-generated images achieved high quantitative accuracy, with concordance correlation coefficients above 0.92 between organ mean SUVs and assigned activity values. The synthesized images showed noise levels and texture characteristics similar to target PET images and produced comparable tumor
segmentation performance. In a two-alternative forced-choice observer study, four readers achieved approximately 50% accuracy, indicating visual indistinguishability between synthesized and target images. PAD also generated realistic PET images from XCAT-derived activity maps, demonstrating compatibility with phantom-based anatomical priors. Overall, PAD provides a
diffusion-based framework for generating clinically relevant heterogeneous PET images from uniform organ activity maps derived from clinical
segmentations or digital phantoms, supporting data augmentation and downstream imaging studies.
Read the full abstract ▾
All authors Suya Li, Kaushik Dutta, Debojyoti Pal, Jingqin Luo, Kooresh I. Shoghi
Computer Vision · Cross-listed from cs.IR
Efficient Table QA via TableGrid Navigation and Progressive Inference Prompting
Amritansh Maurya, Navjot Singh, Mohammed Javed et al. (+1 more)
Large
Language Models (LLMs) have shown promising results on NLP tasks, however, their performance on tabular data still needs research
attention, because Table Question-Answering (TQA) requires precise
cell retrieval and multi-step structured
reasoning. Existing work improves TQA either by fine-tuning or training LLMs on task-specific tabular data, but often lacks verifiable control over how the model navigates tables and derives answers. In this work, we propose a training-free TQA approach with two structured prompting frameworks: TableGrid
Navigation (TGN), which iteratively navigates rows and columns via a three-module loop to locate evidence and refine answers, and Progressive Inference Prompting (PIP), which enforces columns identification for explicit progressive row selection constraint according to the query. We evaluate 17 LLMs against 6 baselines on TableBench and FeTaQa
dataset. On TableBench, TGN improves over the strongest baseline by 3.8 points, and on FeTaQa, PIP achieves SOTA performance over ReAct and Chain-of-Thought. Beyond inference-time gains, PIP and TGN can also serve as supervision templates to fine-tune small models, narrowing the performance gap to much larger architectures in resource-constrained settings, offering versatile and cost-efficient solution for TQA.
Read the full abstract ▾
All authors Amritansh Maurya, Navjot Singh, Mohammed Javed, Omar Moured
Computer Vision · Cross-listed from cs.LG
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
Yang Liu, Toan Nguyen and Flora D. Salim
Catastrophic forgetting remains a major obstacle to continual learning in large
language models (LLMs) and
vision–
language models (VLMs). Although Mixture-of-Experts (MoE) architectures offer an efficient path to scaling, existing LoRA-based MoE continual learning methods still face a fundamental trade-off: they either isolate experts too aggressively, limiting knowledge transfer across tasks, or allow task-specific updates to overwrite important existing parameters, leading to severe forgetting. To address this, we propose CP-MoE, a continual learning framework built around a transient expert that captures early task-specific updates and guides their integration into stable experts. CP-MoE introduces a consistency-preserving routing bias, which uses the transient expert to estimate representation similarity with stable experts and steer routing towards more compatible expert selection, and a transient expert-guided regularisation mechanism, which selectively protects important historical parameters during merging. Together, these components reduce parameter interference and forgetting while preserving cross-task knowledge transfer. We validate CP-MoE on both unimodal and
multimodal continual learning
benchmarks with LLM-based and VLM-based MoE models. On SuperNI
benchmark, spanning diverse sequential
language tasks, CP-MoE achieves state-of-the-art performance and stronger zero-shot transfer to unseen tasks. On VQA v2
dataset, it scales effectively to
multimodal visual
reasoning, consistently reduces forgetting, and outperforms strong MoE baselines.
Read the full abstract ▾
Computer Vision
Efficient coding along the visual hierarchy
Ananya Passi, Brian S. Robinson and Michael F. Bonner
Biological visual systems learn from limited experience, unlike deep learning models that rely on millions of training images. What learning principles make this possible? We tested whether efficient coding, the idea that neural representations capture the statistical structure of natural inputs, can build a hierarchy of human-aligned visual features from limited data. We developed an unsupervised learning procedure in which each layer of a deep network compresses its inputs onto the dominant modes of variation in natural images, using only local statistics and no labels, tasks, or backpropagation. This unsupervised procedure yields features that progress from edges and colors to textures and shapes. The features of this deep efficient coding model are readily recognized by human observers and are predictive of image-evoked fMRI responses in human visual cortex. Furthermore, a hybrid learning procedure that combines efficient coding with supervised fine-tuning yields better
brain alignment in low-data settings and more rapid category learning. These findings suggest that efficient coding may shape representations across the entire visual hierarchy and help explain the data efficiency of biological
vision.
Read the full abstract ▾
Computer Vision
Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models
Svetlana Orlova, Niccolò Cavagnero and Gijs Dubbelman
Video foundation models achieve strong performance across many
video understanding tasks, but typically require large-scale pre-training on massive
video datasets, resulting in substantial data and compute costs. In contrast, modern image foundation models already provide powerful spatial representations. This raises an important question: can competitive
video models be built by reusing these spatial representations and pre-training only for temporal
reasoning? We take initial steps toward exploring a lightweight training paradigm that freezes a pre-trained image foundation model and trains only a recurrent temporal module to process streaming
video. By reusing an image foundation model as a spatial encoder, this approach could significantly reduce the amount of
video data and compute required compared to end-to-end
video pre-training. In this work, we explore the feasibility of this approach before investing in computing for
video pre-training. Our empirical findings across multiple
video understanding tasks suggest that strong temporal performance can emerge without large-scale
video pre-training, motivating future work on recurrent
video foundation models obtained by pre-training a temporal module on top of a frozen image foundation model. Code: https://github.com/tue-mps/towards-
video-image-frozen .
Read the full abstract ▾
Computer Vision
Knowing When Not to Predict: Self Supervised Learning and Abstention for Safer DR Screening
Muskaan Chopra, Lorenz Sparrenberg, Jan H. Terheyden et al. (+1 more)
Self-supervised learning (SSL) is now a standard way to pretrain medical image models, but performance is still mostly judged by downstream accuracy. For safety-critical screening tasks such as diabetic retinopathy grading, this is not enough: a model must also know when its predictions are unreliable and defer uncertain cases for clinical review. In this work, we examine how the length of SSL pretraining influences calibrated confidence and confidence-based abstention. We evaluate multiple SSL checkpoints under a fixed fine-tuning protocol and assess calibrated confidence, coverage, selective accuracy, and selective macro-F1. Across
datasets and data regimes, SSL pretraining improves selective prediction compared to training from scratch. Unlike prior SSL studies that primarily evaluate downstream accuracy or AUROC, we analyze how SSL pretraining duration influences confidence behavior under calibrated confidence-based abstention. However, once accuracy saturates, selective performance can still change markedly across checkpoints, and longer pretraining does not consistently improve reliability. These results underscore the importance of abstention-aware evaluation and suggest that pretraining length should be treated as an important reliability-related design choice rather than only a computational detail. Code is available at GitHub.
Read the full abstract ▾
All authors Muskaan Chopra, Lorenz Sparrenberg, Jan H. Terheyden, Rafet Sifa
Computer Vision · Cross-listed from cs.LG
EgoBabyVLM: Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Data
Dongyan Lin, Phillip Rust, Angel Villar Corrales et al. (+19 more)
Children acquire
language grounding with remarkable robustness from limited visuo-linguistic input in ways that surpass today's best large
multimodal models. Recent research suggests current
vision-
language models (VLMs) trained on curated web data fail to generalize to the sparse, weakly-aligned
egocentric streams produced by wearable devices,
embodied agents, and infant head-cams – and no fixed evaluation pipeline exists for measuring progress on this regime. We train VLMs on
datasets with varying degrees of semantic
alignment between visual and linguistic inputs, including naturalistic infant and adult
egocentric videos, and evaluate them with a comprehensive suite spanning
multimodal language grounding and unimodal
vision and
language tasks. At the core of this suite is Machine-DevBench, a corpus-grounded
benchmark of lexical and grammatical competence, automatically generated from the model's training vocabulary across logarithmic frequency bins to eliminate the train/eval mismatch and low statistical power of prior developmental
benchmarks. Our results show that current VLM paradigms hinge on the tight semantic
alignment of curated data and fail to exploit the weakly-aligned signal that dominates naturalistic
egocentric input – the very regime in which humans thrive. To motivate progress, we introduce the EgoBabyVLM Challenge to drive the development of models capable of grounded
language learning from the kind of naturalistic data that human infants experience.
Read the full abstract ▾
All authors Dongyan Lin, Phillip Rust, Angel Villar Corrales, Alvin W. M. Tan, Mahi Luthra, Charles-Éric Saint-James, Rashel Moritz, Sheila Krogh-Jespersen, Vanessa Stark, Surya Parimi, Jiayi Shen, Youssef Benchekroun, Yosuke Higuchi, Martin Gleize, Tom Fizycki, Nicolas Hamilakis, Manel Khentout, Sho Tsuji, Balázs Kégl, Juan Pino, Michael C. Frank, Emmanuel Dupoux
Computer Vision
FAGER: Factually Grounded Evaluation and Refinement of Text-to-Image Models
Youngsun Lim, Cusuh Ham, Pin-Yu Chen et al. (+1 more)
Existing text-to-image (T2I) evaluation metrics mainly assess whether generated images align with information explicitly stated in the
prompt, but often fail to capture factual requirements that are implicit, externally grounded, or identity-defining. As a result, they are not well suited for evaluating factual correctness in
prompts involving scientific knowledge, historical facts, products, or culture-specific concepts. We propose FActually Grounded Evaluation and Refinement (FAGER), an agentic framework that evaluates whether generated images correctly reflect visually verifiable facts grounded in or implied by the
prompt, while also providing actionable feedback for improvement. FAGER first constructs a structured factual rubric by combining LLM-based fact proposal with reference-guided visual fact extraction and verification, then converts the rubric into question-answer pairs for VLM-based evaluation. To validate FAGER as a factuality metric, we introduce a Factual A/B test, which measures whether a metric prefers factual reference images over corresponding generated images. Across five
datasets spanning science, history, products, culture, and knowledge-intensive concepts, FAGER consistently outperforms prior metrics on this test. We further show that FAGER can be used to refine T2I outputs in a fully training-free manner, yielding substantial factuality gains across
datasets.
Read the full abstract ▾
All authors Youngsun Lim, Cusuh Ham, Pin-Yu Chen, Deepti Ghadiyaram
Computer Vision
CRAFT: Critic-Refined Adaptive Key-Frame Targeting for Multimodal Video Question Answering
Mahesh Bhosale, Abdul Wasi, Vishvesh Trivedi et al. (+3 more)
Grounded multi-
video question answering over real-world news events requires systems to surface query-relevant evidence across heterogeneous
video archives while attributing every claim to its supporting source. We introduce CRAFT (Critic-Refined Adaptive Key-Frame Targeting), a query-conditioned pipeline that combines dynamic keyframe selection, per-
video ASR with
multilingual fallback, and a hybrid critic loop to iteratively verify and repair claims before consolidation. The pipeline integrates UNLI temporal entailment, DeBERTa-v3 cross-claim screening, and a Llama-3.2-3B adjudicator, with a final citation-merging stage that emits each fact once with all supporting source identifiers. On MAGMaR 2026, CRAFT achieves the best overall average (0.739), reference recall (0.810), and citation F1 (0.635). We further evaluate on a MAGMaR-style conversion of WikiVideo with 52 non-overlapping event queries, where CRAFT also performs strongly (0.823 Avg), showing that its claim-centric evidence aggregation generalizes beyond MAGMaR. Ablations show that atomic claims, ASR, and the critic loop drive the main gains over the vanilla query-conditioned baseline. Code and implementation details are publicly available at https://github.com/bhosalems/CRAFT.
Read the full abstract ▾
All authors Mahesh Bhosale, Abdul Wasi, Vishvesh Trivedi, Pengyu Yan, Akhil Gorugantu, David Doermann
Computer Vision
Learning Long-Term Temporal Dependencies in Photovoltaic Power Output Prediction Through Multi-Horizon Forecasting
Sumit Laha, Ankit Sharma and Hassan Foroosh
The rapid global expansion of
solar photovoltaic (PV) capacity-reaching a record 597 GW in 2024-highlights the urgent need for robust
forecasting models to mitigate the grid instability caused by the intermittent nature of
solar irradiance. While deep learning-based direct
forecasting using ground-based sky images (GSI) has emerged as a dominant approach, existing literature is often constrained by single-architecture evaluations and an exclusive focus on single-horizon (point) prediction. This paper proposes a transition from traditional single-horizon estimation toward a multi-horizon
forecasting framework, leading to an architecture-independent improvement in accuracy. We hypothesize and demonstrate experimentally that joint optimization over a sequence of future values allows deep neural networks to better capture latent inter-step temporal dependencies by avoiding precocious convergence of the network in terms of both weight gradients and filter diversity. Leveraging this architecture-independent improvement that integrates sequential sky imagery with historical PV generation data, we evaluate the models' abilities to predict power output across multiple discrete future time steps simultaneously. Our methodology is validated through a comparative analysis across diverse deep learning architectures. The results demonstrate that this multi-horizon approach significantly enhances predictive accuracy and robustness across the entire forecast horizon while maintaining computational parsimony. By achieving superior performance with negligible overhead compared to single-horizon models, this work provides a scalable and efficient solution to improve the resilience of modern power grids.
Read the full abstract ▾
Computer Vision
Efficient Factorized Volume Synthesis with Lifted Inter-Slice Feature Trajectories
Xinhe Zhang, Yuyang Zhang, Pengfei Jin et al. (+3 more)
High-resolution 3D medical image generation remains challenging because fully volumetric models are computationally expensive, while efficient 2D slice generators often fail to preserve anatomical consistency across the third dimension. We propose LiFT, a framework for Lifted inter-slice Feature Trajectories that factorizes 3D volume synthesis into per-slice image generation and inter-slice trajectory learning. Rather than using a fully volumetric synthesis backbone, LiFT treats a volume as an ordered trajectory in feature space, capturing how anatomical structures appear, transform, and disappear across depth. We demonstrate the LiFT framework in both unconditional and conditional settings. In unconditional generation, a tri-planar drifting loss matches feature distributions between generated and real volumes across three orthogonal planes; in paired translation, a bidirectional $z$-context mixer trained with paired target supervision supplies through-plane coherence while preserving per-slice fidelity. We evaluate LiFT on BraTS 2023 for unconditional and missing-modality MR synthesis, and on SynthRAD2023 for MR-to-CT translation. LiFT achieves competitive image quality while retaining the efficiency of predominantly 2D synthesis, operating up to $390\times$ faster than a volumetric
diffusion baseline and consistently improving through-plane coherence through explicit inter-slice modeling. These results establish lightweight inter-slice trajectory learning as an efficient approach to high-resolution 3D medical synthesis.
Read the full abstract ▾
All authors Xinhe Zhang, Yuyang Zhang, Pengfei Jin, Arnau Marin-Llobet, Na Li, Quanzheng Li
Computer Vision · Cross-listed from cs.RO
RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuning
Ehsan Ahmadi, Hunter Schofield, Behzad Khamidehi et al. (+5 more)
Supervised open-loop training has been widely adopted for training traffic simulation models; however, it fails to capture the inherently dynamic, multi-
agent interactions common in complex driving scenarios. We introduce RLFTSim, a reinforcement-learning-based fine-tuning framework that enhances scenario realism by aligning simulator rollouts with real-world data distributions and provides a method for distilling goal-conditioned controllability in scenario generation. We instantiate RLFTSim on top of a pre-trained simulation model, design a reward that balances fidelity and controllability, and perform comprehensive experiments on the Waymo Open Motion
Dataset. Our results show improvements in realism, achieving state-of-the-art performance. Compared with other heuristic search-based fine-tuning methods, RLFTSim requires significantly fewer samples due to a proposed low-variance and dense reward signal, and it directly addresses the realism
alignment issue by design. We also demonstrate the effectiveness of our approach for distilling traffic simulation controllability through goal conditioning. The project page is available at https://ehsan-ami.github.io/rlftsim.
Read the full abstract ▾
All authors Ehsan Ahmadi, Hunter Schofield, Behzad Khamidehi, Fazel Arasteh, Jinjun Shan, Lili Mou, Dongfeng Bai, Kasra Rezaee
Computer Vision
Personalized Face Privacy Protection From a Single Image
Zachary Yahn, Fatih Ilhan, Tiansheng Huang et al. (+5 more)
Photos of faces uploaded online are vulnerable to malicious actors who can scrape facial images from online sources and intrude on personal
privacy via unauthorized use of facial recognition models. This paper presents FaceCloak, a novel personalized face
privacy protection system, which can generate defensive identity-specific universal face
privacy masks from a single image of a user, causing facial recognition to fail. FaceCloak introduces a three-stage personalized face perturbation learning methodology: (1) It generates a small set of high-variety synthetic face images of a person based on a single image of the person. (2) It learns face cloaking by adding more protection to key facial-identity leakage regions through iterative perturbation generation over the small set of synthetic images, effectively shifting a user's identity
embedding towards a distant anchor identity and away from a similar one. (3) It generates a personalized identity-protective mask in the form of pixel-wise cloaking, which is light-weight and can be efficiently applied to any facial image of a user while maintaining good perceptual quality. Extensive experiments on three popular face
datasets across ten recognition models show the effectiveness of FaceCloak compared to 29 other existing representative methods. Code is available at https://github.com/zacharyyahn/FaceCloak
Read the full abstract ▾
All authors Zachary Yahn, Fatih Ilhan, Tiansheng Huang, Selim Tekin, Sihao Hu, Yichang Xu, Margaret Loper, Ling Liu
Computer Vision
MedFM-Robust: Benchmarking Robustness of Medical Foundation Models
Xiangxiang Cui, Tianjin Huang, Yifang Wang et al. (+2 more)
Medical foundation models have achieved remarkable clinical performance, yet their robustness under real-world perturbations remains underexplored. We present a robustness
benchmark comprising 40 perturbation types (12 base, 28 medical-specific) across eight imaging modalities, evaluating five VLMs (LLaVA-Med, MedGemma, MedGemma-1.5, Gemini-2.5-flash and GPT-4o-mini) on VQA, visual grounding, and captioning, alongside two
segmentation models (MedSAM, SAM-Med2D) with five fine-tuning strategies. Our findings reveal: (1) Fine-tuning strategy dominates robustness, with LoRA exhibiting nearly double the degradation of full fine-tuning, while SAM-Med2D's Adapter offers favorable efficiency-robustness trade-off. (2) Medical-specific perturbations disproportionately damage
segmentation, with 9 of 15 top corruptions being domain-specific. (3) LoRA-tuned visual grounding drops over 40 points, whereas zero-shot captioning remains stable (<7% drop). Zero-shot VQA shows model-dependent robustness–medical models drop under 20% while Gemini-2.5-flash drops 54%. General-purpose VLMs achieve higher VQA accuracy but fail on grounding; among medical VLMs, MedGemma demonstrates the best overall stability. These results provide deployment guidelines and underscore the necessity of domain-specific robustness evaluation for medical AI. Our code is available at: https://abnerai.github.io/MedFM-Robust.
Read the full abstract ▾
All authors Xiangxiang Cui, Tianjin Huang, Yifang Wang, Lijie Hu, Lu Yin
Computer Vision
A Systematic Failure Analysis of Vision Foundation Models for Open Set Iris Presentation Attack Detection
Rahul Anand, Siddharth Singh, Dileep A D et al. (+2 more)
Vision foundation models have demonstrated strong transferability across diverse visual recognition tasks and are increasingly considered for biometric applications. Their suitability for iris Presentation Attack Detection (PAD), particularly under realistic open-set operating conditions, remains insufficiently examined. This work presents a systematic failure analysis of general-purpose
vision foundation models for open-set iris PAD using periocular imagery. Five representative foundation models are evaluated under three open-set protocols that explicitly separate different sources of distribution shift: unseen Presentation Attack Instruments (PAIs), unseen
datasets captured with different sensors and cross-spectral transfer from near-infrared (NIR) to visible spectrum (VIS) imagery. Both frozen feature representations and parameter-efficient task adaptation using Low-Rank Adaptation (LoRA) are assessed within a unified experimental framework. The results indicate that foundation models can transfer across
datasets with similar sensing characteristics, but fail to generalise reliably to unseen attack instruments and degrade sharply under cross-spectral evaluation. While LoRA improves performance in certain cross-
dataset settings, it frequently amplifies failure under attack-level and spectral shifts. Additional validation experiments using segmented iris inputs, full backbone fine-tuning, joint cross-
dataset and cross-PAI shifts, and reverse VIS to NIR transfer further confirm that these failures are not simply artefacts of periocular input, weak adaptation, or one-directional spectral evaluation. These findings show that strong closed-set or cross-
dataset performance should not be treated as evidence of robust open-set security, and highlight the need for PAD representations that maintain sensitivity to presentation artefacts while remaining stable under realistic deployment variation.
Read the full abstract ▾
All authors Rahul Anand, Siddharth Singh, Dileep A D, Mahadeva Prasanna, Raghavendra Ramachandra
Computer Vision
Ahmad Yehia, Abduallah Mohamed, Tianyi Wang et al. (+4 more)
Accurately
forecasting human trajectories from an
egocentric perspective plays a central role in applications such as
humanoid robotics, wearable sensing systems, and assistive
navigation. However, progress in this direction remains limited due to the scarcity of
egocentric trajectory
datasets collected in real-world environments. Addressing this need, we introduce EgoTraj, an
egocentric multimodal open
dataset recorded using Meta Quest Pro (MQPro). EgoTraj contains 75 sequences of human
navigation collected from multiple MQPro wearers in real-world urban environments. Each recording provides synchronized RGB
video along with ground-truth data, including continuous time-synchronized 6-degree-of-freedom head poses, per-frame 3D eye gaze vectors, scene annotations. To the best of our knowledge, EgoTraj differs from typical
egocentric trajectory
datasets by capturing long-horizon, self-directed
navigation across diverse urban routes with broad participant diversity. To demonstrate the potential of the
dataset, we
benchmark several state-of-the-art methods for
egocentric trajectory prediction and conduct ablation studies to analyze the contributions of gaze, scene, and motion cues. The results highlight the utility of EgoTraj for AR-based perception,
navigation, and assistive systems. The EgoTraj
dataset, code, and EgoViz Dashboard are publicly available at https://github.com/yehiahmad/EgoTraj.
Read the full abstract ▾
All authors Ahmad Yehia, Abduallah Mohamed, Tianyi Wang, Jiseop Byeon, Kun Qian, Junfeng Jiao, Christian Claudel
Computer Vision
Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos
Yuqi Tang, Yang Shi, Zhuoran Zhang et al. (+21 more)
Recent
video generative models have greatly improved the realism of AI-generated
videos, yet their outputs still exhibit artifacts such as temporal inconsistencies, structural distortions, and semantic incoherence. While
Multimodal Large
Language Models (MLLMs) show strong visual understanding capabilities, their ability to perceive and reason about such artifacts remains unclear. Existing
benchmarks often lack systematic evaluation of artifact-aware perception and fine-grained diagnostic
reasoning, especially across diverse AI-generated
video domains beyond photorealistic content. To address this gap, we introduce Artifact-Bench, a comprehensive
benchmark for evaluating MLLMs on AI-generated
video artifact detection and analysis. We first establish a three-level hierarchical taxonomy of realism artifacts, covering photorealistic, animated, and CG-style
videos. Based on this taxonomy, Artifact-Bench defines three complementary tasks: real vs. AI-generated
video classification, pairwise realism comparison, and fine-grained artifact identification. Experiments on 19 leading MLLMs reveal substantial limitations in artifact perception and
reasoning, with many models approaching random or even below-random performance in challenging settings. We further observe significant misalignment between MLLM judgments and human perceptual preferences, highlighting their limited reliability as general evaluators for AI-generated
video realism.
Read the full abstract ▾
All authors Yuqi Tang, Yang Shi, Zhuoran Zhang, Qixun Wang, Xuehai Bai, Yue Ding, Ruizhe Chen, Bohan Zeng, Xinlong Chen, Xuanyu Zhu, Bozhou Li, Yuran Wang, Yifan Dai, Chengzhuo Tong, Xinyu Liu, Yiyan Ji, Yujie Wei, Yuhao Dong, Shilin Yan, Fengxiang Wang, Yi-Fan Zhang, Haotian Wang, Yuanxing Zhang, Pengfei Wan
Computer Vision
Harnessing Self-Supervised Features for Art Classification
Federico Melis, Davide Bilardello, Emanuele Prato et al. (+2 more)
Classifying artworks presents a significant challenge due to the complex interplay of fine-grained details and abstract features that condition the style or genre of an artwork. This paper presents a systematic investigation of the effectiveness of supervised and self-supervised backbones as feature extractors for both artwork classification and
retrieval, with a particular focus on paintings. We conduct an extensive experimental evaluation using the DINO family and CLIP models, assessing multiple classification strategies and feature representations. Our results demonstrate that employing a self-supervised backbone leads to consistent improvements in artwork classification performance. Moreover, our work provides insights into the applicability of classification and
retrieval modules in real-world applications, such as virtual reality (VR) applications that support museum
navigation.
Read the full abstract ▾
All authors Federico Melis, Davide Bilardello, Emanuele Prato, Evelyn Turri, Lorenzo Baraldi
Computer Vision
MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation
Bizhu Wu, Jinheng Xie, Wenting Chen et al. (+5 more)
Recent motion-
language models unify tasks like comprehension and generation but operate at a coarse granularity, lacking fine-grained understanding and nuanced control over body parts needed for animation or interaction. This stems from fundamental issues in both the model and the data, in which the model can't focus on motion's localized pattern, and the training data lacks fine-grained supervision. To tackle this, we propose MotionMERGE, a unified framework that bridges the granularity gap. First, we pioneer the study of fine-grained languageguided motion control, including detailed understanding and localized editing, by explicitly modeling motion at part and temporal levels within a single LLM, thereby endowing the model with robust priors for precise control. Second, we design ReasoningAware Granularity-Synergy pre-training, a novel strategy that employs joint supervision for cross-granularity
alignment, temporal grounding, localized
alignment, motion coherency, and motion-grounded chain-of-thought (CoT)
reasoning. This equips the model with fine-grained motion-
language alignment, crossgranularity synergy, and explicit
reasoning ability. Third, we curate MotionFineEdit, a large-scale
dataset (837K atomic + 144K complex triplets) with the first fine-grained spatio-temporal corrective instructions and motion-grounded CoT annotations, establishing a new
benchmark for fine-grained text-driven motion editing and motion-grounded
reasoning. Extensive experiments demonstrate the capability of MotionMERGE for more precise motion generation, understanding, and editing, and compelling zero-shot generalization to other complex motion tasks. This work represents a significant step toward models that interact with motion in finer granularity and human-like
reasoning.
Read the full abstract ▾
All authors Bizhu Wu, Jinheng Xie, Wenting Chen, Zhe Kong, Jianfeng Ren, Linlin Shen, Ruibin Bai, Rong Qu
Computer Vision · Cross-listed from eess.IV
From Division to Decision: Leveraging Temporal Cell-Stage Segmentation for Embryo Transferability Prediction
Yasmine Hachani, Patrick Bouthemy, Elisa Fromont et al. (+3 more)
Accurate selection of bovine embryos is a challenging task, as current practice relies on a single expert assessment on the seventh day after insemination, resulting in high rates of pregnancy loss. Time-lapse videomicroscopy provides detailed information on early development, but is difficult to exploit because of complex motion patterns and time-consuming analysis. We propose TransFACT, a
transformer-based framework for modeling early developmental stages and embryo transferability using 2D time-lapse
videos from the first four days of development. TransFACT combines frame-level temporal features with stage-level representations, using developmental stages as auxiliary supervision to predict transferability on day four. Our experiments demonstrate that TransFACT, by leveraging an existing method designed for action recognition, achieves superior performance than its competitor in predicting embryo transferability.
Read the full abstract ▾
All authors Yasmine Hachani, Patrick Bouthemy, Elisa Fromont, Véronique Duranthon, Ludivine Laffont, Alline de Paula Reis
Computer Vision · Cross-listed from cs.MM
CounterFlow: A Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation
Gyubin Lee, Junwon Lee and Juhan Nam
We investigate Counterfactual
Video Foley Generation, which aims to adopt a sound-source identity that contradicts the visual evidence while remaining temporally synchronized to a silent
video. Existing
Video&Text-to-Audio (VT2A) models struggle with this, often remaining anchored to the visually implied sound source when
video and text contents disagree. We present ConterFlow, an inference-time dual-phase sampling scheme for pretrained flow-matching VT2A models. Phase 1 builds a
video-derived temporal structure while suppressing the visually implied source; Phase 2 drops
video conditioning to focus entirely on shaping audio timbre toward the target
prompt. ConterFlow substantially improves counterfactual
Video Foley generation compared to naive negative prompting and state-of-the-art baselines. To evaluate replacement quality, we propose a metric leveraging a text-audio co-
embedding space to measure both target-
prompt evidence and residual visually implied source leakage.
Video demonstrations and code are available at https://gyubin-lee.github.io/counterflow-demo/
Read the full abstract ▾
Computer Vision
Can These Views Be One Scene? Evaluating Multiview 3D Consistency when 3D Foundation Models Hallucinate
Soumava Paul, Prakhar Kaushik and Alan Yuille
Multiview 3D evaluation assumes that the images being scored are observations of one static 3D scene. This assumption can fail in NVS and sparse-view reconstruction: inputs or generated outputs may contain artifacts, outlier frames, repeated views, or noise, yet still receive high 3D consistency scores. Existing reference-based metrics require ground truth, while ground-truth-free metrics such as MEt3R depend on learned reconstruction backbones whose failure modes are poorly characterized. We study this reliability problem by comparing neural reconstruction priors with classical
geometric verification. We introduce \benchmark, a controlled robustness
benchmark for multiview 3D consistency, and a parametric family that decomposes neural metrics into backbone, residual, and aggregation components. This family recovers MEt3R and yields variants up to $3\times$ more robust. Our analysis shows that VGGT, MASt3R,
DUSt3R, and Fast3R can hallucinate dense geometry and cross-view support for unrelated scenes, repeated images, and random noise. We introduce COLMAP-based metrics that use matches, registration, dense support, and reconstruction failure as failure-aware consistency signals. On real NVS outputs and a structured human study, these metrics achieve up to $4\times$ higher correlation with human judgments than MEt3R.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.SD
WavFlow: Audio Generation in Waveform Space
Feiyan Zhou, Luyuan Wang, Shoufa Chen et al. (+6 more)
Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D
token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic
alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality
video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. Experimental results show that WavFlow achieves competitive performance on the
video-to-audio
benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio
benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding the performance of established latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for
multimodal audio generation.
Read the full abstract ▾
All authors Feiyan Zhou, Luyuan Wang, Shoufa Chen, Zhe Wang, Zhiheng Liu, Yuren Cong, Xiaohui Zhang, Fanny Yang, Belinda Zeng
Computer Vision
Aurora: Unified Video Editing with a Tool-Using Agent
Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao et al. (+4 more)
Recent
video editing models have converged on a unified conditioning design: a single
diffusion transformer jointly consumes text, source
video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic
video editing framework that pairs a tool-augmented
vision-
language model (VLM)
agent with a unified
video diffusion transformer. The VLM
agent maps a raw user request to a structured edit plan aligned with the
transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM
agent with supervised data for complete edit planning and reference-image selection, together with preference pairs for robust tool use and instruction refinement. We introduce AgentEdit-Bench to evaluate
agent-enhanced
video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing
video editing
benchmarks show that Aurora improves over instruction-only baselines and that the VLM
agent transfers to compatible frozen
video editing models. Project page: https://yeates.github.io/Aurora-Page
Read the full abstract ▾
All authors Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, Jiebo Luo
Computer Vision
ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop
Yining Hong, Jiageng Liu, Han Yin et al. (+5 more)
Spatial intelligence unfolds through a perception-action loop:
agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive
benchmark for
embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems.
Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with
agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes
reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more
embodied interaction alone can close.
Read the full abstract ▾
All authors Yining Hong, Jiageng Liu, Han Yin, Manling Li, Leonidas Guibas, Li Fei-Fei, Jiajun Wu, Yejin Choi
Computer Vision
Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation
Qianhao Yuan, Jie Lou, Xing Yu et al. (+4 more)
Multimodal Large
Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image. We observe a regional-to-global perception gap: the same MLLM answers fine-grained questions more accurately when conditioned on evidence-centered crops than on the corresponding full images, suggesting that many failures stem from difficulty to focus on relevant evidence rather than insufficient local recognition ability. Motivated by this observation, we propose
Vision-OPD (
Vision On-Policy
Distillation), a regional-to-global self-
distillation framework that transfers the model's own privileged regional perception to its full-image policy.
Vision-OPD instantiates two conditional policies from the same MLLM: a crop-conditioned teacher and a full-image-conditioned student. The student generates on-policy rollouts, and
Vision-OPD minimizes
token-level divergence between the teacher and student next-
token distributions along these rollouts. This enables the model to internalize the benefit of visual zooming without external teacher models, ground-truth labels, reward verifiers, or inference-time tool use. Experiments on multiple fine-grained visual understanding
benchmarks show that
Vision-OPD models achieve competitive or superior performance against much larger open-source, closed-source, and "Thinking-with-Images" agentic models. The code is available at https://github.com/VisionOPD/
Vision-OPD
Read the full abstract ▾
All authors Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, Yaojie Lu
Computer Vision
LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation
Yukang Chen, Luozhou Wang, Wei Huang et al. (+13 more)
We present LongLive-2.0, an NVFP4-based parallel infrastructure throughout the full training and inference workflow of long
video generation, addressing speed and
memory bottlenecks. For training, we introduce sequence-parallel autoregressive (AR) training, instantiated as Balanced SP, which co-designs the efficient teacher-forcing layout with SP execution by pairing clean-history and noisy-target temporal chunks on each rank, enabling a natural teacher-forcing mask with SP-aware chunked VAE encoding. Combined with NVFP4 precision, it reduces GPU
memory cost and accelerates GEMM computation during training, the proportion of which increases as
video length grows. Moreover, we show that a high-quality infrastructure and
dataset enable a remarkably clean training pipeline. Unlike existing Self-Forcing series methods that rely on ODE initialization and subsequent distribution matching
distillation (DMD), LongLive-2.0 directly tunes a
diffusion model into a long, multi-shot, interactive auto-regressive (AR)
diffusion model. It can be further converted to real-time generation (4 to 2 denoising steps) with standalone LoRA weights. For inference on Blackwell GPUs, we enable W4A4 NVFP4 inference, quantize KV cache into NVFP4 for
memory savings, and boost end-to-end throughput with asynchronous streaming VAE decoding. On non-Blackwell GPU architectures, we deploy SP inference to match the speed on Blackwell GPUs, while the quantized KV cache can lower inter-GPU communication of SP. Experiments show up to 2.15x speedup in training, and 1.84x in inference. LongLive-2.0-5B achieves 45.7 FPS inference while attaining strong performance on
benchmarks. To our knowledge, LongLive-2.0 is the first NVFP4 training and inference system for long
video generation.
Read the full abstract ▾
All authors Yukang Chen, Luozhou Wang, Wei Huang, Shuai Yang, Bohan Zhang, Yicheng Xiao, Ruihang Chu, Weian Mao, Qixin Hu, Shaoteng Liu, Yuyang Zhao, Huizi Mao, Ying-Cong Chen, Enze Xie, Xiaojuan Qi, Song Han
Computer Vision
Spectral Progressive Diffusion for Efficient Image and Video Generation
Howard Xiao, Brian Chao, Lior Yariv et al. (+1 more)
Diffusion models have been shown to implicitly generate visual content autoregressively in the frequency domain, where low-frequency components are generated earlier in the denoising process while high-frequency details emerge only in later timesteps. This structure offers a natural opportunity for efficient generation, as high-resolution computation on noise-dominated frequencies is largely redundant. We propose Spectral Progressive
Diffusion, a general framework that progressively grows resolution along the denoising trajectory of pretrained
diffusion models. To this end, we develop a spectral noise expansion mechanism and derive an optimal resolution schedule from the model's power spectrum. Our framework supports training-free acceleration and a novel fine-tuning recipe that further improves efficiency and quality. We demonstrate significant speedups on state-of-the-art pretrained image and
video generation models while preserving visual quality.
Read the full abstract ▾
All authors Howard Xiao, Brian Chao, Lior Yariv, Gordon Wetzstein
Computer Vision
PIXLRelight: Controllable Relighting via Intrinsic Conditioning
Miguel Farinha and Ronald Clark
We present PIXLRelight, a feed-forward approach for physically controllable single-image relighting. Existing methods either provide limited lighting control (e.g. through text or environment maps), accumulate errors when chaining inverse and forward rendering, or require costly per-image optimization. Our key idea is to bridge physically based rendering (PBR) and learned image synthesis through a shared intrinsic conditioning that can be obtained from either real photographs or PBR renders. At training time, paired multi-illumination photographs are decomposed into albedo, diffuse shading, and non-diffuse residuals, which condition the model. At inference time, the same conditioning is computed from a path-traced render of a coarse 3D reconstruction of the input under user-specified PBR lights. A
transformer-based neural renderer then applies the target illumination to the source photograph, preserving fine image detail through a per-pixel affine modulation. PIXLRelight enables arbitrary PBR-style lighting control, achieves state-of-the-art relighting quality, and runs in under a tenth of a second per image. Code and models are available at https://mlfarinha.github.io/pixl-relight/.
Read the full abstract ▾
Computer Vision
Ruiping Liu, Junwei Zheng, Yufan Chen et al. (+7 more)
Egocentric memory is widely used in
embodied intelligence, but it may be insufficient for comprehensive spatial-temporal
reasoning. Inspired by human recall from both field and observer perspectives, we introduce EgoExoMem, the first
benchmark for cross-view
memory reasoning over synchronized
egocentric and exocentric
videos. EgoExoMem contains $2.6K$ high-quality MCQs across eight temporal, spatial, and cross-view QA types. To support dual-view
retrieval, we propose E$^2$-Select, a training-free frame selection method for synchronized ego-exo
videos. It combines relevance-based budget allocation with per-view k-DPP sampling to handle view asymmetry and cross-view temporal consistency. Experiments show that ego and exo views provide complementary
memory cues, while existing MLLMs remain far from solving the
benchmark: the best model reaches only $55.3\%$. E$^2$-Select achieves state-of-the-art performance of $58.2\%$ over frame-selection and RAG-based
memory baselines. Further analysis reveals systematic view-preference conflicts between question framing and answer grounding, underscoring the novelty and challenge of cross-view
memory reasoning.
Read the full abstract ▾
All authors Ruiping Liu, Junwei Zheng, Yufan Chen, Di Wen, Shaofang Quan, Chengzhi Wu, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen