Computer Vision
ArtContext: Contextualizing Artworks with Open-Access Art History Articles and Wikidata Knowledge through a LoRA-Tuned CLIP Model
Samuel Waugh and Stuart James
Many Art History articles discuss artworks in general as well as specific parts of works, such as layout, iconography, or material culture. However, when viewing an artwork, it is not trivial to identify what different articles have said about the piece. Therefore, we propose ArtContext, a pipeline for taking a corpus of Open-Access Art History articles and Wikidata Knowledge and annotating Artworks with this information. We do this using a novel corpus collection pipeline, then learn a bespoke CLIP model adapted using Low-Rank Adaptation (LoRA) to make it domain-specific. We show that the new model, PaintingCLIP, which is weakly supervised by the collected corpus, outperforms CLIP and provides context for a given artwork. The proposed pipeline is generalisable and can be readily applied to numerous humanities areas.
Read the full abstract ▾
Computer Vision
Exploring Real-Time Super-Resolution: Benchmarking and Fine-Tuning for Streaming Content
Evgeney Bogatyrev, Khaled Abud, Ivan Molodetskikh et al. (+2 more)
Recent advancements in real-time super-resolution have enabled higher-quality
video streaming, yet existing methods struggle with the unique challenges of compressed
video content. Commonly used
datasets do not accurately reflect the characteristics of streaming media, limiting the relevance of current
benchmarks. To address this gap, we introduce a comprehensive
dataset - StreamSR - sourced from YouTube, covering a wide range of
video genres and resolutions representative of real-world streaming scenarios. We
benchmark 11 state-of-the-art real-time super-resolution models to evaluate their performance for the streaming use-case. Furthermore, we propose EfRLFN, an efficient real-time model that integrates Efficient Channel
Attention and a
hyperbolic tangent activation function - a novel design choice in the context of real-time super-resolution. We extensively optimized the architecture to maximize efficiency and designed a composite loss function that improves training convergence. EfRLFN combines the strengths of existing architectures while improving both visual quality and runtime performance. Finally, we show that fine-tuning other models on our
dataset results in significant performance gains that generalize well across various standard
benchmarks. We made the
dataset, the code, and the
benchmark available at https://github.com/EvgeneyBogatyrev/EfRLFN.
Read the full abstract ▾
All authors Evgeney Bogatyrev, Khaled Abud, Ivan Molodetskikh, Nikita Alutis, Dmitriy Vatolin
Computer Vision · Cross-listed from cs.RO
MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation
Yejin Kim, Wilbert Pumacay, Omar Rayyan et al. (+23 more)
Deploying
robots at scale demands robustness to the long tail of everyday situations. The countless variations in scene layout, object geometry, and task specifications that characterize real environments are vast and underrepresented in existing
robot benchmarks. Measuring this level of generalization requires infrastructure at a scale and diversity that physical evaluation alone cannot provide. We introduce MolmoSpaces, a fully open ecosystem to support large-scale benchmarking of
robot policies. MolmoSpaces consists of over 230k diverse indoor environments, ranging from handcrafted household scenes to procedurally generated multiroom houses, populated with 130k richly annotated object assets, including 48k manipulable objects with 42M stable grasps. Crucially, these environments are simulator-agnostic, supporting popular options such as MuJoCo, Isaac, and ManiSkill. The ecosystem supports the full spectrum of
embodied tasks: static and mobile manipulation,
navigation, and multiroom long-horizon tasks requiring coordinated perception, planning, and interaction across entire indoor environments. We also design MolmoSpaces-Bench, a
benchmark suite of 8 tasks in which
robots interact with our diverse scenes and richly annotated objects. Our experiments show MolmoSpaces-Bench exhibits strong sim-to-real correlation (R = 0.96, h̊o} = 0.98), confirm newer and stronger zero-shot policies outperform earlier versions in our
benchmarks, and identify key sensitivities to
prompt phrasing, initial joint positions, and camera occlusion. Through MolmoSpaces and its open-source assets and tooling, we provide a foundation for scalable data generation, policy training, and
benchmark creation for
robot learning research.
Read the full abstract ▾
All authors Yejin Kim, Wilbert Pumacay, Omar Rayyan, Max Argus, Winson Han, Eli VanderBilt, Jordi Salvador, Abhay Deshpande, Rose Hendrix, Snehal Jauhri, Shuo Liu, Nur Muhammad Mahi Shafiullah, Maya Guru, Ainaz Eftekhar, Karen Farley, Donovan Clay, Jiafei Duan, Arjun Guru, Piper Wolters, Alvaro Herrasti, Ying-Chun Lee, Georgia Chalvatzaki, Yuchen Cui, Ali Farhadi, Dieter Fox, Ranjay Krishna
Computer Vision
MDE-VIO: Enhancing Visual-Inertial Odometry Using Learned Depth Priors
Arda Alniak, Sinan Kalkan, Mustafa Mert Ankarali et al. (+2 more)
Traditional
monocular Visual-Inertial Odometry (VIO) systems struggle in low-texture environments where sparse visual features are insufficient for accurate pose estimation. To address this, dense
Monocular Depth Estimation (MDE) has been widely explored as a complementary information source. While recent
Vision Transformer (ViT) based complex foundational models offer dense, geometrically consistent depth, their computational demands typically preclude them from real-time edge deployment. Our work bridges this gap by integrating learned depth priors directly into the VINS-Mono optimization backend. We propose a novel framework that enforces affine-invariant depth consistency and pairwise ordinal constraints, explicitly filtering unstable artifacts via variance-based gating. This approach strictly adheres to the computational limits of edge devices while robustly recovering metric scale. Extensive experiments on the TartanGround and M3ED
datasets demonstrate that our method prevents divergence in challenging scenarios and delivers significant accuracy gains, reducing Absolute Trajectory Error (ATE) by up to 28.3%. Code will be made available.
Read the full abstract ▾
All authors Arda Alniak, Sinan Kalkan, Mustafa Mert Ankarali, Afsar Saranli, Abdullah Aydin Alatan
Computer Vision
Selective Prior Synchronization via SYNC Loss
Ishan Mishra, Jiajie Li, Deepak Mishra et al. (+1 more)
Prediction under uncertainty is a critical requirement for the deep neural network to succeed responsibly. This paper focuses on selective prediction, which allows DNNs to make informed decisions about when to predict or abstain based on the uncertainty level of their predictions. Current methods are either ad-hoc such as SelectiveNet, focusing on how to modify the network architecture or objective function, or post-hoc such as softmax response, achieving selective prediction through analyzing the model's probabilistic outputs. We observe that post-hoc methods implicitly generate uncertainty information, termed the selective prior, which has traditionally been used only during inference. We argue that the selective prior provided by the selection mechanism is equally vital during the training stage. Therefore, we propose the SYNC loss which introduces a novel integration of ad-hoc and post-hoc method. Specifically, our approach incorporates the softmax response into the training process of SelectiveNet, enhancing its selective prediction capabilities by examining the selective prior. Evaluated across various
datasets, including CIFAR-100, ImageNet-100, and Stanford Cars, our method not only enhances the model's generalization capabilities but also surpasses previous works in selective prediction performance, and sets new
benchmarks for state-of-the-art performance.
Read the full abstract ▾
All authors Ishan Mishra, Jiajie Li, Deepak Mishra, Jinjun Xiong
Computer Vision
Advancing Digital Twin Generation Through a Novel Simulation Framework and Quantitative Benchmarking
Jacob Rubinstein, Avi Donaty and Don Engel
The generation of 3D models from real-world objects has often been accomplished through photogrammetry, i.e., by taking 2D photos from a variety of perspectives and then triangulating matched point-based features to create a textured mesh. Many design choices exist within this framework for the generation of digital twins, and differences between such approaches are largely judged qualitatively. Here, we present and test a novel pipeline for generating synthetic images from high-quality 3D models and programmatically generated camera poses. This enables a wide variety of repeatable, quantifiable experiments which can compare ground-truth knowledge of virtual camera parameters and of virtual objects against the reconstructed estimations of those perspectives and subjects.
Read the full abstract ▾
Computer Vision
Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
Sethuraman T V, Savya Khosla, Aditi Tiwari et al. (+12 more)
Video-
Language Models (VidLMs) achieve strong
benchmark scores, yet these scores often hide whether models use the
video at all. We show that VidLM failures follow two pathways: some visual signals are never reliably encoded, while others are encoded but overridden by model priors. We introduce REVEAL, a diagnostic stress-test
benchmark for quantifying when and why VidLMs under-use visual evidence. REVEAL contains five controlled probes: camera-motion sensitivity, cross-frame integration,
video sycophancy,
language-only shortcuts, and temporal expectation bias. Together, they test whether models encode basic
video signals, combine evidence across frames, and preserve visual evidence against user assertions,
language cues, and learned event expectations. Across 12 VidLMs we find systematic failures along both pathways, with most models falling below chance on the binary and six-way probes that humans solve at 78–100% accuracy. Under assertive
prompts, a model's output distribution becomes nearly invariant to whether it is shown a real
video or random noise, making visual evidence effectively causally inert. We further carry out mechanistic probes to identify where these failures arise in the model pipeline and why visual evidence is lost. REVEAL provides a scalable, human-verified framework for moving beyond aggregate scores toward structured, reproducible evaluation of
multimodal reliability.
Read the full abstract ▾
All authors Sethuraman T V, Savya Khosla, Aditi Tiwari, Vidya Ganesh, Rakshana Jayaprakash, Aditya Jain, Vignesh Srinivasakumar, Onkar Kishor Susladkar, Srinidhi Sunkara, Aditya Shanmugham, Rakesh Vaideeswaran, Abbaas Alif Mohamed Nishar, Simon Jenni, Rohan Maheshwari, Derek Hoiem
Computer Vision
ReTracing: An Archaeological Approach Through Body, Machine, and Generative Systems
Yitong Wang and Yue Yao
We present ReTracing, a multi-
agent embodied performance art that adopts an archaeological approach to examine how artificial intelligence shapes, constrains, and produces bodily movement. Drawing from science-fiction novels, the project extracts sentences that describe human-machine interaction. We use large
language models (LLMs) to generate paired
prompts "what to do" and "what not to do" for each excerpt. A
diffusion-based text-to-
video model transforms these
prompts into choreographic guides for a human performer and motor commands for a quadruped
robot. Both
agents enact the actions on a mirrored floor, captured by multi-camera motion tracking and reconstructed into 3D point clouds and motion trails, forming a digital archive of motion traces. Through this process, ReTracing serves as a novel approach to reveal how generative systems encode socio-cultural biases through choreographed movements. Through an immersive interplay of AI, human, and
robot, ReTracing confronts a critical question of our time: What does it mean to be human among AIs that also move, think, and leave traces behind?
Read the full abstract ▾
Computer Vision
Active Zero: Self-Evolving Vision-Language Models through Active Environment Exploration
Jinghan He, Junfeng Fang, Feng Xiong et al. (+5 more)
Self-play has enabled large
language models to autonomously improve through self-generated challenges. However, existing self-play methods for
vision-
language models rely on passive interaction with static image collections, resulting in strong dependence on initial
datasets and inefficient learning. Without the ability to actively seek visual data tailored to their evolving capabilities,
agents waste computational effort on samples that are either trivial or beyond their current skill level. To address these limitations, we propose Active-Zero, a framework that shifts from passive interaction to active exploration of visual environments. Active-Zero employs three co-evolving
agents: a Searcher that retrieves images from open-world repositories based on the model's capability frontier, a Questioner that synthesizes calibrated
reasoning tasks, and a Solver refined through accuracy rewards. This closed loop enables self-scaffolding auto-curricula where the model autonomously constructs its learning trajectory. On Qwen2.5-VL-7B-Instruct across 12
benchmarks, Active-Zero achieves 53.97 average accuracy on
reasoning tasks (5.7% improvement) and 59.77 on general understanding (3.9% improvement), consistently outperforming existing self-play baselines. These results highlight active exploration as a key ingredient for scalable and adaptive self-evolving
vision-
language systems.
Read the full abstract ▾
All authors Jinghan He, Junfeng Fang, Feng Xiong, Zijun Yao, Fei Shen, Haiyun Guo, Jinqiao Wang, Tat-Seng Chua
Computer Vision
Toward Reliable Tea Leaf Disease Diagnosis Using Deep Learning Model: Enhancing Robustness With Explainable AI and Adversarial Training
Samanta Ghosh, Jannatul Adan Mahi, Shayan Abrar et al. (+4 more)
Tea is a valuable asset for the economy of Bangladesh. So, tea cultivation plays an important role to boost the economy. These valuable plants are vulnerable to various kinds of leaf infections which may cause less production and low quality. It is not so easy to detect these diseases manually. It may take time and there could be some errors in the detection.Therefore, the purpose of the study is to develop an automated deep learning model for tea leaf disease classification based on the teaLeafBD
dataset so that anyone can detect the diseases more easily and efficiently. There are 5,278 high-resolution images in this
dataset. The images are classified into seven categories. Six of them represents various diseases and the rest one represents healthy leaves. The proposed pipeline contains data preprocessing, data splitting,
adversarial training, augmentation, model training, evaluation, and comprehension made possible with Explainable AI strategies. DenseNet201 and EfficientNetB3 were employed to perform the classification task. To prepare the model more robustly, we applied
adversarial training so it can operate effectively even with noisy or disturbed inputs. In addition, Grad-CAM visualization was executed to analyze the model's predictions by identifying the most influential regions of each image. Our experimental outcomes revealed that EfficientNetB3 achieved the highest classification accuracy of 93%, while DenseNet201 reached 91%. The outcomes prove that the effectiveness of the proposed approach can accurately detect tea leaf diseases and provide a practical solution for advanced agricultural management.
Read the full abstract ▾
All authors Samanta Ghosh, Jannatul Adan Mahi, Shayan Abrar, Md Parvez Mia, Asaduzzaman Rayhan, Abdul Awal Yasir, Asaduzzaman Hridoy
Computer Vision
ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learning
Yandan Yang, Shuang Zeng, Tong Lin et al. (+11 more)
Building general-purpose
embodied agents across diverse hardware remains a central challenge in robotics, often framed as the ”one-
brain, many-forms” paradigm. Progress is hindered by fragmented data, inconsistent representations, and misaligned training objectives. We present ABot-M0, a framework that builds a systematic data curation pipeline while jointly optimizing model architecture and training strategies, enabling end-to-end transformation of heterogeneous raw data into unified, efficient representations. From six public
datasets, we clean, standardize, and balance samples to construct UniACT-
dataset, a large-scale
dataset with over 6 million trajectories and 9,500 hours of data, covering diverse
robot morphologies and task scenarios. Unified pre-training improves knowledge transfer and generalization across platforms and tasks, supporting general-purpose
embodied intelligence. To improve action prediction efficiency and stability, we propose the Action
Manifold Hypothesis: effective
robot actions lie not in the full high-dimensional space but on a low-dimensional, smooth
manifold governed by physical laws and task constraints. Based on this, we introduce Action
Manifold Learning (AML), which uses a DiT backbone to predict clean, continuous action sequences directly. This shifts learning from denoising to projection onto feasible
manifolds, improving decoding speed and policy stability. ABot-M0 supports modular perception via a dual-stream mechanism that integrates VLM semantics with
geometric priors and multi-view inputs from plug-and-play 3D modules such as VGGT and Qwen-Image-Edit, enhancing spatial understanding without modifying the backbone and mitigating standard VLM limitations in 3D
reasoning. Experiments show components operate independently with additive benefits. We will release all code and pipelines for reproducibility and future research.
Read the full abstract ▾
All authors Yandan Yang, Shuang Zeng, Tong Lin, Xinyuan Chang, Dekang Qi, Junjin Xiao, Haoyun Liu, Ronghan Chen, Yuzhi Chen, Dongjie Huo, Feng Xiong, Xing Wei, Zhiheng Ma, Mu Xu
Computer Vision
DD-MDN: Human Trajectory Forecasting with Diffusion-Based Dual Mixture Density Networks and Uncertainty Self-Calibration
Manuel Hetzel, Kerim Turacan, Hannes Reichert et al. (+2 more)
Human Trajectory
Forecasting (HTF) predicts future human movements from past trajectories and environmental context, with applications in Autonomous Driving, Smart Surveillance, and Human-
Robot Interaction. While prior work has focused on accuracy, social interaction modeling, and diversity, little
attention has been paid to uncertainty modeling, calibration, and forecasts from short observation periods, which are crucial for downstream tasks such as path planning and collision avoidance. We propose DD-MDN, an end-to-end probabilistic HTF model that combines high positional accuracy, calibrated uncertainty, and robustness to short observations. Using a few-shot denoising
diffusion backbone and a dual mixture density network, our method learns self-calibrated residence areas and probability-ranked anchor paths, from which diverse trajectory hypotheses are derived, without predefined anchors or endpoints. Experiments on the ETH/UCY, SDD, inD, and IMPTC
datasets demonstrate state-of-the-art accuracy, robustness at short observation intervals, and reliable uncertainty modeling. The code is available at: https://github.com/kav-institute/ddmdn.
Read the full abstract ▾
All authors Manuel Hetzel, Kerim Turacan, Hannes Reichert, Konrad Doll, Bernhard Sick
Computer Vision
SurfPhase: 3D Interfacial Dynamics in Two-Phase Flows from Sparse Videos
Yue Gao, Hong-Xing Yu, Sanghyeon Chang et al. (+5 more)
Interfacial dynamics in two-phase flows govern momentum, heat, and mass transfer, yet remain difficult to measure experimentally. Classical techniques face intrinsic limitations near moving interfaces, while existing neural rendering methods target single-phase flows with diffuse boundaries and cannot handle sharp, deformable
liquid-vapor interfaces. We propose SurfPhase, a novel model for reconstructing 3D interfacial dynamics from sparse camera views. Our approach integrates dynamic Gaussian surfels with a signed distance function formulation for
geometric consistency, and leverages a
video diffusion model to synthesize novel-view
videos to refine reconstruction from sparse observations. We evaluate on a new
dataset of high-speed pool boiling
videos, demonstrating high-quality view synthesis and velocity estimation from only two camera views. Project website: https://yuegao.me/SurfPhase.
Read the full abstract ▾
All authors Yue Gao, Hong-Xing Yu, Sanghyeon Chang, Qianxi Fu, Bo Zhu, Yoonjin Won, Juan Carlos Niebles, Jiajun Wu
Computer Vision
Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modeling
Gongye Liu, Bo Yang, Yida Zhi et al. (+8 more)
Preference optimization for
diffusion and flow-matching models relies on reward functions that are both discriminatively robust and computationally efficient.
Vision-
Language Models (VLMs) have emerged as the primary reward provider, leveraging their rich
multimodal priors to guide
alignment. However, their computation and
memory cost can be substantial, and optimizing a latent
diffusion generator through a pixel-space reward introduces a domain mismatch that complicates
alignment. In this paper, we propose DiNa-LRM, a
diffusion-native latent reward model that formulates preference learning directly on noisy
diffusion states. Our method introduces a noise-calibrated Thurstone likelihood with
diffusion-noise-dependent uncertainty. DiNa-LRM leverages a pretrained latent
diffusion backbone with a timestep-conditioned reward head, and supports inference-time noise ensembling, providing a
diffusion-native mechanism for test-time scaling and robust rewarding. Across image
alignment benchmarks, DiNa-LRM substantially outperforms existing
diffusion-based reward baselines and achieves performance competitive with state-of-the-art VLMs at a fraction of the computational cost. In preference optimization, we demonstrate that DiNa-LRM improves preference optimization dynamics, enabling faster and more resource-efficient model
alignment.
Read the full abstract ▾
All authors Gongye Liu, Bo Yang, Yida Zhi, Zhizhou Zhong, Lei Ke, Didan Deng, Han Gao, Yongxiang Huang, Kaihao Zhang, Hongbo Fu, Wenhan Luo
Computer Vision · Cross-listed from cs.LG
GENIUS: Generative Fluid Intelligence Evaluation Suite
Ruichuan An, Sihan Yang, Ziyu Guo et al. (+8 more)
Unified
Multimodal Models (UMMs) have shown remarkable progress in visual generation. Yet, existing
benchmarks predominantly assess $\textit{Crystallized Intelligence}$, which relies on recalling accumulated knowledge and learned schemas. This focus overlooks $\textit{Generative Fluid Intelligence (GFI)}$: the capacity to induce patterns, reason through constraints, and adapt to novel scenarios on the fly. To rigorously assess this capability, we introduce $\textbf{GENIUS}$ ($\textbf{GEN}$ Fluid $\textbf{I}$ntelligence Eval$\textbf{U}$ation $\textbf{S}$uite). We formalize $\textit{GFI}$ as a synthesis of three primitives. These include $\textit{Inducing Implicit Patterns}$ (e.g., inferring personalized visual preferences), $\textit{Executing Ad-hoc Constraints}$ (e.g., visualizing abstract metaphors), and $\textit{Adapting to Contextual Knowledge}$ (e.g., simulating counter-intuitive physics). Collectively, these primitives challenge models to solve problems grounded entirely in the immediate context. Our systematic evaluation of 12 representative models reveals significant performance deficits in these tasks. Crucially, our diagnostic analysis disentangles these failure modes. It demonstrates that deficits stem from limited context comprehension rather than insufficient intrinsic generative capability. To bridge this gap, we propose a training-free
attention intervention strategy. Ultimately, $\textbf{GENIUS}$ establishes a rigorous standard for $\textit{GFI}$, guiding the field beyond knowledge utilization toward dynamic, general-purpose
reasoning. Our
dataset and code will be released at: $\href{https://github.com/arctanxarc/GENIUS}{https://github.com/arctanxarc/GENIUS}$.
Read the full abstract ▾
All authors Ruichuan An, Sihan Yang, Ziyu Guo, Wei Dai, Zijun Shen, Haodong Li, Renrui Zhang, Xinyu Wei, Guopeng Li, Wenshan Wu, Wentao Zhang
Computer Vision · Cross-listed from cs.LG
Meltdown: Circuits and Bifurcations in Point-Cloud-Conditioned 3D Diffusion Transformers
Maximilian Plattner, Fabian Paischer, Johannes Brandstetter et al. (+1 more)
Sparse point clouds are a common input modality for 3D surface reconstruction, including in safety-critical settings such as
surgical navigation and autonomous perception. Recent point-cloud-conditioned 3D
diffusion transformers achieve state-of-the-art results in this regime by leveraging learned priors. We show that these models can fail catastrophically under realistic input variation, and present a mechanistic case study of why. We identify a failure mode we call Meltdown: tiny on-surface perturbations to a sparse input point cloud can fracture the reconstructed output into hundreds of disconnected pieces.
Adversarial search recovers Meltdown in 89.9-100% of shapes across the two open-weight state-of-the-art architectures we study (WaLa, Make-a-Shape) on real-world
datasets (GSO, SimJEB) and under both DDPM and DDIM sampling. We trace Meltdown along the forward pass: it is governed by how uniformly the points are distributed on the surface, faithfully transduced through the point-cloud encoder, and committed by a single early-denoising cross-
attention write in the
diffusion backbone.
Diffusion-trajectory ensembles exhibit
symmetry-breaking near this commit step, consistent with a bifurcation of the reverse process. Through a suite of matched-magnitude controls, we show that the variable on which the model commits is directional, concentrated in a low-rank subspace of the write's perturbation drift. Motivated by this finding, we introduce PowerRemap, a test-time control that reshapes the singular spectrum of the localized write to suppress this drift, with rescue rates of 98.3% on WaLa and 84.6% on Make-a-Shape. Together, these results link a
circuit-level cross-
attention mechanism to a trajectory-level account of the failure, demonstrating how mechanistic analysis can explain and guide behavior in conditional
diffusion transformers.
Read the full abstract ▾
All authors Maximilian Plattner, Fabian Paischer, Johannes Brandstetter, Arturs Berzins
Computer Vision
PhyCritic: Multimodal Critic Models for Physical AI
Tianyi Xiong, Shihao Wang, Guilin Liu et al. (+5 more)
With the rapid development of large
multimodal models, reliable judge and critic models have become essential for open-ended evaluation and preference
alignment, providing pairwise preferences, numerical scores, and explanatory justifications for assessing model-generated responses. However, existing critics are primarily trained in general visual domains such as captioning or image question answering, leaving physical AI tasks involving perception, causal
reasoning, and planning largely underexplored. We introduce PhyCritic, a
multimodal critic model optimized for physical AI through a two-stage RLVR pipeline: a physical skill warmup stage that enhances physically oriented perception and
reasoning, followed by self-referential critic finetuning, where the critic generates its own prediction as an internal reference before judging candidate responses, improving judgment stability and physical correctness. Across both physical and general-purpose
multimodal judge
benchmarks, PhyCritic achieves strong performance gains over open-source baselines and, when applied as a policy model, further improves perception and
reasoning in physically grounded tasks.
Read the full abstract ▾
All authors Tianyi Xiong, Shihao Wang, Guilin Liu, Yi Dong, Ming Li, Heng Huang, Jan Kautz, Zhiding Yu
Computer Vision
HairWeaver: Few-Shot Photorealistic Hair Motion Synthesis with Sim-to-Real Guided Video Diffusion
Di Chang, Ji Hou, Aljaz Bozic et al. (+8 more)
We present HairWeaver, a
diffusion-based pipeline that animates a single human image with realistic and expressive hair dynamics. While existing methods successfully control body pose, they lack specific control over hair, and as a result, fail to capture the intricate hair motions, resulting in stiff and unrealistic animations. HairWeaver overcomes this limitation using two specialized modules: a Motion-Context-LoRA to integrate motion conditions and a Style-
Alignment-LoRA to preserve the subject's photoreal appearance across different data domains. These lightweight components are designed to guide a
video diffusion backbone while maintaining its core generative capabilities. By training on a specialized
dataset of dynamic human motion generated from a CG simulator, HairWeaver affords fine control over hair motion and ultimately learns to produce highly realistic hair that responds naturally to movement. Comprehensive evaluations demonstrate that our approach sets a new state of the art, producing lifelike human hair animations with dynamic details.
Read the full abstract ▾
All authors Di Chang, Ji Hou, Aljaz Bozic, Assaf Neuberger, Felix Juefei-Xu, Olivier Maury, Gene Wei-Chin Lin, Tuur Stuyck, Doug Roble, Mohammad Soleymani, Stephane Grabli
Computer Vision
FastFlow: Accelerating The Generative Flow Matching Models with Bandit Inference
Divya Jyoti Bajpai, Dhruv Bhardwaj, Soumya Roy et al. (+4 more)
Flow-matching models deliver state-of-the-art fidelity in image and
video generation, but the inherent sequential denoising process renders them slower. Existing acceleration methods like
distillation, trajectory truncation, and consistency approaches are static, require retraining, and often fail to generalize across tasks. We propose FastFlow, a plug-and-play adaptive inference framework that accelerates generation in flow matching models. FastFlow identifies denoising steps that produce only minor adjustments to the denoising path and approximates them without using the full neural network models used for velocity predictions. The approximation utilizes finite-difference velocity estimates from prior predictions to efficiently extrapolate future states, enabling faster advancements along the denoising path at zero compute cost. This enables skipping computation at intermediary steps. We model the decision of how many steps to safely skip before requiring a full model computation as a multi-armed bandit problem. The bandit learns the optimal skips to balance speed with performance. FastFlow integrates seamlessly with existing pipelines and generalizes across image generation,
video generation, and editing tasks. Experiments demonstrate a speedup of over 2.6x while maintaining high-quality outputs. The source code for this work can be found at https://github.com/Div290/FastFlow.
Read the full abstract ▾
All authors Divya Jyoti Bajpai, Dhruv Bhardwaj, Soumya Roy, Tejas Duseja, Harsh Agarwal, Aashay Sandansing, Manjesh Kumar Hanawal
Computer Vision
First International StepUP Competition for Biometric Footstep Recognition: Methods, Results and Remaining Challenges
Robyn Larracy, Eve MacDonald, Angkoon Phinyomark et al. (+5 more)
Biometric footstep recognition, based on a person's unique pressure patterns under their feet during walking, is an emerging field with growing applications in security and safety. However, progress in this area has been limited by the lack of large, diverse
datasets necessary to address critical challenges such as generalization to new users and robustness to shifts in factors like footwear or walking speed. The recent release of the UNB StepUP-P150
dataset, the largest and most comprehensive collection of high-resolution footstep pressure recordings to date, opens new opportunities for addressing these challenges through deep learning. To mark this milestone, the First International StepUP Competition for Biometric Footstep Recognition was launched. Competitors were tasked with developing robust recognition models using the StepUP-P150
dataset that were then evaluated on a separate, dedicated test set designed to assess verification performance under challenging variations, given limited and relatively homogeneous reference data. The competition attracted global participation, with 23 registered teams from academia and industry. The top-performing team, Saeid_UCC, achieved the best equal error rate (EER) of 10.77% using a generative reward machine (GRM) optimization strategy. Overall, the competition showcased strong solutions, but persistent challenges in generalizing to unfamiliar footwear highlight a critical area for future work.
Read the full abstract ▾
All authors Robyn Larracy, Eve MacDonald, Angkoon Phinyomark, Saeid Rezaei, Mahdi Laghaei, Ali Hajighasem, Aaron Tabor, Erik Scheme
Computer Vision
Chatting with Images for Introspective Visual Thinking
Junfei Wu, Jian Guan, Qiang Liu et al. (+4 more)
Current large
vision-
language models (LVLMs) typically rely on text-only
reasoning based on a single-pass visual encoding, which often leads to loss of fine-grained visual information. Recently the proposal of ”thinking with images” attempts to alleviate this limitation by manipulating images via external tools or code; however, the resulting visual states are often insufficiently grounded in linguistic semantics, impairing effective cross-modal
alignment - particularly when visual semantics or
geometric relationships must be reasoned over across distant regions or multiple images. To address these challenges, we propose ”chatting with images”, a new framework that reframes visual manipulation as
language-guided feature modulation. Under the guidance of expressive
language prompts, the model dynamically performs joint re-encoding over multiple image regions, enabling tighter coupling between linguistic
reasoning and visual state updates. We instantiate this paradigm in ViLaVT, a novel LVLM equipped with a dynamic
vision encoder explicitly designed for such interactive visual
reasoning, and trained it with a two-stage curriculum combining supervised fine-tuning and reinforcement learning to promote effective
reasoning behaviors. Extensive experiments across eight
benchmarks demonstrate that ViLaVT achieves strong and consistent improvements, with particularly pronounced gains on complex multi-image and
video-based spatial
reasoning tasks.
Read the full abstract ▾
All authors Junfei Wu, Jian Guan, Qiang Liu, Shu Wu, Liang Wang, Wei Wu, Tieniu Tan
Computer Vision
PuriLight: A Lightweight Shuffle and Purification Framework for Monocular Depth Estimation
Yujie Chen, Li Zhang, Xiaomeng Chu et al. (+1 more)
We propose PuriLight, a lightweight and efficient framework for self-supervised
monocular depth estimation, to address the dual challenges of computational efficiency and detail preservation. While recent advances in self-supervised depth estimation have reduced reliance on ground truth supervision, existing approaches remain constrained by either bulky architectures compromising practicality or lightweight models sacrificing structural precision. These dual limitations underscore the critical need to develop lightweight yet structurally precise architectures. Our framework addresses these limitations through a three-stage architecture incorporating three novel modules: the Shuffle-Dilation Convolution (SDC) module for local feature extraction, the Rotation-Adaptive Kernel
Attention (RAKA) module for hierarchical feature enhancement, and the Deep Frequency Signal Purification (DFSP) module for global feature purification. Through effective collaboration, these modules enable PuriLight to achieve both lightweight and accurate feature extraction and processing. Extensive experiments demonstrate that PuriLight achieves state-of-the-art performance with minimal training parameters while maintaining exceptional computational efficiency. Codes will be available at https://github.com/ishrouder/PuriLight.
Read the full abstract ▾
All authors Yujie Chen, Li Zhang, Xiaomeng Chu, Tian Zhang
Computer Vision
Chain-of-Look Spatial Reasoning for Dense Surgical Instrument Counting
Rishikesh Bhyri, Brian R Quaranto, Philip J Seger et al. (+7 more)
Accurate counting of
surgical instruments in Operating Rooms (OR) is a critical prerequisite for ensuring patient safety during surgery. Despite recent progress of large visual-
language models and agentic AI, accurately counting such instruments remains highly challenging, particularly in dense scenarios where instruments are tightly clustered. To address this problem, we introduce Chain-of-Look, a novel visual
reasoning framework that mimics the sequential human counting process by enforcing a structured visual chain, rather than relying on classic object detection which is unordered. This visual chain guides the model to count along a coherent spatial trajectory, improving accuracy in complex scenes. To further enforce the physical plausibility of the visual chain, we introduce the neighboring loss function, which explicitly models the spatial constraints inherent to densely packed
surgical instruments. We also present SurgCount-HD, a new
dataset comprising 1,464 high-density
surgical instrument images. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches for counting (e.g., CountGD, REC) as well as Multimodality Large
Language Models (e.g., Qwen, ChatGPT) in the challenging task of dense
surgical instrument counting.
Read the full abstract ▾
All authors Rishikesh Bhyri, Brian R Quaranto, Philip J Seger, Kaity Tung, Brendan Fox, Gene Yang, Steven D. Schwaitzberg, Junsong Yuan, Nan Xi, Peter C W Kim
Computer Vision · Cross-listed from cs.RO
ContactGaussian-WM: Learning Physics-Grounded World Model from Videos
Meizhong Wang, Wanxin Jin, Kun Cao et al. (+2 more)
Developing world models that understand complex physical interactions is essential for advancing robotic planning and simulation.However, existing methods often struggle to accurately model the environment under conditions of data scarcity and complex contact-rich dynamic motion.To address these challenges, we propose ContactGaussian-WM, a differentiable physics-grounded rigid-body world model capable of learning intricate physical laws directly from sparse and contact-rich
video sequences.Our framework consists of two core components: (1) a unified Gaussian representation for both visual appearance and collision geometry, and (2) an end-to-end differentiable learning framework that differentiates through a closed-form physics engine to infer physical properties from sparse visual observations.Extensive simulations and real-world evaluations demonstrate that ContactGaussian-WM outperforms state-of-the-art methods in learning complex scenarios, exhibiting robust generalization capabilities.Furthermore, we showcase the practical utility of our framework in downstream applications, including data synthesis and real-time MPC.
Read the full abstract ▾
All authors Meizhong Wang, Wanxin Jin, Kun Cao, Lihua Xie, Yiguang Hong
Computer Vision
LaSSM: Efficient Semantic-Spatial Query Decoding via Local Aggregation and State Space Models for 3D Instance Segmentation
Lei Yao, Yi Wang, Yawen Cui et al. (+2 more)
Query-based 3D scene instance
segmentation from point clouds has attained notable performance. However, existing methods suffer from the query initialization dilemma due to the sparse nature of point clouds and rely on computationally intensive
attention mechanisms in query decoders. We accordingly introduce LaSSM, prioritizing simplicity and efficiency while maintaining competitive performance. Specifically, we propose a hierarchical semantic-spatial query initializer to derive the query set from superpoints by considering both semantic cues and spatial distribution, achieving comprehensive scene coverage and accelerated convergence. We further present a coordinate-guided state space model (SSM) decoder that progressively refines queries. The novel decoder features a local aggregation scheme that restricts the model to focus on geometrically coherent regions and a spatial dual-path SSM block to capture underlying dependencies within the query set by integrating associated coordinates information. Our design enables efficient instance prediction, avoiding the incorporation of noisy information and reducing redundant computation. LaSSM ranks first place on the latest ScanNet++ V2 leaderboard, outperforming the previous best method by 2.5% mAP with only 1/3 FLOPs, demonstrating its superiority in challenging large-scale scene instance
segmentation. LaSSM also achieves competitive performance on ScanNet, ScanNet200, S3DIS and ScanNet++ V1
benchmarks with less computational cost. Extensive ablation studies and qualitative results validate the effectiveness of our design. The code and weights are available at https://github.com/RayYoh/LaSSM.
Read the full abstract ▾
All authors Lei Yao, Yi Wang, Yawen Cui, Moyun Liu, Lap-Pui Chau
Computer Vision
Vasileios Arampatzakis, George Pavlidis, Nikolaos Mitianoudis et al. (+1 more)
Monocular depth estimation is a central problem in computer
vision with applications in robotics, AR, and autonomous driving, yet the self-
attention mechanisms that drive modern
Transformer architectures remain opaque. We introduce SVD-Inspired
Attention (SVDA) into the Dense Prediction
Transformer (DPT), providing the first spectrally structured formulation of
attention for dense prediction tasks. SVDA decouples directional
alignment from spectral modulation by
embedding a learnable diagonal
matrix into normalized query-key interactions, enabling
attention maps that are intrinsically interpretable rather than post-hoc approximations. Experiments on KITTI and NYU-v2 show that SVDA preserves or slightly improves predictive accuracy while adding only minor computational overhead. More importantly, SVDA unlocks six spectral indicators that quantify
entropy, rank, sparsity,
alignment, selectivity, and robustness. These reveal consistent cross-
dataset and depth-wise patterns in how
attention organizes during training, insights that remain inaccessible in standard
Transformers. By shifting the role of
attention from opaque mechanism to quantifiable descriptor, SVDA redefines interpretability in
monocular depth estimation and opens a principled avenue toward transparent dense prediction models.
Read the full abstract ▾
All authors Vasileios Arampatzakis, George Pavlidis, Nikolaos Mitianoudis, Nikos Papamarkos
Computer Vision
Enhancing Predictability of Multi-Tenant DNN Inference for Autonomous Vehicles' Perception
Liangkai Liu, Kang G. Shin, Jinkyu Lee et al. (+2 more)
Autonomous vehicles (AVs) rely on sensors and deep neural networks (DNNs) to perceive their surrounding environment and make maneuver decisions in real time. However, achieving real-time DNN inference in the AV's perception pipeline is challenging due to the large gap between the computation requirement and the AV's limited resources. Most, if not all, of existing studies focus on optimizing the DNN inference time to achieve faster perception by compressing the DNN model with
pruning and quantization. In contrast, we present a Predictable Perception system with DNNs (PP-DNN) that reduce the amount of image data to be processed while maintaining the same level of accuracy for multi-tenant DNNs by dynamically selecting critical frames and regions of interest (ROIs). PP-DNN is based on our key insight that critical frames and ROIs for AVs vary with the AV's surrounding environment. However, it is challenging to identify and use critical frames and ROIs in multi-tenant DNNs for predictable inference. Given image-frame streams, PP-DNN leverages an ROI generator to identify critical frames and ROIs based on the similarities of consecutive frames and traffic scenarios. PP-DNN then leverages a FLOPs predictor to predict multiply-accumulate operations (MACs) from the dynamic critical frames and ROIs. The ROI scheduler coordinates the processing of critical frames and ROIs with multiple DNN models. Finally, we design a detection predictor for the perception of non-critical frames. We have implemented PP-DNN in an ROS-based AV pipeline and evaluated it with the BDD100K and the nuScenes
dataset. PP-DNN is observed to significantly enhance perception predictability, increasing the number of fusion frames by up to 7.3x, reducing the fusion delay by >2.6x and fusion-delay variations by >2.3x, improving detection completeness by 75.4% and the cost-effectiveness by up to 98% over the baseline.
Read the full abstract ▾
All authors Liangkai Liu, Kang G. Shin, Jinkyu Lee, Chengmo Yang, Weisong Shi
Computer Vision
Interpretable Vision Transformers in Image Classification via SVDA
Vasileios Arampatzakis, George Pavlidis, Nikolaos Mitianoudis et al. (+1 more)
Vision Transformers (ViTs) have achieved state-of-the-art performance in image classification, yet their
attention mechanisms often remain opaque and exhibit dense, non-structured behaviors. In this work, we adapt our previously proposed SVD-Inspired
Attention (SVDA) mechanism to the ViT architecture, introducing a geometrically grounded formulation that enhances interpretability, sparsity, and spectral structure. We apply the use of interpretability indicators – originally proposed with SVDA – to monitor
attention dynamics during training and assess structural properties of the learned representations. Experimental evaluations on four widely used
benchmarks – CIFAR-10, FashionMNIST, CIFAR-100, and ImageNet-100 – demonstrate that SVDA consistently yields more interpretable
attention patterns without sacrificing classification accuracy. While the current framework offers descriptive insights rather than prescriptive guidance, our results establish SVDA as a comprehensive and informative tool for analyzing and developing structured
attention models in computer
vision. This work lays the foundation for future advances in explainable AI, spectral diagnostics, and
attention-based model compression.
Read the full abstract ▾
All authors Vasileios Arampatzakis, George Pavlidis, Nikolaos Mitianoudis, Nikos Papamarkos
Computer Vision
DFIC: Towards a balanced facial image dataset for automatic ICAO compliance verification
Nuno Gonçalves, Diogo Nunes, Carla Guerra et al. (+1 more)
Ensuring compliance with ISO/IEC and ICAO standards for facial images in machine-readable travel documents (MRTDs) is essential for reliable identity verification, but current manual inspection methods are inefficient in high-demand environments. This paper introduces the DFIC
dataset, a novel comprehensive facial image
dataset comprising around 58,000 annotated images and 2706
videos of more than 1000 subjects, that cover a broad range of non-compliant conditions, in addition to compliant portraits. Our
dataset provides a more balanced demographic distribution than the existing public
datasets, with one partition that is nearly uniformly distributed, facilitating the development of automated ICAO compliance verification methods. Using DFIC, we fine-tuned a novel method that heavily relies on spatial
attention mechanisms for the automatic validation of ICAO compliance requirements, and we have compared it with the state-of-the-art aimed at ICAO compliance verification, demonstrating improved results. DFIC
dataset is now made public (https://github.com/visteam-isr-uc/DFIC) for the training and validation of new models, offering an unprecedented diversity of faces, that will improve both robustness and adaptability to the intrinsically diverse combinations of faces and props that can be presented to the validation system. These results emphasize the potential of DFIC to enhance automated ICAO compliance methods but it can also be used in many other applications that aim to improve the security,
privacy, and
fairness of facial recognition systems.
Read the full abstract ▾
All authors Nuno Gonçalves, Diogo Nunes, Carla Guerra, João Marcos
Computer Vision
VFGS-Net: Frequency-Guided State-Space Learning for Topology-Preserving Retinal Vessel Segmentation
Ruiqi Song, Lei Liu, Ya-Nan Zhang et al. (+3 more)
Accurate retinal vessel
segmentation is a critical prerequisite for quantitative analysis of retinal images and computer-aided diagnosis of vascular diseases such as diabetic retinopathy. However, the elongated morphology, wide scale variation, and low contrast of retinal vessels pose significant challenges for existing methods, making it difficult to simultaneously preserve fine capillaries and maintain global topological continuity. To address these challenges, we propose the Vessel-aware Frequency-domain and Global Spatial modeling Network (VFGS-Net), an end-to-end
segmentation framework that seamlessly integrates frequency-aware feature enhancement, dual-path convolutional representation learning, and bidirectional asymmetric spatial state-space modeling within a unified architecture. Specifically, VFGS-Net employs a dual-path feature convolution module to jointly capture fine-grained local textures and multi-scale contextual semantics. A novel vessel-aware frequency-domain channel
attention mechanism is introduced to adaptively reweight spectral components, thereby enhancing vessel-relevant responses in high-level features. Furthermore, at the network bottleneck, we propose a bidirectional asymmetric Mamba2-based spatial modeling block to efficiently capture long-range spatial dependencies and strengthen the global continuity of vascular structures. Extensive experiments on four publicly available retinal vessel
datasets demonstrate that VFGS-Net achieves competitive or superior performance compared to state-of-the-art methods. Notably, our model consistently improves
segmentation accuracy for fine vessels, complex branching patterns, and low-contrast regions, highlighting its robustness and clinical potential.
Read the full abstract ▾
All authors Ruiqi Song, Lei Liu, Ya-Nan Zhang, Chao Wang, Xiaoning Li, Nan Mu