Computer Vision
RewardFlow: Generate Images by Optimizing What You Reward
Onkar Susladkar, Dong-Hwan Jang, Tushar Prakash et al. (+7 more)
We introduce RewardFlow, an inversion-free framework that steers pretrained
diffusion and flow-matching models at inference time through multi-reward Langevin dynamics. RewardFlow unifies complementary differentiable rewards for semantic
alignment, perceptual fidelity, localized grounding, object consistency, and human preference, and further introduces a differentiable VQA-based reward that provides fine-grained semantic supervision through
language-
vision reasoning. To coordinate these heterogeneous objectives, we design a
prompt-aware adaptive policy that extracts semantic primitives from the instruction, infers edit intent, and dynamically modulates reward weights and step sizes throughout sampling. Across several image editing and compositional generation
benchmarks, RewardFlow delivers state-of-the-art edit fidelity and compositional
alignment.
Read the full abstract ▾
All authors Onkar Susladkar, Dong-Hwan Jang, Tushar Prakash, Adheesh Juvekar, Vedant Shah, Ayush Barik, Nabeel Bashir, Muntasir Wahed, Ritish Shrirao, Ismini Lourentzou
Computer Vision · Cross-listed from cs.RO
Fail2Drive: Benchmarking Closed-Loop Driving Generalization
Simon Gerstenecker, Andreas Geiger and Katrin Renz
Generalization under distribution shift remains a central bottleneck for closed-loop autonomous driving. Although simulators like CARLA enable safe and scalable testing, existing
benchmarks rarely measure true generalization: they typically reuse training scenarios at test time. Success can therefore reflect memorization rather than robust driving behavior. We introduce Fail2Drive, the first paired-route
benchmark for closed-loop generalization in CARLA, with 200 routes and 17 new scenario classes spanning appearance, layout, behavioral, and robustness shifts. Each shifted route is matched with an in-distribution counterpart, isolating the effect of the shift and turning qualitative failures into quantitative diagnostics. Evaluating multiple state-of-the-art models reveals consistent degradation, with an average success-rate drop of 22.8%. Our analysis uncovers unexpected failure modes, such as ignoring objects clearly visible in the
LiDAR and failing to learn the fundamental concepts of free and occupied space. To accelerate follow-up work, Fail2Drive includes an open-source toolbox for creating new scenarios and validating solvability via a privileged expert policy. Together, these components establish a reproducible foundation for benchmarking and improving closed-loop driving generalization. We open-source all code, data, and tools at https://github.com/autonomousvision/fail2drive .
Read the full abstract ▾
Computer Vision
Self-Improving 4D Perception via Self-Distillation
Nan Huang, Pengcheng Yu, Weijia Zeng et al. (+4 more)
Large-scale multi-view reconstruction models have made remarkable progress, but most existing approaches still rely on fully supervised training with ground-truth 3D/4D annotations. Such annotations are expensive and particularly scarce for dynamic scenes, limiting scalability. We propose SelfEvo, a self-improving framework that continually improves pretrained multi-view reconstruction models using unlabeled
videos. SelfEvo introduces a self-
distillation scheme using spatiotemporal context asymmetry, enabling self-improvement for learning-based 4D perception without external annotations. We systematically study design choices that make self-improvement effective, including loss signals, forms of asymmetry, and other training strategies. Across eight
benchmarks spanning diverse
datasets and domains, SelfEvo consistently improves pretrained baselines and generalizes across base models (e.g. VGGT and $\pi^3$), with significant gains on dynamic scenes. Overall, SelfEvo achieves up to 36.5% relative improvement in
video depth estimation and 20.1% in camera estimation, without using any labeled data. Project Page: https://self-evo.github.io/.
Read the full abstract ▾
All authors Nan Huang, Pengcheng Yu, Weijia Zeng, James M. Rehg, Angjoo Kanazawa, Haiwen Feng, Qianqian Wang
Computer Vision
FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On
Johanna Karras, Yuanhao Wang, Yingwei Li et al. (+1 more)
Given a person and a garment image, virtual try-on (VTO) aims to synthesize a realistic image of the person wearing the garment, while preserving their original pose and identity. Although recent VTO methods excel at visualizing garment appearance, they largely overlook a crucial aspect of the try-on experience: the accuracy of garment fit – for example, depicting how an extra-large shirt looks on an extra-small person. A key obstacle is the absence of
datasets that provide precise garment and body size information, particularly for "ill-fit" cases, where garments are significantly too large or too small. Consequently, current VTO methods default to generating well-fitted results regardless of the garment or person size. In this paper, we take the first steps towards solving this open problem. We introduce FIT (Fit-Inclusive Try-on), a large-scale VTO
dataset comprising over 1.13M try-on image triplets accompanied by precise body and garment measurements. We overcome the challenges of data collection via a scalable synthetic strategy: (1) We programmatically generate 3D garments using GarmentCode and drape them via physics simulation to capture realistic garment fit. (2) We employ a novel re-texturing framework to transform synthetic renderings into photorealistic images while strictly preserving geometry. (3) We introduce person identity preservation into our re-texturing model to generate paired person images (same person, different garments) for supervised training. Finally, we leverage our FIT
dataset to train a baseline fit-aware virtual try-on model. Our data and results set the new state-of-the-art for fit-aware virtual try-on, as well as offer a robust
benchmark for future research. We will make all data and code publicly available on our project page: https://johannakarras.github.io/FIT.
Read the full abstract ▾
All authors Johanna Karras, Yuanhao Wang, Yingwei Li, Ira Kemelmacher-Shlizerman
Computer Vision
UniversalVTG: A Universal and Lightweight Foundation Model for Video Temporal Grounding
Joungbin An, Agrim Jain and Kristen Grauman
Video temporal grounding (VTG) is typically tackled with
dataset-specific models that transfer poorly across domains and query styles. Recent efforts to overcome this limitation have adapted large
multimodal language models (MLLMs) to VTG, but their high compute cost and limited
video context still hinder long-
video grounding. We instead scale unified supervision while keeping the model lightweight. We present UniversalVTG, a single VTG model trained with large-scale cross-
dataset pretraining. An offline Query Unifier canonicalizes heterogeneous query formats into a shared declarative space, reducing linguistic mismatch and preventing the negative transfer observed under naïve joint training. Combined with an efficient grounding head, UniversalVTG scales to long, untrimmed
videos. Across diverse
benchmarks-GoalStep-StepGrounding, Ego4D-NLQ, TACoS, Charades-STA, and ActivityNet-Captions-one UniversalVTG checkpoint achieves state-of-the-art performance versus dedicated VTG models. Moreover, despite being $>100\times$ smaller than recent MLLM-based approaches, UniversalVTG matches or exceeds their accuracy on multiple
benchmarks, offering a practical alternative to parameter-heavy MLLMs.
Read the full abstract ▾
Computer Vision
MolmoWeb: Open Visual Web Agent and Open Data for the Open Web
Tanmay Gupta, Piper Wolters, Zixian Ma et al. (+13 more)
Web
agents–autonomous systems that navigate and execute tasks on the web on behalf of users–have the potential to transform how people interact with the digital world. However, the most capable web
agents today rely on proprietary models with undisclosed training data and recipes, limiting scientific understanding, reproducibility, and community-driven progress. We believe
agents for the open web should be built in the open. To this end, we introduce (1) MolmoWebMix, a large and diverse mixture of browser task demonstrations and web-GUI perception data and (2) MolmoWeb, a family of fully open
multimodal web
agents. Specifically, MolmoWebMix combines over 100K synthetic task trajectories from multiple complementary generation pipelines with 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data, including referring expression grounding and screenshot question answering. MolmoWeb
agents operate as instruction-conditioned visual-
language action policies: given a task instruction and a webpage screenshot, they predict the next browser action, requiring no access to HTML, accessibility trees, or specialized APIs. Available in 4B and 8B size, on browser-use
benchmarks like WebVoyager, Online-Mind2Web, and DeepShop, MolmoWeb
agents achieve state-of-the-art results outperforming similar scale open-weight-only models such as Fara-7B, UI-Tars-1.5-7B, and Holo1-7B. MolmoWeb-8B also surpasses set-of-marks (SoM)
agents built on much larger closed frontier models like GPT-4o. We further demonstrate consistent gains through test-time scaling via parallel rollouts with best-of-N selection, achieving 94.7% and 60.5% pass@4 (compared to 78.2% and 35.3% pass@1) on WebVoyager and Online-Mind2Web respectively. We will release model checkpoints, training data, code, and a unified evaluation harness to enable reproducibility and accelerate open research on web
agents.
Read the full abstract ▾
All authors Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Taylor Blanton, Caleb Ouellette, Winson Han, Ali Farhadi, Ranjay Krishna
Computer Vision
When Fine-Tuning Changes the Evidence: Architecture-Dependent Semantic Drift in Chest X-Ray Explanations
Kabilan Elangovan and Daniel Ting
Transfer learning followed by fine-tuning is widely adopted in medical image classification due to consistent gains in diagnostic performance. However, in multi-class settings with overlapping visual features, improvements in accuracy do not guarantee stability of the visual evidence used to support predictions. We define semantic drift as systematic changes in the attribution structure supporting a model's predictions between transfer learning and full fine-tuning, reflecting potential shifts in underlying visual
reasoning despite stable classification performance. Using a five-class chest X-ray task, we evaluate DenseNet201, ResNet50V2, and InceptionV3 under a two-stage training protocol and quantify drift with reference-free metrics capturing spatial localization and structural consistency of attribution maps. Across architectures, coarse anatomical localization remains stable, while overlap IoU reveals pronounced architecture-dependent reorganization of evidential structure. Beyond single-method analysis, stability rankings can reverse across LayerCAM and GradCAM++ under converged predictive performance, establishing explanation stability as an interaction between architecture, optimization phase, and attribution objective.
Read the full abstract ▾
Computer Vision
Hang Ye, Xiaoxuan Ma, Fan Lu et al. (+3 more)
Digital human generation has been studied for decades and supports a wide range of real-world applications. However, most existing systems are passively animated, relying on privileged state or scripted control, which limits scalability to novel environments. We instead ask: how can digital humans actively behave using only visual observations and specified goals in novel scenes? Achieving this would enable populating any 3D environments with digital humans at scale that exhibit spontaneous, natural, goal-directed behaviors. To this end, we introduce Visually-grounded
Humanoid Agents, a coupled two-layer (world-
agent) paradigm that replicates humans at multiple levels: they look, perceive, reason, and behave like real people in real-world 3D scenes. The World Layer reconstructs semantically rich 3D Gaussian scenes from real-world
videos via an occlusion-aware pipeline and accommodates animatable Gaussian-based human avatars. The
Agent Layer transforms these avatars into autonomous
humanoid agents, equipping them with first-person RGB-D perception and enabling them to perform accurate,
embodied planning with spatial awareness and iterative
reasoning, which is then executed at the low level as full-body actions to drive their behaviors in the scene. We further introduce a
benchmark to evaluate
humanoid-scene interaction in diverse reconstructed environments. Experiments show our
agents achieve robust autonomous behavior, yielding higher task success rates and fewer collisions than ablations and state-of-the-art planning methods. This work enables active digital human population and advances human-centric
embodied AI. Data, code, and models will be open-sourced.
Read the full abstract ▾
All authors Hang Ye, Xiaoxuan Ma, Fan Lu, Wayne Wu, Kwan-Yee Lin, Yizhou Wang
Computer Vision
Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamics
Ying Shen, Jerry Xiong, Tianjiao Yu et al. (+1 more)
Recent advances in generative
video modeling, driven by large-scale
datasets and powerful architectures, have yielded remarkable visual realism. However, emerging evidence suggests that simply scaling data and model size does not endow these systems with an understanding of the underlying physical laws that govern real-world dynamics. Existing approaches often fail to capture or enforce such physical consistency, resulting in unrealistic motion and dynamics. In his work, we investigate whether integrating the inference of latent physical properties directly into the
video generation process can equip models with the ability to produce physically plausible
videos. To this end, we propose Phantom, a Physics-Infused
Video Generation model that jointly models the visual content and latent physical dynamics. Conditioned on observed
video frames and inferred physical states, Phantom jointly predicts latent physical dynamics and generates future
video frames. Phantom leverages a physics-aware
video representation that serves as an abstract yet informaive
embedding of the underlying physics, facilitating the joint prediction of physical dynamics alongside
video content without requiring an explicit specification of a complex set of physical dynamics and properties. By integrating the inference of physical-aware
video representation directly into the
video generation process, Phantom produces
video sequences that are both visually realistic and physically consistent. Quantitative and qualitative results on both standard
video generation and physics-aware
benchmarks demonstrate that Phantom not only outperforms existing methods in terms of adherence to physical dynamics but also delivers competitive perceptual fidelity.
Read the full abstract ▾
All authors Ying Shen, Jerry Xiong, Tianjiao Yu, Ismini Lourentzou
Computer Vision
Quantifying Explanation Consistency: The C-Score Metric for CAM-Based Explainability in Medical Image Classification
Kabilan Elangovan and Daniel Ting
Class Activation Mapping (CAM) methods are widely used to generate visual explanations for deep learning classifiers in medical imaging. However, existing evaluation frameworks assess whether explanations are correct, measured by localisation fidelity against radiologist annotations, rather than whether they are consistent: whether the model applies the same spatial
reasoning strategy across different patients with the same pathology. We propose the C-Score (Consistency Score), a confidence-weighted, annotation-free metric that quantifies intra-class explanation reproducibility via intensity-emphasised pairwise soft IoU across correctly classified instances. We evaluate six CAM techniques: GradCAM, GradCAM++, LayerCAM, EigenCAM, ScoreCAM, and MS GradCAM++ across three CNN architectures (DenseNet201, InceptionV3, ResNet50V2) over thirty training epochs on the Kermany chest X-ray
dataset, covering transfer learning and fine-tuning phases. We identify three distinct mechanisms of AUC-consistency dissociation, invisible to standard classification metrics: threshold-mediated gold list collapse, technique-specific attribution collapse at peak AUC, and class-level consistency masking in global aggregation. C-Score provides an early warning signal of impending model instability. ScoreCAM deterioration on ResNet50V2 is detectable one full checkpoint before catastrophic AUC collapse and yields architecture-specific clinical deployment recommendations grounded in explanation quality rather than predictive ranking alone.
Read the full abstract ▾
Computer Vision
FrameCrafter: Novel View Synthesis as Video Completion
Qi Wu, Khiem Vuong, Minsik Jeon et al. (+2 more)
We tackle the problem of sparse novel view synthesis (NVS) using
video diffusion models; given $K$ ($\approx 5$) multi-view images of a scene and their camera poses, we predict the view from a target camera pose. Many prior approaches leverage generative image priors encoded via
diffusion models. However, models trained on single images lack multi-view knowledge. We instead argue that
video models already contain implicit multi-view knowledge and so should be easier to adapt for NVS. Our key insight is to formulate sparse NVS as a low frame-rate
video completion task. However, one challenge is that sparse NVS is defined over an unordered set of inputs, often too sparse to admit a meaningful order, so the models should be $\textit{invariant}$ to permutations of that input set. To this end, we present FrameCrafter, which adapts
video models (naturally trained with coherent frame orderings) to permutation-invariant NVS through several architectural modifications, including per-frame latent encodings and removal of temporal positional
embeddings. Our results suggest that
video models can be easily trained to "forget" about time with minimal supervision, producing competitive performance on sparse-view NVS
benchmarks. Project page: https://frame-crafter.github.io/
Read the full abstract ▾
All authors Qi Wu, Khiem Vuong, Minsik Jeon, Srinivasa Narasimhan, Deva Ramanan
Computer Vision
What They Saw, Not Just Where They Looked: Semantic Scanpath Similarity via VLMs and NLP metric
Mohamed Amine Kerkouri, Marouane Tliba, Bin Wang et al. (+3 more)
Scanpath similarity metrics are central to eye-movement research, yet existing methods predominantly evaluate spatial and temporal
alignment while neglecting semantic equivalence between attended image regions. We present a semantic scanpath similarity framework that integrates
vision-
language models (VLMs) into eye-tracking analysis. Each fixation is encoded under controlled visual context (patch-based and marker-based strategies) and transformed into concise textual descriptions, which are aggregated into scanpath-level representations. Semantic similarity is then computed using
embedding-based and lexical NLP metrics and compared against established spatial measures, including MultiMatch and DTW. Experiments on free-viewing eye-tracking data demonstrate that semantic similarity captures partially independent variance from
geometric alignment, revealing cases of high content agreement despite spatial divergence. We further analyze the impact of contextual encoding on description fidelity and metric stability. Our findings suggest that
multimodal foundation models enable interpretable, content-aware extensions of classical scanpath analysis, providing a complementary dimension for gaze research within the ETRA community.
Read the full abstract ▾
All authors Mohamed Amine Kerkouri, Marouane Tliba, Bin Wang, Aladine Chetouani, Ulas Bagci, Alessandro Bruno
Computer Vision
Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization
Sai Srinivas Kancheti, Aditya Kanade, Rohit Sinha et al. (+2 more)
Multimodal reasoning models (MRMs) trained with reinforcement learning with verifiable rewards (RLVR) show improved accuracy on visual
reasoning benchmarks. However, we observe that accuracy gains often come at the cost of
reasoning quality: generated Chain-of-Thought (CoT) traces are frequently inconsistent with the final answer and poorly grounded in the visual evidence. We systematically study this phenomenon across seven challenging real-world spatial
reasoning benchmarks and find that it affects contemporary MRMs such as ViGoRL-Spatial, TreeVGR as well as our own models trained with standard Group Relative Policy Optimization (GRPO). We characterize CoT
reasoning quality along two complementary axes: "logical consistency" (does the CoT entail the final answer?) and "visual grounding" (does each
reasoning step accurately describe objects, attributes, and spatial relationships in the image?). To address this, we propose Faithful GRPO (FGRPO), a variant of GRPO that enforces consistency and grounding as constraints via Lagrangian dual ascent. FGRPO incorporates batch-level consistency and grounding constraints into the advantage computation within a group, adaptively adjusting the relative importance of constraints during optimization. We evaluate FGRPO on Qwen2.5-VL-7B and 3B backbones across seven spatial
datasets. Our results show that FGRPO substantially improves
reasoning quality, reducing the inconsistency rate from 24.5% to 1.7% and improving visual grounding scores by +13%. It also improves final answer accuracy over simple GRPO, demonstrating that faithful
reasoning enables better answers.
Read the full abstract ▾
All authors Sai Srinivas Kancheti, Aditya Kanade, Rohit Sinha, Vineeth N Balasubramanian, Tanuja Ganu
Computer Vision
LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation
Jingjing Wang, Zhengdong Hong, Chong Bao et al. (+3 more)
Human-like generalization in open-world remains a fundamental challenge for robotic manipulation. Existing learning-based methods, including reinforcement learning, imitation learning, and
vision-
language-action-models (VLAs), often struggle with novel tasks and unseen environments. Another promising direction is to explore generalizable representations that capture fine-grained spatial and
geometric relations for open-world manipulation. While large-
language-model (LLMs) and
vision-
language-model (VLMs) provide strong semantic
reasoning based on
language or annotated 2D representations, their limited 3D awareness restricts their applicability to fine-grained manipulation. To address this, we propose LAMP, which lifts image-editing as 3D priors to extract inter-object 3D transformations as continuous, geometry-aware representations. Our key insight is that image-editing inherently encodes rich 2D spatial cues, and lifting these implicit cues into 3D transformations provides fine-grained and accurate guidance for open-world manipulation. Extensive experiments demonstrate that \codename delivers precise 3D transformations and achieves strong zero-shot generalization in open-world manipulation. Project page: https://zju3dv.github.io/LAMP/.
Read the full abstract ▾
All authors Jingjing Wang, Zhengdong Hong, Chong Bao, Yuke Zhu, Junhan Sun, Guofeng Zhang
Computer Vision
OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance
Haoxi Zeng, Qiankun Liu, Yi Bin et al. (+5 more)
Open-Vocabulary
Segmentation (OVS) aims to segment image regions beyond predefined category sets by leveraging semantic descriptions. While CLIP based approaches excel in semantic generalization, they frequently lack the fine-grained spatial awareness required for dense prediction. Recent efforts have incorporated
Vision Foundation Models (VFMs) like DINO to alleviate these limitations. However, these methods still struggle with the precise edge perception necessary for high fidelity
segmentation. In this paper, we analyze internal representations of DINO and discover that its inherent boundary awareness is not absent but rather undergoes progressive attenuation as features transition into deeper
transformer blocks. To address this, we propose OVS-DINO, a novel framework that revitalizes latent edge-sensitivity of DINO through structural
alignment with the Segment Anything Model (SAM). Specifically, we introduce a Structure-Aware Encoder (SAE) and a Structure-Modulated Decoder (SMD) to effectively activate boundary features of DINO using SAM's structural priors, complemented by a supervision strategy utilizing SAM generated pseudo-masks. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple weakly-supervised OVS
benchmarks, improving the average score by 2.1% (from 44.8% to 46.9%). Notably, our approach significantly enhances
segmentation accuracy in complex, cluttered scenarios, with a gain of 6.3% on Cityscapes (from 36.6% to 42.9%).
Read the full abstract ▾
All authors Haoxi Zeng, Qiankun Liu, Yi Bin, Haiyue Zhang, Yujuan Ding, Guoqing Wang, Deqiang Ouyang, Heng Tao Shen
Computer Vision
CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning
Rui Gan, Junyi Ma, Pei Li et al. (+4 more)
Cooperative autonomous driving requires traffic scene understanding from both vehicle and infrastructure perspectives. While
vision-
language models (VLMs) show strong general
reasoning capabilities, their performance in safety-critical traffic scenarios remains insufficiently evaluated due to the ego-vehicle focus of existing
benchmarks. To bridge this gap, we present CrashSight, a large-scale
vision-
language benchmark for roadway crash understanding using real-world roadside camera data. The
dataset comprises 250 crash
videos, annotated with 13K multiple-choice question-answer pairs organized under a two-tier taxonomy. Tier 1 evaluates the visual grounding of scene context and involved parties, while Tier 2 probes higher-level
reasoning, including crash mechanics, causal attribution, temporal progression, and post-crash outcomes. We
benchmark 8 state-of-the-art VLMs and show that, despite strong scene description capabilities, current models struggle with temporal and causal
reasoning in safety-critical scenarios. We provide a detailed analysis of failure scenarios and discuss directions for improving VLM crash understanding. The
benchmark provides a standardized evaluation framework for infrastructure-assisted perception in cooperative autonomous driving. The CrashSight
benchmark, including the full
dataset and code, is accessible at https://mcgrche.github.io/crashsight.
Read the full abstract ▾
All authors Rui Gan, Junyi Ma, Pei Li, Xingyou Yang, Kai Chen, Sikai Chen, Bin Ran
Computer Vision
Marcel Gröpl, Jaewoo Jung, Seungryong Kim et al. (+2 more)
Despite rapid progress, pretrained
vision-
language models still struggle when answers depend on tiny visual details or on combining clues spread across multiple regions, as in documents and compositional queries. We address this by framing grounding as test-time evidence
retrieval: given a query, the model should actively identify where to look next to resolve ambiguity. To this end, we propose a training-free, model-intrinsic grounding method that uses uncertainty as supervision. Specifically, we compute the
entropy of the model's next-
token distribution and backpropagate it to the visual
token embeddings to obtain an
entropy-gradient relevance map, without auxiliary detectors or
attention-map heuristics. We then extract and rank multiple coherent regions to support multi-evidence queries, and introduce an iterative zoom-and-reground procedure with a spatial-
entropy stopping rule to avoid over-refinement. Experiments on seven
benchmarks across four VLM architectures demonstrate consistent improvements over existing methods, with the largest gains on detail-critical and high-resolution settings, while also producing more interpretable evidence localizations.
Read the full abstract ▾
All authors Marcel Gröpl, Jaewoo Jung, Seungryong Kim, Marc Pollefeys, Sunghwan Hong
Computer Vision
HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment
Changdao Chen, Qinqiuhong Ye, Hao Chen et al. (+1 more)
It remains challenging to assess driver fatigue from untrimmed
videos under constrained computational budgets, due to the difficulty of modeling long-range temporal dependencies in subtle facial expressions. Some existing approaches rely on computationally heavy architectures, whereas others employ traditional lightweight pairwise graph networks, despite their limited capacity to model high-order synergies and global temporal context. Therefore, we propose HST-HGN, a novel Heterogeneous Spatial-Temporal Hypergraph Network driven by Bidirectional State Space Models. Spatially, we introduce a hierarchical hypergraph network to fuse pose-disentangled
geometric topologies with multi-modal texture patches dynamically. This formulation encapsulates high-order synergistic facial deformations, effectively overcoming the limitations of conventional methods. In temporal terms, a Bi-Mamba module with linear complexity is applied to perform bidirectional sequence modeling. This explicit temporal-
evolution filtering enables the network to distinguish highly ambiguous transient actions, such as yawning versus speaking, while encompassing their complete physiological lifecycles. Extensive evaluations across diverse fatigue
benchmarks demonstrate that HST-HGN achieves state-of-the-art performance. In particular, our method strikes a balance between discriminative power and computational efficiency, making it well-suited for real-time in-cabin edge deployment.
Read the full abstract ▾
All authors Changdao Chen, Qinqiuhong Ye, Hao Chen, Jinyu Wang
Computer Vision
BLaFA: Bridging Language to 3D Functional Affordances for Dexterous Robotic Grasping in Tool Use
Fan Yang, Wenrui Chen, Guorun Yan et al. (+7 more)
In industrial assembly, equipment maintenance, and everyday tool-use scenarios,
robots need not only to grasp objects stably but also to establish task-appropriate contact configurations according to task intentions and tool functionality. However, existing methods lack explicit mappings from open-vocabulary instructions to 3D functional contact structures and dexterous grasp configurations, limiting
robots' adaptability to diverse tools and functional tasks. To address this limitation, we propose BLaFA (Bridging
Language to 3D Functional Affordances for Dexterous Robotic Grasping), a structured framework for
language-driven functional dexterous grasping. First, a Knowledge-guided
Language Parsing (KLP) module leverages structured knowledge to transform natural-
language instructions into semantic sextuples of manipulation constraints. Second, a 3D Gaussian Splatting based hierarchical semantic representation and a Triangular Functional Point Localization (TriLocation) module are developed to localize 3D functional contact keypoints under task-tool
topology constraints. Finally, a 3D Keypoint Grasp
Matrix Transformation (KGT3D+) module maps the localized keypoints and semantic constraints to executable wrist poses and finger-level joint and force commands. Evaluations on 100 open-vocabulary
language instructions, 14 task-tool combinations covering four functional intents, and real-world robotic experiments in representative scenes demonstrate that BLaFA can effectively translate natural-
language instructions into task-aligned 3D functional contact structures and corresponding dexterous grasps, thereby enabling
language-driven robotic tool use across diverse tasks. The source code will be made publicly available. Code will be publicly available at https://github.com/PopeyePxx/BLaFA.
Read the full abstract ▾
All authors Fan Yang, Wenrui Chen, Guorun Yan, Ruize Liao, Wanjun Jia, Dongsheng Luo, Jiacheng Lin, Kailun Yang, Zhiyong Li, Yaonan Wang
Computer Vision
SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation
Wenli Zhang, Xianglong Shi, Sirui Zhao et al. (+5 more)
Diffusion-based audio-driven talking-head generation enables realistic portrait animation, but also introduces risks of misuse, such as fraud and misinformation. Existing protection methods are largely limited to a single modality, and neither image-only nor audio-only attacks can effectively suppress
speech-driven facial dynamics. To address this gap, we propose SyncBreaker, a stage-aware
multimodal protection framework that jointly perturbs portrait and audio inputs under modality-specific perceptual constraints. Our key contributions are twofold. First, for the image stream, we introduce nullifying supervision with Multi-Interval Sampling (MIS) across
diffusion stages to steer the generation toward the static reference portrait by aggregating guidance from multiple denoising intervals. Second, for the audio stream, we propose Cross-
Attention Fooling (CAF), which suppresses interval-specific audio-conditioned cross-
attention responses. Both streams are optimized independently and combined at inference time to enable flexible deployment. We evaluate SyncBreaker in a white-box proactive protection setting. Extensive experiments demonstrate that SyncBreaker more effectively degrades lip synchronization and facial dynamics than strong single-modality baselines, while preserving input perceptual quality and remaining robust under purification. Code: https://github.com/kitty384/SyncBreaker.
Read the full abstract ▾
All authors Wenli Zhang, Xianglong Shi, Sirui Zhao, Xinqi Chen, Guo Cheng, Yifan Xu, Tong Xu, Yong Liao
Computer Vision
Nam Duong Tran and Phi Le Nguyen
Recent advances in
Vision-
Language Models (VLMs) have greatly enhanced the integration of visual perception and linguistic
reasoning, driving rapid progress in
multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to
backdoor attacks, remains significantly underexplored. Existing
backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM
backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g.,
vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive
backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings.
Read the full abstract ▾
Computer Vision
SurfelSplat: Learning Efficient and Generalizable Gaussian Surfel Representations for Sparse-View Surface Reconstruction
Chensheng Dai, Shengjun Zhang, Min Chen et al. (+1 more)
3D Gaussian Splatting (3DGS) has demonstrated impressive performance in 3D scene reconstruction. Beyond novel view synthesis, it shows great potential for multi-view surface reconstruction. Existing methods employ optimization-based reconstruction pipelines that achieve precise and complete surface extractions. However, these approaches typically require dense input views and high time consumption for per-scene optimization. To address these limitations, we propose SurfelSplat, a feed-forward framework that generates efficient and generalizable pixel-aligned Gaussian surfel representations from sparse-view images. We observe that conventional feed-forward structures struggle to recover accurate
geometric attributes of Gaussian surfels because the spatial frequency of pixel-aligned primitives exceeds Nyquist sampling rates. Therefore, we propose a cross-view feature aggregation module based on the Nyquist sampling
theorem. Specifically, we first adapt the
geometric forms of Gaussian surfels with spatial sampling rate-guided low-pass filters. We then project the filtered surfels across all input views to obtain cross-view feature correlations. By processing these correlations through a specially designed feature fusion network, we can finally regress Gaussian surfels with precise geometry. Extensive experiments on DTU reconstruction
benchmarks demonstrate that our model achieves comparable results with state-of-the-art methods, and predict Gaussian surfels within 1 second, offering a 100x speedup without costly per-scene training.
Read the full abstract ▾
All authors Chensheng Dai, Shengjun Zhang, Min Chen, Yueqi Duan
Computer Vision · Cross-listed from cs.LG
SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization
Seyed Mahmoud Sajjadi Mohammadabadi, Xiaolong Ma, Lei Yang et al. (+2 more)
Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, enable scalable adaptation of foundation models by injecting low-rank adapters. However, their communication and storage costs remain a major bottleneck in resource-constrained settings. We propose
SOLAR (Subspace-Oriented Latent Adapter Reparameterization), a post-training compression framework that substantially reduces the communication cost (i.e., the number of parameters to transmit or store) of PEFT adapters.
SOLAR expresses each PEFT update as a linear combination of basis vectors formed from the foundation model's singular vectors with controlled random perturbations. By exploiting the subspace similarity (the
alignment of principal directions) between the foundation model and task-specific fine-tuned updates,
SOLAR decouples the adapter size from PEFT structure and ensures compact yet expressive representations. It is model-agnostic and compatible with existing PEFT methods, including LoRA, AdaLoRA, and other adapter modules. We theoretically establish a bound on the reconstruction error. Experiments on
language and
vision tasks using LLaMA, GPT, and ViT models demonstrate that
SOLAR preserves task performance while significantly reducing model representation sizes, offering an effective and communication-efficient solution for deployment in distributed systems and edge devices.
Read the full abstract ▾
All authors Seyed Mahmoud Sajjadi Mohammadabadi, Xiaolong Ma, Lei Yang, Feng Yan, Junshan Zhang
Computer Vision · Cross-listed from cs.LG
Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems
Tolga Dimlioglu, Nadine Chang, Maying Shen et al. (+2 more)
Large-scale deep learning models for physical AI applications depend on diverse training data collection efforts. These models and correspondingly, the training data, must address different evaluation criteria necessary for the models to be deployable in real-world environments. Data selection policies can guide the development of the training set, but current frameworks do not account for the ambiguity in how data points affect different metrics. In this work, we propose Mixture Optimization via Scaling-Aware Iterative Collection (MOSAIC), a general data selection framework that operates by: (i) partitioning the
dataset into domains; (ii) fitting neural scaling laws from each data domain to the evaluation metrics; and (iii) optimizing a data mixture by iteratively adding data from domains that maximize the change in metrics. We apply MOSAIC to autonomous driving (AD), where an End-to-End (E2E) planner model is evaluated on the Extended Predictive Driver Model Score (EPDMS), an aggregate of driving rule compliance metrics. Here, MOSAIC outperforms a diverse set of baselines on EPDMS with up to 80% less data.
Read the full abstract ▾
All authors Tolga Dimlioglu, Nadine Chang, Maying Shen, Rafid Mahmood, Jose M. Alvarez
Computer Vision
MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping
Junyao Gao, Sibo Liu, Jiaxing Li et al. (+6 more)
In this paper, we introduce MegaStyle, a novel and scalable data curation pipeline that constructs an intra-style consistent, inter-style diverse and high-quality style
dataset. We achieve this by leveraging the consistent text-to-image style mapping capability of current large generative models, which can generate images in the same style from a given style description. Building on this foundation, we curate a diverse and balanced
prompt gallery with 170K style
prompts and 400K content
prompts, and generate a large-scale style
dataset MegaStyle-1.4M via content-style
prompt combinations. With MegaStyle-1.4M, we propose style-supervised contrastive learning to fine-tune a style encoder MegaStyle-Encoder for extracting expressive, style-specific representations, and we also train a FLUX-based style transfer model MegaStyle-FLUX. Extensive experiments demonstrate the importance of maintaining intra-style consistency, inter-style diversity and high-quality for style
dataset, as well as the effectiveness of the proposed MegaStyle-1.4M. Moreover, when trained on MegaStyle-1.4M, MegaStyle-Encoder and MegaStyle-FLUX provide reliable style similarity measurement and generalizable style transfer, making a significant contribution to the style transfer community. More results are available at our project website https://jeoyal.github.io/MegaStyle/.
Read the full abstract ▾
All authors Junyao Gao, Sibo Liu, Jiaxing Li, Yanan Sun, Yuanpeng Tu, Fei Shen, Weidong Zhang, Cairong Zhao, Jun Zhang
Computer Vision
Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time
Ruizhi Zhang, Ye Huang, Yuangang Pan et al. (+5 more)
While artificial intelligence has mastered structured games like chess and Go,
vision-
language agents still struggle in visually-driven 3D games without access to game states. Existing game environments typically evaluate a fixed
agent configuration, rather than an
agent's ability to improve its configuration across consecutive episodes of the same task—a paradigm known as test-time learning (TTL). Furthermore, current TTL methods typically optimize single modalities—such as text
prompts or actions—in isolation, ignoring the synergy between perception,
reasoning, and control. To bridge these gaps, we first introduce PokeGym, a long-horizon
benchmark built upon the 3D open-world game Pokémon Legends: Z-A, where
agents act from visual observations without access to game states, designed to evaluate an
agent's ability to learn and adapt across consecutive episodes of the task. To tackle this challenging environment, we propose Graph-Guided Evolutionary
Multimodal Agent Configuration (G-EvoMAC), a graph-guided framework that jointly optimizes visual perception, strategy, and action set synergistically. Extensive experiments show that G-EvoMAC achieves a 60.18% average success rate on PokeGym, outperforming the strongest baseline by over 11 percentage points, validating the power of cross-modal co-
evolution.
Read the full abstract ▾
All authors Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Haijun Lei, Lixin Duan
Computer Vision
InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding
Ashutosh Kumar, Rajat Saini, Jingjing Pan et al. (+5 more)
Current
vision-
language pre-training (VLP) paradigms excel at global scene understanding but struggle with instance-level
reasoning due to global-only supervision. We introduce InstAP, an Instance-Aware Pre-training framework that jointly optimizes global
vision-text
alignment and fine-grained, instance-level contrastive
alignment by grounding textual mentions to specific spatial-temporal regions. To support this, we present InstVL, a large-scale
dataset (2 million images, 50,000
videos) with dual-granularity annotations: holistic scene captions and dense, grounded instance descriptions. On the InstVL
benchmark, InstAP substantially outperforms existing VLP models on instance-level
retrieval, and also surpasses a strong VLP baseline trained on the exact same data corpus, isolating the benefit of our instance-aware objective. Moreover, instance-centric pre-training improves global understanding: InstAP achieves competitive zero-shot performance on multiple
video benchmarks, including MSR-VTT and DiDeMo. Qualitative visualizations further show that InstAP localizes textual mentions to the correct instances, while global-only models exhibit more diffuse, scene-level
attention.
Read the full abstract ▾
All authors Ashutosh Kumar, Rajat Saini, Jingjing Pan, Mustafa Erdogan, Mingfang Zhang, Betty Le Dem, Norimasa Kobori, Quan Kong
Computer Vision
Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification
Xun Zhu, Fanbin Mo, Xi Chen et al. (+6 more)
The rise of
multimodal large
language models (MLLMs) has sparked an unprecedented wave of applications in the field of medical imaging analysis. However, as one of the earliest and most fundamental tasks integrated into this paradigm, medical image classification reveals a sobering reality: state-of-the-art medical MLLMs consistently underperform compared to traditional deep learning models, despite their overwhelming advantages in pre-training data and model parameters. This paradox
prompts a critical rethinking: where exactly does the performance degradation originate? In this paper, we conduct extensive experiments on 14 open-source medical MLLMs across three representative image classification
datasets. Moving beyond superficial performance benchmarking, we employ feature probing to track the information flow of visual features module-by-module and layer-by-layer throughout the entire MLLM pipeline, enabling explicit visualization of where and how classification signals are distorted, diluted, or overridden. As the first attempt to dissect classification performance degradation in medical MLLMs, our findings reveal four failure modes: 1) quality limitation in visual representation, 2) fidelity loss in connector projection, 3) comprehension deficit in LLM
reasoning, and 4) misalignment of semantic mapping. Meanwhile, we introduce quantitative scores that characterize the healthiness of feature
evolution, enabling principled comparisons across diverse MLLMs and
datasets. Furthermore, we provide insightful discussions centered on the critical barriers that prevent current medical MLLMs from fulfilling their promised clinical potential. We hope that our work provokes rethinking within the community-highlighting that the road from high expectations to clinically deployable MLLMs remains long and winding.
Read the full abstract ▾
All authors Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, Ji Wu
Computer Vision
Fundus-R1: Training a Fundus-Reading MLLM on Public Data
Yuchuan Deng, Qijie Wei, Kaiheng Qian et al. (+6 more)
Fundus imaging such as CFP, OCT and SLO is crucial for the early detection of retinal anomalies and diseases. Fundus image understanding, due to its knowledge-intensive nature, poses a challenging
vision-
language task. An emerging approach to addressing the task is to post-train a generic
multimodal large
language model (MLLM), either by supervised finetuning (SFT) or by reinforcement learning with verifiable rewards (RLVR), on a considerable amount of in-house samples paired with high-quality clinical reports. However, these valuable samples are not publicly accessible, which not only hinders reproducibility but also practically limits research to few players. In this paper, we make a novel attempt to train a
reasoning-enhanced fundus-reading MLLM, termed Fundus-R1, using exclusively public
datasets, wherein over 94% of the data are annotated with image-level labels only. Our technical contributions are two-fold. First, we propose a RAG-based method for composing image-specific, knowledge-aware
reasoning traces. Such auto-generated traces link visual findings identified by a generic MLLM to the image labels in terms of ophthalmic knowledge. Second, we enhance RLVR with a process reward that encourages self-consistency of the generated
reasoning trace in each rollout. Extensive experiments on three fundus-reading
benchmarks, i.e., FunBench, Omni-Fundus and GMAI-Fundus, show that Fundus-R1 clearly outperforms multiple baselines, including its generic counterpart (Qwen2.5-VL) and a stronger edition post-trained without using the generated traces. This work paves the way for training powerful fundus-reading MLLMs with publicly available data.
Read the full abstract ▾
All authors Yuchuan Deng, Qijie Wei, Kaiheng Qian, Jiazhen Liu, Bangxiang Lan, Qingwei Shen, Jianfeng Dong, Weihong Yu, Xirong Li
Computer Vision
Weakly-Supervised Lung Nodule Segmentation via Training-Free Guidance of 3D Rectified Flow
Richard Petersen, Fredrik Kahl and Jennifer Alvén
Dense annotations, such as
segmentation masks, are expensive and time-consuming to obtain, especially for 3D medical images where expert voxel-wise labeling is required. Weakly supervised approaches aim to address this limitation, but often rely on attribution-based methods that struggle to accurately capture small structures such as lung nodules. In this paper, we propose a weakly-supervised
segmentation method for lung nodules by combining pretrained state-of-the-art rectified flow and predictor models in a plug-and-play manner. Our approach uses training-free guidance of a 3D rectified flow model, requiring only fine-tuning of the predictor using image-level labels and no retraining of the generative model. The proposed method produces improved-quality
segmentations for two separate predictors, consistently detecting lung nodules of varying size and shapes. Experiments on LUNA16 demonstrate improvements over baseline methods, highlighting the potential of generative foundation models as tools for weakly supervised 3D medical image
segmentation.
Read the full abstract ▾