Computer Vision
PhysHMR: Learning Humanoid Control Policies from Vision for Physically Plausible Human Motion Reconstruction
Qiao Feng, Yiming Huang, Yufu Wang et al. (+2 more)
Reconstructing physically plausible human motion from
monocular videos remains a challenging problem in computer
vision and graphics. Existing methods primarily focus on kinematics-based pose estimation, often leading to unrealistic results due to the lack of physical constraints. To address such artifacts, prior methods have typically relied on physics-based post-processing following the initial kinematics-based motion estimation. However, this two-stage design introduces error accumulation, ultimately limiting the overall reconstruction quality. In this paper, we present PhysHMR, a unified framework that directly learns a visual-to-action policy for
humanoid control in a physics-based simulator, enabling motion reconstruction that is both physically grounded and visually aligned with the input
video. A key component of our approach is the pixel-as-ray strategy, which lifts 2D keypoints into 3D spatial rays and transforms them into global space. These rays are incorporated as policy inputs, providing robust global pose guidance without depending on noisy 3D root predictions. This soft global grounding, combined with local visual features from a pretrained encoder, allows the policy to reason over both detailed pose and global positioning. To overcome the sample inefficiency of reinforcement learning, we further introduce a
distillation scheme that transfers motion knowledge from a mocap-trained expert to the
vision-conditioned policy, which is then refined using physically motivated reinforcement learning rewards. Extensive experiments demonstrate that PhysHMR produces high-fidelity, physically plausible motion across diverse scenarios, outperforming prior approaches in both visual accuracy and physical realism.
Read the full abstract ▾
All authors Qiao Feng, Yiming Huang, Yufu Wang, Jiatao Gu, Lingjie Liu
Computer Vision
Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback
Derek Shi, Ruben Glatt, Christine Klymko et al. (+5 more)
Recent advances in large
video-
language models (VLMs) rely on extensive fine-tuning techniques that strengthen
alignment between textual and visual comprehension. Leading pipelines typically pair supervised fine-tuning (SFT) with reinforcement learning from preference data to enhance
video comprehension. However, as VLMs scale in parameter size, so does the cost of gathering enough human feedback. To make fine-tuning more cost-effective, recent frameworks explore reinforcement learning with AI feedback (RLAIF), which replace human preference with AI as a judge. Current RLAIF frameworks rely on a specialized reward model trained with
video narratives to create calibrated scalar rewards – an expensive and restrictive pipeline. We propose Oracle-RLAIF, a novel framework that replaces the trained reward model with a more general Oracle ranker which acts as a drop-in model ranking candidate model responses rather than scoring them. Alongside Oracle-RLAIF, we introduce $GRPO_{rank}$, a novel rank-based loss function based on Group Relative Policy Optimization (GRPO) that directly optimizes ordinal feedback with rank-aware advantages. Empirically, we demonstrate that Oracle-RLAIF consistently outperforms leading VLMs using existing fine-tuning methods when evaluated across various
video comprehension
benchmarks. Oracle-RLAIF paves the path to creating flexible and data-efficient frameworks for aligning large multi-modal
video models with reinforcement learning from rank rather than score.
Read the full abstract ▾
All authors Derek Shi, Ruben Glatt, Christine Klymko, Shubham Mohole, Hongjun Choi, Shashank Kushwaha, Sam Sakla, Felipe Leno da Silva
Computer Vision
Exploring OCR-augmented Generation for Bilingual VQA
JoonHo Lee and Sunho Park
We investigate OCR-augmented generation with
Vision Language Models (VLMs), exploring tasks in Korean and English toward multilingualism. To support research in this domain, we train and release KLOCR, a strong bilingual OCR baseline trained on 100M instances to augment VLMs with OCR ability. To complement existing VQA
benchmarks, we curate KOCRBench for Korean VQA, and analyze different prompting methods. Extensive experiments show that OCR-extracted text significantly boosts performance across open source and commercial models. Our work offers new insights into OCR-augmented generation for bilingual VQA. Model, code, and data are available at https://github.com/JHLee0513/KLOCR.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Learning a distance measure from the information-estimation geometry of data
Guy Ohayon, Pierre-Etienne H. Fiquet, Florentin Guth et al. (+2 more)
We introduce the Information-Estimation Metric (IEM), a novel form of distance function derived from an underlying continuous probability density over a domain of signals. The IEM is rooted in a fundamental relationship between information theory and estimation theory, which links the log-probability of a signal with the errors of an optimal denoiser, applied to noisy observations of the signal. In particular, the IEM between a pair of signals is obtained by comparing their denoising error vectors over a range of noise amplitudes. Geometrically, this amounts to comparing the score vector fields of the blurred density around the signals over a range of blur levels. We prove that the IEM is a valid global distance metric and derive a closed-form expression for its local second-order approximation, which yields a Riemannian metric. For Gaussian-distributed signals, the IEM coincides with the Mahalanobis distance. But for more complex distributions, it adapts, both locally and globally, to the geometry of the distribution. In practice, the IEM can be computed using a learned denoiser (analogous to generative
diffusion models) and solving a one-dimensional integral. To demonstrate the value of our framework, we learn an IEM on the ImageNet database. Experiments show that this IEM is competitive with or outperforms state-of-the-art supervised image quality metrics in predicting human perceptual judgments.
Read the full abstract ▾
All authors Guy Ohayon, Pierre-Etienne H. Fiquet, Florentin Guth, Jona Ballé, Eero P. Simoncelli
Computer Vision · Cross-listed from cs.RO
SIMSplat: Language-Aligned 4D Gaussian Splatting for Driving Scenario Generation
Sung-Yeon Park, Adam Lee, Juanwu Lu et al. (+6 more)
Driving scene manipulation using real-world sensor data has emerged as a promising alternative to traditional driving simulators. Despite advances in
language control and neural scene representations, existing methods treat grounding, editing, and simulation as loosely connected stages, relying on heuristic object localization, manual guidance, and single-
agent validation, thereby constraining semantic expressiveness and hindering scalable, reactive scenario generation. We introduce SIMSplat, a driving scene editor built on scene-graph-based 4D Gaussian Splatting augmented with
language-aligned features. By
embedding appearance, motion, and location semantics directly into Gaussian scene-graph nodes, SIMSplat makes reconstructed scenes queryable through free-form natural
language, bridging
language understanding to object-level editing and multi-
agent simulation within a single framework. Building on this
language-grounded scene graph, SIMSplat supports diverse edits including fine-grained pedestrian manipulation, while a multi-
agent path refinement module propagates changes across all
agents to ensure reactive, physically plausible simulations. The pipeline further integrates with
Vision-
Language Models for automated scenario mining. Experiments show that SIMSplat more than doubles baseline grounding accuracy, achieves the highest task completion rate, and produces the lowest failure rates across diverse driving scenarios.
Read the full abstract ▾
All authors Sung-Yeon Park, Adam Lee, Juanwu Lu, Can Cui, Luyang Jiang, Rohit Gupta, Kyungtae Han, Ahmadreza Moradipari, Ziran Wang
Computer Vision · Cross-listed from cs.CL
Words That Make Language Models Perceive
Sophie L. Wang, Phillip Isola and Brian Cheung
Large
language models (LLMs) trained purely on text ostensibly lack any direct perceptual experience, yet their internal representations are implicitly shaped by
multimodal regularities encoded in
language. We test the hypothesis that explicit sensory prompting can surface this latent structure, bringing a text-only LLM into closer representational
alignment with specialist
vision and audio encoders. When a sensory
prompt tells the model to 'see' or 'hear', it cues the model to resolve its next-
token predictions as if they were conditioned on latent visual or auditory evidence that is never actually supplied. Our findings reveal that lightweight
prompt engineering can reliably activate modality-appropriate representations in purely text-trained LLMs.
Read the full abstract ▾
Computer Vision
FOCUS: Optimal Control for Multi-Entity World Modeling in Text-to-Image Generation
Eric Tillmann Bill, Enis Simsar and Thomas Hofmann
Text-to-image (T2I) models excel on single-entity
prompts but struggle with multi-entity scenes, often exhibiting attribute leakage, identity
entanglement, and subject omissions. We present a principled theoretical framework that steers sampling toward multi-subject fidelity by casting flow matching (FM) as
stochastic optimal control (SOC), yielding a single hyperparameter controlled trade-off between fidelity and object-centric state separation / binding consistency. Within this framework, we derive two architecture-agnostic
algorithms: (i) a training-free test-time controller that perturbs the base velocity with a single-pass update, and (ii) Adjoint Matching, a lightweight fine-tuning rule that regresses a control network to a backward adjoint signal. The same formulation unifies prior
attention heuristics, extends to
diffusion models via a flow–
diffusion correspondence, and provides the first fine-tuning route explicitly designed for multi-subject fidelity. In addition, we also introduce FOCUS (Flow Optimal Control for Unentangled Subjects), a probabilistic
attention-binding objective compatible with both
algorithms. Empirically, on Stable
Diffusion 3.5 and FLUX.1, both
algorithms consistently improve multi-subject
alignment while maintaining base-model style; test-time control runs efficiently on commodity GPUs, and fine-tuned models generalize to unseen
prompts.
Read the full abstract ▾
Computer Vision
StealthAttack: Robust 3D Gaussian Splatting Poisoning via Density-Guided Illusions
Bo-Hsu Ke, You-Zhe Xie, Yu-Lun Liu et al. (+1 more)
3D scene representation methods like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have significantly advanced novel view synthesis. As these methods become prevalent, addressing their vulnerabilities becomes critical. We analyze 3DGS robustness against image-level poisoning attacks and propose a novel density-guided poisoning method. Our method strategically injects Gaussian points into low-density regions identified via Kernel Density Estimation (KDE),
embedding viewpoint-dependent illusory objects clearly visible from poisoned views while minimally affecting innocent views. Additionally, we introduce an adaptive noise strategy to disrupt multi-view consistency, further enhancing attack effectiveness. We propose a KDE-based evaluation protocol to assess attack difficulty systematically, enabling objective benchmarking for future research. Extensive experiments demonstrate our method's superior performance compared to state-of-the-art techniques. Project page: https://hentci.github.io/stealthattack/
Read the full abstract ▾
All authors Bo-Hsu Ke, You-Zhe Xie, Yu-Lun Liu, Wei-Chen Chiu
Computer Vision
Clink! Chop! Thud! – Learning Object Sounds from Real-World Interactions
Mengyu Yang, Yiming Chen, Haozheng Pei et al. (+3 more)
Can a model distinguish between the sound of a spoon hitting a hardwood floor versus a carpeted one? Everyday object interactions produce sounds unique to the objects involved. We introduce the sounding object detection task to evaluate a model's ability to link these sounds to the objects directly involved. Inspired by human perception, our
multimodal object-aware framework learns from in-the-wild
egocentric videos. To encourage an object-centric approach, we first develop an automatic pipeline to compute
segmentation masks of the objects involved to guide the model's focus during training towards the most informative regions of the interaction. A slot
attention visual encoder is used to further enforce an object prior. We demonstrate state of the art performance on our new task along with existing
multimodal action understanding tasks.
Read the full abstract ▾
All authors Mengyu Yang, Yiming Chen, Haozheng Pei, Siddhant Agarwal, Arun Balajee Vasudevan, James Hays
Computer Vision
Inferring Dynamic Physical Properties from Video Foundation Models
Guanqi Zhan, Xianzheng Ma, Weidi Xie et al. (+1 more)
We study the task of predicting dynamic physical properties from
videos. More specifically, we consider physical properties that require temporal information to be inferred: elasticity of a bouncing object, viscosity of a flowing
liquid, and dynamic friction of an object sliding on a surface. To this end, we make the following contributions: (i) We collect a new
video dataset for each physical property, consisting of synthetic training and testing splits, as well as a real split for real world evaluation. (ii) We explore three ways to infer the physical property from
videos: (a) an oracle method where we supply the visual cues that intrinsically reflect the property using classical computer
vision techniques; (b) a simple read out mechanism using a visual
prompt and trainable
prompt vector for cross-
attention on pre-trained
video generative and self-supervised models; and (c)
prompt strategies for Multi-modal Large
Language Models (MLLMs). (iii) We show that a
video foundation model trained in a generative (DynamiCrafter) or trained in a self-supervised manner (V-JEPA-2) achieve a generally similar performance, though behind that of the oracle, and that MLLMs are currently inferior to the other models, though their performance can be improved through suitable prompting. The
dataset, model, and code are available at https://www.
robots.ox.ac.uk/~vgg/research/idpp/.
Read the full abstract ▾
All authors Guanqi Zhan, Xianzheng Ma, Weidi Xie, Andrew Zisserman
Computer Vision
NoiseShift: Resolution-Aware Noise Recalibration for Better Low-Resolution Image Generation
Ruozhen He, Moayed Haji-Ali, Ziyan Yang et al. (+1 more)
Text-to-image
diffusion models often degrade when sampled at resolutions outside the final training resolution set. Prior work has largely emphasized higher resolution generation, enabling pretrained
diffusion models to extrapolate beyond the resolutions seen during training. In this work, we instead target lower-resolution generation, performing inference at reduced resolution to significantly cut computational cost. We show that network conditioning of the noise level induces a train-test mismatch that directly degrades low-resolution generation: the same scheduled noise level can correspond to a different perceptual corruption level at lower resolutions, mis-calibrating the denoiser timestep and noise
embedding. To this end, we propose NoiseShift, a training-free recalibration method that keeps the original noise sampling schedule unchanged and instead re-indexes the noise conditioning of the denoiser to restore local forward-reverse consistency. Using a lightweight coarse-to-fine calibration on a small set of image-text pairs, NoiseShift learns a resolution-specific mapping from scheduler noise to conditioning noise, reducing train-test mismatch and improving lower-resolution generation quality. When NoiseShift is applied to Stable
Diffusion 3 (SD3), Stable
Diffusion 3.5 (SD3.5), and Flux-Dev, generation quality at low resolutions improves consistently. Particularly, SD3 generation at 128x128 resolution gets an improved FID score from 203 to 171, and SD3.5 gets an improved FID score from 310 to 277 on LAION-COCO. Even Flux-Dev which already implements a complementary time-shifting strategy gets a modest boost from NoiseShift with an improved FID score from 120 to 113 at 64x64 resolution. More importantly, NoiseShift achieves such improvements with minimal implementation changes and no additional inference overhead.
Read the full abstract ▾
All authors Ruozhen He, Moayed Haji-Ali, Ziyan Yang, Vicente Ordonez
Computer Vision · Cross-listed from cs.LG
Equilibrium Matching: Generative Modeling with Implicit Energy-Based Models
Runqian Wang and Yilun Du
We introduce
Equilibrium Matching (EqM), a generative modeling framework built from an
equilibrium dynamics perspective. EqM discards the non-
equilibrium, time-conditional dynamics in traditional
diffusion and flow-based generative models and instead learns the
equilibrium gradient of an implicit energy landscape. Through this approach, we can adopt an optimization-based sampling process at inference time, where samples are obtained by gradient descent on the learned landscape with adjustable step sizes, adaptive optimizers, and adaptive compute. Empirically, EqM achieves the best FID among the tested methods, reaching 1.90 on ImageNet 256$\times$256, while remaining competitive on other metrics. Theoretically, we show under standard assumptions that real images are attracting critical points of the learned field and the energy interpretation applies when the field is conservative. Beyond generation, EqM is a flexible framework that naturally handles tasks including partially noised image denoising, OOD detection, and image composition. By replacing time-conditional velocities with a unified
equilibrium landscape, EqM offers a tighter bridge between flow and energy-based models and a simple route to optimization-driven inference.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Continual Personalization for Diffusion Models
Yu-Chien Liao, Jr-Jen Chen, Chi-Pin Huang et al. (+3 more)
Updating
diffusion models in an incremental setting would be practical in real-world applications yet computationally challenging. We present a novel learning strategy of Concept
Neuron Selection (CNS), a simple yet effective approach to perform personalization in a continual learning scheme. CNS uniquely identifies
neurons in
diffusion models that are closely related to the target concepts. In order to mitigate catastrophic forgetting problems while preserving zero-shot text-to-image generation ability, CNS finetunes concept
neurons in an incremental manner and jointly preserves knowledge learned of previous concepts. Evaluation of real-world
datasets demonstrates that CNS achieves state-of-the-art performance with minimal parameter adjustments, outperforming previous methods in both single and multi-concept personalization works. CNS also achieves fusion-free operation, reducing
memory storage and processing time for continual personalization.
Read the full abstract ▾
All authors Yu-Chien Liao, Jr-Jen Chen, Chi-Pin Huang, Ci-Siang Lin, Meng-Lin Wu, Yu-Chiang Frank Wang
Computer Vision
VideoNSA: Native Sparse Attention Scales Video Understanding
Enxin Song, Wenhao Chai, Shusheng Yang et al. (+5 more)
Video understanding in
multimodal language models remains limited by context length: models often miss key transition frames and struggle to maintain coherence across long time scales. To address this, we adapt Native Sparse
Attention (NSA) to
video-
language models. Our method, VideoNSA, adapts Qwen2.5-VL through end-to-end training on a 216K
video instruction
dataset. We employ a hardware-aware hybrid approach to
attention, preserving dense
attention for text, while employing NSA for
video. Compared to
token-compression and training-free sparse baselines, VideoNSA achieves improved performance on long-
video understanding, temporal
reasoning, and spatial
benchmarks. Further ablation analysis reveals four key findings: (1) reliable scaling to 128K
tokens; (2) an optimal global-local
attention allocation at a fixed budget; (3) task-dependent branch usage patterns; and (4) the learnable combined sparse
attention help induce dynamic
attention sinks. Project Page: https://enxinsong.com/VideoNSA-web/, Code: https://github.com/Espere-1119-Song/VideoNSA
Read the full abstract ▾
All authors Enxin Song, Wenhao Chai, Shusheng Yang, Ethan Armand, Xiaojun Shan, Haiyang Xu, Jianwen Xie, Zhuowen Tu
Computer Vision · Cross-listed from cs.CL
From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens
Hala Sheta, Eric Huang, Shuyu Wu et al. (+9 more)
We introduce VLM-Lens, a toolkit designed to enable systematic benchmarking, analysis, and interpretation of
vision-
language models (VLMs) by supporting the extraction of intermediate outputs from any layer during the forward pass of open-source VLMs. VLM-Lens provides a unified, YAML-configurable interface that abstracts away model-specific complexities and supports user-friendly operation across diverse VLMs. It currently supports 16 state-of-the-art base VLMs and their over 30 variants, and is extensible to accommodate new models without changing the core logic. The toolkit integrates easily with various interpretability and analysis methods. We demonstrate its usage with two simple analytical experiments, revealing systematic differences in the hidden representations of VLMs across layers and target concepts. VLM-Lens is released as an open-sourced project to accelerate community efforts in understanding and improving VLMs.
Read the full abstract ▾
All authors Hala Sheta, Eric Huang, Shuyu Wu, Ilia Alenabi, Jiajun Hong, Ryker Lin, Ruoxi Ning, Daniel Wei, Jialin Yang, Jiawei Zhou, Ziqiao Ma, Freda Shi
Computer Vision · Cross-listed from cs.LG
Test-Time Anchoring for Discrete Diffusion Posterior Sampling
Litu Rout, Andreas Lugmayr, Yasamin Jafarian et al. (+4 more)
While continuous
diffusion models have achieved remarkable success, discrete
diffusion offers a unified framework for jointly modeling text and images. Beyond unification, discrete
diffusion provides faster inference, finer control, and principled training-free guidance, making it well-suited for posterior sampling. Existing approaches to posterior sampling using discrete
diffusion face severe challenges: derivative-free guidance yields sparse signals, continuous relaxations limit applicability, and split Gibbs samplers suffer from the curse of dimensionality. To overcome these limitations, we introduce Anchored Posterior Sampling (APS), built on two key innovations: quantized expectation for gradient-like guidance in discrete
embedding space, and anchored remasking for adaptive decoding. APS achieves state-of-the-art performance among discrete
diffusion samplers on both linear and nonlinear inverse problems across the standard image
benchmarks. We demonstrate the generality of APS through training-free stylization and text-guided editing. We further apply APS to a large-scale
diffusion language model, showing consistent improvement in question answering.
Read the full abstract ▾
All authors Litu Rout, Andreas Lugmayr, Yasamin Jafarian, Srivatsan Varadharajan, Constantine Caramanis, Sanjay Shakkottai, Ira Kemelmacher-Shlizerman
Computer Vision
MultiModal Action Conditioned Video Generation
Yichen Li and Antonio Torralba
Current
video models fail as world model as they lack fine-graiend control. General-purpose household
robots require real-time fine motor control to handle delicate tasks and urgent situations. In this work, we introduce fine-grained
multimodal actions to capture such precise control. We consider senses of proprioception, kinesthesia, force haptics, and muscle activation. Such
multimodal senses naturally enables fine-grained interactions that are difficult to simulate with text-conditioned generative models. To effectively simulate fine-grained multisensory actions, we develop a feature learning paradigm that aligns these modalities while preserving the unique information each modality provides. We further propose a regularization scheme to enhance causality of the action trajectory features in representing intricate interaction dynamics. Experiments show that incorporating
multimodal senses improves simulation accuracy and reduces temporal drift. Extensive ablation studies and downstream applications demonstrate the effectiveness and practicality of our work.
Read the full abstract ▾
Computer Vision
Learning to Generate Rigid Body Interactions with Video Diffusion Models
David Romero, Ariana Bermudez, Viacheslav Iablochnikov et al. (+3 more)
Recent
video generation models have achieved remarkable progress and are now deployed in film, social media production, and advertising. Beyond their creative potential, such models also hold promise as world simulators for robotics and
embodied decision making. Despite strong advances, current approaches still struggle to generate physically plausible object interactions and lack object-level control mechanisms. To address these limitations, we introduce KineMask, an approach for
video generation that enables realistic rigid body control, interactions, and effects. Given a single image and a specified object velocity, our method generates
videos with inferred motions and future object interactions. We propose a two-stage training strategy that gradually removes future motion supervision via object masks. Using this strategy we train
video diffusion models (VDMs) on synthetic scenes of simple interactions and demonstrate significant improvements and generalization to rigid body and hand-object interactions in real scenes. Furthermore, KineMask integrates low-level motion control with high-level textual conditioning via predicted scene descriptions, leading to support for synthesis of complex dynamical phenomena. Our experiments show that KineMask generalizes to different VDMs and achieves strong improvements over recent models of comparable size. Ablation studies further highlight the complementary roles of low- and high-level conditioning in VDMs.
Read the full abstract ▾
All authors David Romero, Ariana Bermudez, Viacheslav Iablochnikov, Hao Li, Fabio Pizzati, Ivan Laptev
Computer Vision
Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
Justin Cui, Jie Wu, Ming Li et al. (+6 more)
Diffusion models have revolutionized image and
video generation, achieving unprecedented visual quality. However, their reliance on
transformer architectures incurs prohibitively high computational costs, particularly when extending generation to long
videos. Recent work has explored autoregressive formulations for long
video generation, typically by distilling from short-horizon bidirectional teachers. Nevertheless, given that teacher models cannot synthesize long
videos, the extrapolation of student models beyond their training horizon often leads to pronounced quality degradation, arising from the compounding of errors within the continuous latent space. In this paper, we propose a simple yet effective approach to mitigate quality degradation in long-horizon
video generation without requiring supervision from long-
video teachers or retraining on long
video datasets. Our approach centers on exploiting the rich knowledge of teacher models to provide guidance for the student model through sampled segments drawn from self-generated long
videos. Our method maintains temporal consistency while scaling
video length by up to 20x beyond teacher's capability, avoiding common issues such as over-exposure and error-accumulation without recomputing overlapping frames like previous methods. When scaling up the computation, our method shows the capability of generating
videos up to 4 minutes and 15 seconds, equivalent to 99.9% of the maximum span supported by our base model's position
embedding and more than 50x longer than that of our baseline model. Experiments on standard
benchmarks and our proposed improved
benchmark demonstrate that our approach substantially outperforms baseline methods in both fidelity and consistency. Our long-horizon
videos demo can be found at https://self-forcing-plus-plus.github.io/
Read the full abstract ▾
All authors Justin Cui, Jie Wu, Ming Li, Tao Yang, Xiaojie Li, Rui Wang, Andrew Bai, Yuanhao Ban, Cho-Jui Hsieh
Computer Vision
VidGuard-R1: AI-Generated Video Detection and Explanation via Reasoning MLLMs and RL
Kyoungjun Park, Yifan Yang, Juheon Yi et al. (+6 more)
The rapid proliferation of AI-generated
video necessitates robust detection tools that offer both high accuracy and human-interpretable explanations. While existing MLLM-based detectors rely on supervised fine-tuning (SFT) or direct preference optimization (DPO), these methods are often bottlenecked by static, pre-labeled
datasets that fail to capture the evolving, multi-step physical inconsistencies of modern generative models. To bridge this gap, we introduce VidGuard-R1, the first
video authenticity detector to utilize group relative policy optimization (GRPO). Moving beyond passive preference matching, VidGuard-R1 employs a reinforcement learning framework that encourages the model to explore and rank multiple
reasoning paths. By introducing specialized reward models for temporal stability and
diffusion-aware complexity, we incentivize the model to discover 'physics-grounded' artifacts. Our contributions include: (1) a curated
dataset of 140,000 challenging real/fake
video pairs; (2) a GRPO-based training paradigm that achieves state-of-the-art zero-shot performance; and (3) a
reasoning-first architecture that provides precise, verifiable rationales for its forensic judgments. Project website: https://vidguard-r1.github.io/.
Read the full abstract ▾
All authors Kyoungjun Park, Yifan Yang, Juheon Yi, Shicheng Zheng, Yifei Shen, Dongqi Han, Caihua Shan, Muhammad Muaz, Lili Qiu
Computer Vision
microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification
Sathira Silva, Eman Ali, Chetan Arora et al. (+1 more)
Unsupervised adaptation of CLIP-based
vision-
language models (VLMs) for fine-grained image classification requires sensitivity to microscopic local cues. While CLIP exhibits strong zero-shot transfer, its reliance on coarse global features restricts its performance on fine-grained classification tasks. Prior efforts inject fine-grained knowledge by aligning large
language model (LLM) descriptions with the CLIP $\texttt{[CLS]}$
token; however, this approach overlooks spatial precision. We propose $\textbf{microCLIP}$, a self-training framework that jointly refines CLIP's visual and textual representations using fine-grained cues. At its core is Saliency-Oriented
Attention Pooling (SOAP) within a lightweight TokenFusion module, which builds a saliency-guided $\texttt{[FG]}$
token from patch
embeddings and fuses it with the global $\texttt{[CLS]}$
token for coarse-fine
alignment. To stabilize adaptation, we introduce a two-headed LLM-derived classifier: a frozen classifier that, via multi-view
alignment, provides a stable text-based prior for pseudo-labeling, and a learnable classifier initialized from LLM descriptions and fine-tuned with TokenFusion. We further develop Dynamic Knowledge Aggregation, which convexly combines fixed LLM/CLIP priors with TokenFusion's evolving logits to iteratively refine pseudo-labels. Together, these components uncover latent fine-grained signals in CLIP, yielding a consistent $2.90\%$ average accuracy gain across 13 fine-grained
benchmarks while requiring only light adaptation. Our code is available at https://github.com/sathiiii/microCLIP.
Read the full abstract ▾
All authors Sathira Silva, Eman Ali, Chetan Arora, Muhammad Haris Khan
Computer Vision · Cross-listed from cs.RO
Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning
Tianchong Jiang, Jingtian Ji, Xiangshan Tan et al. (+4 more)
We study view-invariant imitation learning by explicitly conditioning policies on camera extrinsics. Using Plucker
embeddings of per-pixel rays, we show that conditioning on extrinsics significantly improves generalization across viewpoints for standard behavior cloning policies, including ACT,
Diffusion Policy, and SmolVLA. To evaluate policy robustness under realistic viewpoint shifts, we introduce six manipulation tasks in RoboSuite and ManiSkill that pair "fixed" and "randomized" scene variants, decoupling background cues from camera pose. Our analysis reveals that policies without extrinsics often infer camera pose using visual cues from static backgrounds in fixed scenes; this shortcut collapses when workspace geometry or camera placement shifts. Conditioning on extrinsics restores performance and yields robust RGB-only control without depth. We release the tasks, demonstrations, and code at https://ripl.github.io/know_your_camera/ .
Read the full abstract ▾
All authors Tianchong Jiang, Jingtian Ji, Xiangshan Tan, Jiading Fang, Anand Bhattad, Vitor Guizilini, Matthew R. Walter
Computer Vision
NeuroSwift: A Lightweight Cross-Subject Framework for fMRI Visual Reconstruction of Complex Scenes
Shiyi Zhang, Dong Liang and Yihang Zhou
Reconstructing visual information from
brain activity via computer
vision technology provides an intuitive understanding of visual neural mechanisms. Despite progress in decoding fMRI data with generative models, achieving accurate cross-subject reconstruction of visual stimuli remains challenging and computationally demanding. This difficulty arises from inter-subject variability in neural representations and the
brain's abstract encoding of core semantic features in complex visual inputs. To address these challenges, we propose NeuroSwift, which integrates complementary adapters via
diffusion: AutoKL for low-level features and CLIP for semantics. NeuroSwift's CLIP Adapter is trained on Stable
Diffusion generated images paired with COCO captions to emulate higher visual cortex encoding. For cross-subject generalization, we pretrain on one subject and then fine-tune only 17 percent of parameters (fully connected layers) for new subjects, while freezing other components. This enables state-of-the-art performance with only one hour of training per subject on lightweight GPUs (three RTX 4090), and it outperforms existing methods.
Read the full abstract ▾
Computer Vision
Paving the Way Towards Kinematic Assessment Using Monocular Video: A Preclinical Benchmark of State-of-the-Art Deep-Learning-Based 3D Human Pose Estimators Against Inertial Sensors in Daily Living Activities
Mario Medrano-Paredes, Carmen Fernández-González, Francisco-Javier Díaz-Pernas et al. (+3 more)
Advances in machine learning and wearable sensors offer new opportunities for capturing and analyzing human movement outside specialized laboratories. Accurate assessment of human movement under real-world conditions is essential for telemedicine, sports science, and rehabilitation. This preclinical
benchmark compares
monocular video-based 3D human pose estimation models with inertial measurement units (IMUs), leveraging the VIDIMU
dataset containing a total of 13 clinically relevant daily activities which were captured using both commodity
video cameras and five IMUs. During this initial study only healthy subjects were recorded, so results cannot be generalized to pathological cohorts. Joint angles derived from state-of-the-art deep learning frameworks (MotionAGFormer, MotionBERT, MMPose 2D-to-3D pose lifting, and NVIDIA BodyTrack) were evaluated against joint angles computed from IMU data using OpenSim inverse kinematics following the Human3.6M
dataset format with 17 keypoints. Among them, MotionAGFormer demonstrated superior performance, achieving the lowest overall RMSE ($9.27\deg \pm 4.80\deg$) and MAE ($7.86\deg \pm 4.18\deg$), as well as the highest Pearson correlation ($0.86 \pm 0.15$) and the highest coefficient of determination $R^{2}$ ($0.67 \pm 0.28$). The results reveal that both technologies are viable for out-of-the-lab kinematic assessment. However, they also highlight key trade-offs between
video- and sensor-based approaches including costs, accessibility, and precision. This study clarifies where off-the-shelf
video models already provide clinically promising kinematics in healthy adults and where they lag behind IMU-based estimates while establishing valuable guidelines for researchers and clinicians seeking to develop robust, cost-effective, and user-friendly solutions for telehealth and remote patient monitoring.
Read the full abstract ▾
All authors Mario Medrano-Paredes, Carmen Fernández-González, Francisco-Javier Díaz-Pernas, Hichem Saoudi, Javier González-Alonso, Mario Martínez-Zarzuela
Computer Vision
From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understanding
Guangyu Sun, Archit Singhal, Burak Uzkent et al. (+3 more)
Video Large
Language Models (VLMs) have achieved strong performance on various
vision-
language tasks, yet their practical use is limited by the massive number of visual
tokens produced from raw
video frames, which quickly exhausts the model's context window. Existing solutions mitigate this issue by selecting a sparse set of frames, but such frame-wise selection discards essential temporal dynamics in long-form
videos, leading to suboptimal
reasoning about motion and event continuity. In this work, we systematically examine the role of temporal information and show that extending selection from isolated key frames to temporally coherent key clips improves
video understanding. To maintain a fixed computational budget while accommodating the larger
token footprint of clips, we introduce frame resolution as a controllable factor in frame selection, enabling a trade-off between spatial resolution and clip length. Building on this idea, we propose an adaptive clip length module that dynamically balances these factors to ensure a constant
token count per
video. Experiments on three long-form
video benchmarks demonstrate that our training-free approach, F2C, outperforms uniform sampling by up to 8.1%, 5.6%, and 10.3% on
Video-MME, LongVideoBench, and MLVU, respectively. These results highlight the importance of preserving temporal coherence in frame selection and provide a practical pathway for scaling VLMs to real-world
video understanding applications. Project webpage is available at https://guangyusun.com/f2c .
Read the full abstract ▾
All authors Guangyu Sun, Archit Singhal, Burak Uzkent, Mubarak Shah, Chen Chen, Garin Kessler
Computer Vision
DragFlow: Unleashing DiT Priors with Region Based Supervision for Drag Editing
Zihan Zhou, Shilin Lu, Shuli Leng et al. (+4 more)
Drag-based image editing has long suffered from distortions in the target region, largely because the priors of earlier base models, Stable
Diffusion, are insufficient to project optimized latents back onto the natural image
manifold. With the shift from UNet-based DDPMs to more scalable DiT with flow matching (e.g., SD3.5, FLUX), generative priors have become significantly stronger, enabling advances across diverse editing tasks. However, drag-based editing has yet to benefit from these stronger priors. This work proposes the first framework to effectively harness FLUX's rich prior for drag-based editing, dubbed DragFlow, achieving substantial gains over baselines. We first show that directly applying point-based drag editing to DiTs performs poorly: unlike the highly compressed features of UNets, DiT features are insufficiently structured to provide reliable guidance for point-wise motion supervision. To overcome this limitation, DragFlow introduces a region-based editing paradigm, where affine transformations enable richer and more consistent feature supervision. Additionally, we integrate pretrained open-domain personalization adapters (e.g., IP-Adapter) to enhance subject consistency, while preserving background fidelity through gradient mask-based hard constraints.
Multimodal large
language models (MLLMs) are further employed to resolve task ambiguities. For evaluation, we curate a novel Region-based Dragging
benchmark (ReD Bench) featuring region-level dragging instructions. Extensive experiments on DragBench-DR and ReD Bench show that DragFlow surpasses both point-based and region-based baselines, setting a new state-of-the-art in drag-based image editing. Code and
dataset are available at https://github.com/Edennnnnnnnnn/DragFlow.
Read the full abstract ▾
All authors Zihan Zhou, Shilin Lu, Shuli Leng, Shaocong Zhang, Zhuming Lian, Xinlei Yu, Adams Wai-Kin Kong
Computer Vision · Cross-listed from cs.AI
Scaling Agents for Computer Use
Gonzalo Gonzalez-Pumariega, Vincent Tu, Chih-Lun Lee et al. (+3 more)
Computer-use
agents (CUAs) hold promise for automating everyday digital tasks, but their performance on long-horizon, complex problems remains unreliable. Single-rollout execution is brittle, with small errors compounding over time and leading to high variance in outcomes. While prior work has attempted to scale within a single rollout, such approaches have yielded limited gains. Scaling over multiple rollouts offers a more promising alternative but doing so effectively is challenging due to the difficulty of evaluating and selecting among long-horizon
agent behaviors. We introduce Behavior Judge (BJudge), which addresses this challenge by representing
agent executions as behavior narratives and comparing candidate behaviors at this level, substantially improving robustness and success rates. Using multiple rollouts, BJudge establishes a new state of the art (SoTA) in OSWorld at 72.6%, significantly outperforming prior methods and surpassing human-level performance at 72.36%, with comprehensive ablations validating key design choices. We further demonstrate strong generalization results to different operating systems on WindowsAgentArena and AndroidWorld. Crucially, our results highlight the strong effectiveness of scaling CUAs, when you do it right: effective scaling requires structured trajectory understanding and selection, and BJudge provides a practical framework to achieve this.
Read the full abstract ▾
All authors Gonzalo Gonzalez-Pumariega, Vincent Tu, Chih-Lun Lee, Jiachen Yang, Ang Li, Xin Eric Wang
Computer Vision
RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning
Sicheng Feng, Kaiwen Tuo, Song Wang et al. (+3 more)
Fine-grained visual
reasoning remains a core challenge for
multimodal large
language models (MLLMs). The recently introduced ReasonMap highlights this gap by showing that even advanced MLLMs struggle with spatial
reasoning in structured and information-rich settings such as transit maps, a task of clear practical and scientific importance. However, standard reinforcement learning (RL) on such tasks is impeded by sparse rewards and unstable optimization. To address this, we first construct ReasonMap-Plus, an extended
dataset that introduces dense reward signals through Visual Question Answering (VQA) tasks, enabling effective cold-start training of fine-grained visual understanding skills. Next, we propose RewardMap, a multi-stage RL framework designed to improve both visual understanding and
reasoning capabilities of MLLMs. RewardMap incorporates two key designs. First, we introduce a difficulty-aware reward design that incorporates detail rewards, directly tackling the sparse rewards while providing richer supervision. Second, we propose a multi-stage RL scheme that bootstraps training from simple perception to complex
reasoning tasks, offering a more effective cold-start strategy than conventional Supervised Fine-Tuning (SFT). Experiments on ReasonMap and ReasonMap-Plus demonstrate that each component of RewardMap contributes to consistent performance gains, while their combination yields the best results. Moreover, models trained with RewardMap achieve an average improvement of 3.47% across 6
benchmarks spanning spatial
reasoning, fine-grained visual
reasoning, and general tasks beyond transit maps, underscoring enhanced visual understanding and
reasoning capabilities.
Read the full abstract ▾
All authors Sicheng Feng, Kaiwen Tuo, Song Wang, Lingdong Kong, Jianke Zhu, Huan Wang
Computer Vision · Cross-listed from cs.AI
The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
Phuc Minh Nguyen, Chinh D. La, Duy M. H. Nguyen et al. (+3 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key method for improving Large
Language Models'
reasoning capabilities, yet recent evidence suggests it may paradoxically shrink the
reasoning boundary rather than expand it. This paper investigates the shrinkage issue of RLVR by analyzing its learning dynamics and reveals two critical phenomena that explain this failure. First, we expose negative interference in RLVR, where learning to solve certain training problems actively reduces the likelihood of correct solutions for others, leading to the decline of Pass@$k$ performance, or the probability of generating a correct solution within $k$ attempts. Second, we uncover the winner-take-all phenomenon: RLVR disproportionately reinforces problems with high likelihood, correct solutions, under the base model, while suppressing other initially low-likelihood ones. Through extensive theoretical and empirical analysis on multiple mathematical
reasoning benchmarks, we show that this effect arises from the inherent on-policy sampling in standard RL objectives, causing the model to converge toward narrow solution strategies. Based on these insights, we propose a simple yet effective data curation
algorithm that focuses RLVR learning on low-likelihood problems, achieving notable improvement in Pass@$k$ performance. Our code is available at https://github.com/mail-research/SELF-llm-interference.
Read the full abstract ▾
All authors Phuc Minh Nguyen, Chinh D. La, Duy M. H. Nguyen, Nitesh V. Chawla, Binh T. Nguyen, Khoa D. Doan
Computer Vision
TempoControl: Temporal Attention Guidance for Text-to-Video Models
Shira Schiber, Ofir Lindenbaum and Idan Schwartz
Recent advances in generative
video models have enabled the creation of high-quality
videos based on natural
language prompts. However, these models frequently lack fine-grained temporal control, meaning they do not allow users to specify when particular visual elements should appear within a generated sequence. In this work, we introduce TempoControl, a method that allows for temporal
alignment of visual concepts during inference, without requiring retraining or additional supervision. TempoControl utilizes cross-
attention maps, a key component of text-to-
video diffusion models, to guide the timing of concepts through a novel optimization approach. Our method steers
attention using three complementary principles: aligning its temporal pattern with a control signal (correlation), adjusting its strength where visibility is required (magnitude), and preserving semantic consistency (
entropy). TempoControl provides precise temporal control while maintaining high
video quality and diversity. We demonstrate its effectiveness across various applications, including temporal reordering of single and multiple objects, action timing, and audio-aligned
video generation. Project page: https://shira-schiber.github.io/TempoControl/.
Read the full abstract ▾