Computer Vision
HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration
Jiaxin Li, Yuxiang Wu, Zhenkai Zhang et al. (+11 more)
Extracting dynamic 4D object interactions from massive, in-the-wild
monocular videos offers a highly efficient data collection pathway for scaling
Embodied AI and training VLAs. However, existing
monocular 4D reconstruction methods primarily focus on isolated objects, often failing under the severe occlusions and complex dynamics inherent in multi-object interactions. To bridge this gap, we propose HAT-4D, the first agentic framework designed to reconstruct the 3D geometry, temporal dynamics, and physical interactions of multiple objects from a single
video. By integrating VLMs with a multi-level human-in-the-loop feedback mechanism, HAT-4D efficiently resolves depth ambiguities and interaction-induced occlusions during 3D generation and 4D propagation, yielding physically plausible assets without relying on expensive multicamera rigs. As a scalable data engine, HAT-4D facilitates the creation of MVOIK-4D, an open-world
benchmark for
monocular 4D interaction reconstruction, accompanied by a novel multi-dimensional evaluation protocol focused on physical plausibility and temporal consistency. Extensive experiments demonstrate that HAT-4D achieves SOTA performance on most evaluation metrics, while maintaining competitive semantic
alignment. Ablation studies show that introducing a small amount of human feedback improves interaction reconstruction. Moreover, the data produced by HAT-4D effectively improves baseline performance when used for fine-tuning. Our data and code are available at https://lijiaxin0111.github.io/HAT4D/
Read the full abstract ▾
All authors Jiaxin Li, Yuxiang Wu, Zhenkai Zhang, Xinrui Shi, Haoyuan Wang, Yichen Zhao, Su Linxiang, Chenyang Yu, Mingyu Zhang, Yifan Ding, Boran Wen, Li Zhang, Ruiyang Liu, Yong-Lu Li
Computer Vision · Cross-listed from cs.LG
LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behavior
Qinhong Zhou, Chuang Gan and Anoop Cherian
Embodied agents operating in decentralized and partially observable environments have attracted growing
attention in recent years. However, existing large
language model (LLM)-based
agents often exhibit behaviors that are misaligned with their partners or inconsistent with the environment state, leading to inefficient cooperation and poor task success. To address this challenge, we propose a novel framework, Learning Laws of Cooperation (LLawCo), that enables
embodied agents to autonomously align with both their partners and task objectives. Our framework allows
agents to reflect on past failures to extract misaligned behavioral patterns, which are used to derive high-level behavioral laws, such as "Talk when necessary" and "Wait for partner." These laws are explicitly incorporated into the
agents' chains of thought via supervised fine-tuning, aligning their
reasoning with task requirements and the behavior of other
agents. To evaluate our approach, we introduce PARTNR-Dialog, a large-scale multi-
agent communicative and cooperative planning
benchmark built on the PARTNR environment. Experiments on existing tasks and our new
benchmark demonstrate significant improvements in cooperative efficiency and task success rates. Across four backbone LLMs, our method achieves average success rate improvements of 4.5% on the PARTNR-Dialog
benchmark and 6.8% on the TDW-MAT
benchmark over state-of-the-art open-source communicative
agent frameworks. See the LLawCo project page for details: https://www.merl.com/research/highlights/LLawCo
Read the full abstract ▾
Computer Vision
EchoSonar-R: A Multi-View Reasoning-Enabled Model for Disease Classification and Report Generation in Echocardiography
Darya Taratynova, Ahmed Aly, Numan Saeed et al. (+1 more)
Echocardiography is the most widely used non-invasive cardiac imaging modality, providing essential information for cardiovascular diagnosis. Interpreting an echocardiogram requires synthesizing complementary evidence across multiple heart views to identify abnormalities and produce structured clinical reports. While recent efforts focus on improving classification performance, most models lack explicit diagnostic
reasoning and spatially grounded anatomical evidence, limiting clinician trust. We present EchoSonar-R, a multi-view
reasoning-enabled
vision-
language model that jointly performs multi-label disease classification and report generation from echocardiography studies. EchoSonar-R combines a spatiotemporal
video encoder with a structure-aware cardiac detector that provides spatially grounded anatomical cues to improve interpretability and clinician trust during cross-view
reasoning. EchoSonar-R is trained in two stages: supervised fine-tuning (SFT) on
reasoning-annotated targets, followed by Group Relative Policy Optimization (GRPO) with task-specific rewards that jointly align classification and report generation within a unified reinforcement-learning framework. Across a private multi-view
dataset and two public
benchmarks, EchoSonar-R improves macro balanced accuracy by 17.1% on the private set and 6.1% on MIMICEchoQA over the strongest baseline, achieves a GREEN clinical faithfulness score of 0.800, and produces interpretable
reasoning traces grounded in multi-view visual evidence.
Read the full abstract ▾
All authors Darya Taratynova, Ahmed Aly, Numan Saeed, Mohammad Yaqub
Computer Vision · Cross-listed from eess.IV
Enhanced Neural Video Representation Compression Across Extreme Complexity and Quality Scales
Ho Man Kwan, Tianhao Peng, Fan Zhang et al. (+3 more)
Implicit neural representations (INRs) have recently emerged as a promising approach to
video compression, delivering competitive rate-distortion performance alongside rapid decoding. However, existing neural
video codecs struggle to balance complexity and scalability. Lightweight models often suffer from degraded compression performance when scaled to different bitrate/quality levels, whereas high-performance models exhibit limited scalability, as their model complexity typically increases with quality. This lack of a unified architecture capable of maintaining consistent complexity across a wide range of bitrates severely limits their diverse real-world deployment. To address these challenges, we introduce NVRC++, a novel INR-based
video codec that utilizes a lightweight INR with multiple high-resolution feature grids, providing high scalability at any given complexity level. This is paired with an optimization framework that enables efficient overfitting on high-resolution grids for long
video sequences, thereby exploiting spatio-temporal redundancies without prohibitive computational or
memory overhead. Additionally, an advanced
entropy model is designed for efficiently compressing the high-dimensional grid parameters. As a result, NVRC++ provides four complexity levels (from 7kMACs/pixel to 360kMACs/pixel), each spanning wide bitrate and quality ranges while supporting real-time decoding. The experimental results show that NVRC++ offers a much faster decoding speed (up to 8.5x) compared to the SOTA INR-based
video codec, NVRC, while delivering comparable performance.
Read the full abstract ▾
All authors Ho Man Kwan, Tianhao Peng, Fan Zhang, Mike Nilsson, Andrew Gower, David Bull
Computer Vision
Toward Robust In-Context Segmentation via Concept Guidance
Zhigang Chen, Xiawu Zheng and Rongrong Ji
In-context
segmentation (ICS) requires a model to segment target regions in a query image using only a few reference images and their corresponding masks, without updating any parameters. Despite recent progress, prior ICS studies have largely overlooked a critical aspect: system robustness, ie, whether the model can produce stable
segmentation results for the same query under different references. In this work, we revisit ICS from the robustness perspective and introduce a novel paradigm, Concept-Guided In-Context
Segmentation (CG-ICS), which performs
segmentation by extracting high-level semantic concepts from references rather than relying solely on low-level visual matching. Specifically, CG-ICS introduces a concept
reasoning module that uses an MLLM to propose candidates and a SAM3-driven scoring function with tree-search refinement to select reliable textual concepts, together with a parallel visual exemplar route that provides query-side spatial grounding via a simple context construction. Both the textual concept and the visual exemplar are then used to activate the
segmentation capability of a frozen SAM3 backbone. Extensive experiments on standard ICS
benchmarks demonstrate that CG-ICS not only achieves state-of-the-art accuracy but also substantially improves robustness, yielding a more reliable ICS system with significantly reduced variance across diverse reference choices. Code is available at https://github.com/Kakarot1103/CG-ICS.
Read the full abstract ▾
Computer Vision
Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading
Hong Li, Minqi Meng, Yanjun Liang et al. (+10 more)
Reconstructing high-fidelity, relightable 3D avatars from a single in-the-wild image is a challenging ill-posed problem, primarily hindered by the scarcity of high-quality PBR data and the complexity of disentangling illumination from intrinsic materials. In this paper, we present a data-efficient framework that leverages the robust priors of a unified pre-trained
diffusion backbone to sequentially address texture completion, delighting, and material decomposition. Unlike existing methods that rely on fragmented pipelines or extensive proprietary
datasets, we utilize cascaded Low-Rank Adaptations (LoRAs) to adapt the strong generative prior of the
diffusion model for each sub-task in UV space. Specifically, we first employ an Inpainting LoRA to complete missing UV textures caused by occlusion, leveraging the model's semantic understanding to generate semantically and photometrically coherent details. Subsequently, a Light-Homogenization LoRA and a novel Cross-Intrinsic
Attention mechanism are introduced to remove baked-in lighting and collaboratively synthesize pixel-aligned PBR maps (Albedo, Normal, Roughness, Specular, and Displacement). To ensure physical plausibility, we impose a UV-space differentiable BRDF shading loss during the decomposition stage, forcing the generative process to adhere to the rendering equation without the artifacts typical of rasterization-based supervision. Extensive experiments demonstrate that our method, trained on fewer than 100 real 3D scans, generates comprehensive, 4K-resolution PBR assets with superior realism and generalization compared to state-of-the-art methods, and all training code and model weights will be released upon acceptance.
Read the full abstract ▾
All authors Hong Li, Minqi Meng, Yanjun Liang, Chongjie Ye, Houyuan Chen, Weiqing Xiao, Xianda Guo, Guojun Lei, Xuhui Liu, Chaojie Yang, Yanlun Peng, Hao Zhao, Baochang Zhang
Computer Vision · Cross-listed from astro-ph.IM
Differentiable design of the PIAA-ZWFS: a flexible wavefront sensor that approaches the fundamental limit
A. K. Taras, S. Y. Haffert and L. Desdoigts
Extreme adaptive optics (AO) is necessary for high contrast astronomy at scales of the habitable zone of nearby systems. We seek to evaluate wavefront sensors that approach fundamental limits of wavefront sensing, enabling adaptive optics systems to run faster or on fainter targets. We present the phase-induced amplitude apodisation Zernike wavefront sensor (PIAA-ZWFS): an adaptation of the conventional Zernike wavefront sensor (ZWFS) that leverages lossless apodisation of the pupil to concentrate the starlight in the focal plane. We optimise and evaluate the sensor with a differentiable modelling framework, drawing on concepts from
Bayesian experimental design to minimise the variance of a maximum likelihood
estimator that uses the system in the high Strehl regime. Our architecture shows state-of-the-art performance in simulation for different apertures, bandwidths,
photon fluxes and source sizes, closing the gap to the fundamental limit by a factor 10 (2.5) compared to the conventional ZWFS (optimised ZWFS) in a typical
photon-limited case. For extended sources, we show that even an ideal point source sensor rapidly becomes sub-optimal, and our system outperforms it for
stellar diameters larger than 0.8{\lambda}/D. We verify that these gains do not come at the cost of dynamic range with either linear or non-linear reconstructors. Finally, we present a proof that there must be a trade-off between the information gained about amplitude and phase errors for any wavefront sensor. The PIAA-ZWFS is a viable wavefront sensor operating near the fundamental sensitivity limits.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots
Sijin Chen, Kaixuan Jiang, Haixin Shi et al. (+6 more)
We study whether we can learn novel manipulation skills from human actions to a bi-manual
robot with parallel grippers. Human action data is cheap, abundant, and diverse, making it one of the most promising resources for scaling up
robot learning. Yet transferring skills from humans to
robots remains hard: most prior work treats humans as just another bi-manual 6DoF embodiment, where hand-pose estimates are noisy and the contact patterns of human fingers differ fundamentally from those of a parallel gripper. We argue that learning rotation-inclusive action signals from human data is therefore sub-optimal, and instead propose a bridging action representation: the relative wrist translation within the initial head-camera frame, an action space shared by humans and
robots. To handle the potential absence of certain action components in different embodiments, we build a $\pi_0$-like
vision-
language-action model with interleaved action
tokens and
attention masking. On a suite of novel bi-manual manipulation tasks, our bridging action transfers human manipulation knowledge to
robots far more effectively than noisy 6DoF human actions and scales with the amount of human data.
Read the full abstract ▾
All authors Sijin Chen, Kaixuan Jiang, Haixin Shi, Yanhui Wang, Weiheng Zhong, Haosheng Li, Bo Jiang, Yuxiao Liu, Xihui Liu
Computer Vision
PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation
Peiwen Zhang, Yufan Deng, Shangkun Sun et al. (+11 more)
Video generation models have emerged as a promising paradigm for
embodied world simulation. However, both general-domain
video generators and
robot-specific data fine-tuned models can still produce physically implausible manipulations, including discontinuous motion trajectories and inconsistent
robot-object interactions, which limits their reliability as world simulators. Through extensive experiments, we find that such physical instability mainly arises from two factors: deformation of moving objects and implausible spatio-temporal correlations among interacting entities, particularly during contact. Building on this observation, we propose PhysisForcing, a scalable training framework that strengthens physical consistency by focusing supervision on physics-informative regions through joint optimization of pixel-level and semantic-level features. The framework consists of a pixel-level trajectory
alignment loss, which supervises DiT features using reference point trajectories, and a semantic-level relational
alignment loss, which aligns DiT features with inter-region relations extracted from a frozen
video understanding encoder. Extensive experiments on R-Bench, PAI-Bench, and EZS-Bench show that PhysisForcing consistently improves
embodied video generation over strong baselines, improving the Wan2.2-I2V-A14B and Cosmos3-Nano base models on R-Bench by 22.3% and 9.2% (7.1% and 3.7% over vanilla finetuning), with the Cosmos3-Nano variant attaining the best overall score. Beyond generation, as a world model under the WorldArena action-planner protocol it raises the closed-loop success rate from 16.0% to 24.0% and further improves downstream policy success, indicating that physically aligned
video models yield stronger representations for robotic manipulation.
Read the full abstract ▾
All authors Peiwen Zhang, Yufan Deng, Shangkun Sun, Juncheng Ma, Duomin Wang, Jonas Du, Zilin Pan, Ye Huang, Hao Liang, Songyan Huang, Ruihua Zhang, Enze Xie, Ming-Yu Liu, Daquan Zhou
Computer Vision · Cross-listed from cs.CE
Higher-Order Fourier Neural Operator: Explicit Mode Mixer for Nonlinear PDEs
Alex Colagrande, Paul Caillon, Eva Feillet et al. (+1 more)
Neural operators provide deep neural networks for learning mappings between function spaces. Among them, the
Fourier Neural Operator (FNO) is particularly effective: its spectral convolution relies on low-dimensional
Fourier-domain representations and can handle inputs at different resolutions. This design aligns well with settings where the
Fourier basis diagonalizes the underlying operator, such as linear, constant-coefficient PDEs on periodic domains, in which
Fourier modes evolve independently. However, nonlinear PDEs may benefit from an additional inductive bias, as they exhibit structured interactions between modes, governed by
polynomial nonlinearities. To capture this inductive bias, we introduce the Higher-Order Spectral Convolution, a spectral mixer that extends FNO from diagonal modulation to explicit n-linear mode mixing, aligned with the dynamics of nonlinear PDEs. Our experiments on standard
benchmarks show that the proposed Higher-Order FNO (HO-FNO) retains the efficiency of FNO-based architectures and consistently improves over other spectral neural operators. HO-FNO also performs on par with or better than state-of-the-art
transformers and state-space models on several
datasets, with stronger gains in highly nonlinear regimes, such as the Poisson equation with
polynomial forcing, where a single HO-FNO layer outperforms FNO models with up to 16 layers. We open-source our code for reproducibility at: https://github.com/AlexColagrande/HO-FNO.
Read the full abstract ▾
All authors Alex Colagrande, Paul Caillon, Eva Feillet, Alexandre Allauzen
Computer Vision
BiDeMem: Bidirectional Degradation Memory for Explainable Image Restoration
Xinrui Wu and Lichen Huang
Degradation-aware
prompts, conditions, and latent priors are increasingly used in image restoration, yet they are usually judged by a single endpoint: whether the restored image obtains higher PSNR. This is a weak test of semantics. A condition can help by adding capacity, acting as a global correction bias, or exploiting
dataset shortcuts, without becoming an interpretable degradation prior. We propose BiDeMem, a bidirectional degradation
memory for explainable image restoration. A query built from restoration features and input statistics retrieves a compact top-k subset of
memory slots. The same selected slot identity supports the restoration path at inference time and a training-only forward-degradation explanation path. The study centers on verifiability in a controlled multi-degradation NAFNet setting. New controls separate the gain from a correction head alone, a dense query prior, and a static global prior: these variants are 0.2588, 0.2586, and 0.2839 dB below BiRank, respectively. Strong residual supervision and a wider degradation head also remain below the full bidirectional
memory model. Intervention probes show that BiRank preserves restoration quality while increasing wrong-prior and native-prior sensitivity, framing degradation
memory as both a restoration module and a falsifiable explanation mechanism.
Read the full abstract ▾
Computer Vision
Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training
Francis Xiatian Zhang, Hao Yao, Shengxuan Chen et al. (+4 more)
Vision-based assessment can provide convenient and cost-effective evaluation in Traditional Chinese Medicine (TCM) rehabilitation training, where action quality assessment (AQA) from computer
vision offers a promising solution. Existing automatic AQA frameworks for physical therapy typically rely on skeletal data captured from a single viewpoint, which is inefficient for TCM techniques such as acupuncture or Tuina that involve dense hand self-occlusion and complex hand-object interactions. To address these challenges, we propose CME-AQA, a cross-view,
multimodal vision-based assessment framework that integrates visual-pose fusion to enhance understanding of environmental context and leverages both first-person and third-person
videos during training to improve inference robustness. We collected two dual-view
datasets, TCM-AQA61-A (Acupuncture) and TCM-AQA61-T (Tuina), each containing synchronized first-person and third-person recordings of 61 subjects with expert annotations. Experimental results show that our approach achieves superior or comparable mean performance against competitive baselines, achieving over 10% relative improvement in weighted F1 over the best competing method on key rating tasks such as Needle Depth and Quick Needle Insertion, while also reducing mean absolute error in quantitative measures such as insertion time and manipulation frequency. Testing on a CPR
dataset further demonstrates comparable performance on several posture-based criteria, suggesting applicability to related structured simulated clinical skill assessments where participant motion is central to evaluation. Overall, CME-AQA enhances assessment accuracy for structured TCM rehabilitation training and facilitates more convenient and effective training-oriented skill evaluation.
Read the full abstract ▾
All authors Francis Xiatian Zhang, Hao Yao, Shengxuan Chen, Hong Zhu, Hongxiao Jia, Sisi Zheng, Hubert P. H. Shum
Computer Vision
OSOR: One-Step Diffusion Inpainting for Effect-Aware Object Removal
Qinming Zhou, Chenxi Sun, Deyang Kong et al. (+6 more)
Real-world object removal is challenging due to two key difficulties: the target object's non-local effects, such as shadows and reflections, which are difficult to model, and the fact that user-provided masks are often inaccurate or incomplete. With billions of parameters and tens of denoising steps,
diffusion-based models achieve strong removal performance at the expense of substantial computational cost, limiting their use in interactive applications and on edge devices. To address these challenges, we present OSOR (One-Step Object Removal), which simultaneously achieves efficient, effect-aware, and mask-robust object removal. Concretely, OSOR introduces: (1) an occupancy-guided discriminator for precise boundary supervision, enabling stable single-step
diffusion training; (2) an alpha head that leverages knowledge from pretrained
diffusion models to predict appropriate removal regions with minimal overhead, thereby handling imperfect masks; and (3) a semantic-anchored verification pipeline (SAVP) that filters noisy instruction-based triplets to produce effect-aware supervision at scale. Using SAVP, we curate CORNE, which contains 280K verified removal pairs, and further annotate AnimeEraseBench and TextEraseBench to evaluate performance on more complex removal tasks. Experiments show that OSOR surpasses strong multi-step
diffusion baselines in perceptual quality while achieving $4\times$ to $30\times$ faster inference.
Read the full abstract ▾
All authors Qinming Zhou, Chenxi Sun, Deyang Kong, Junhao He, Xiangheng Tang, Peike Yu, Haotian Wu, Leilei Cao, Linfeng Zhang
Computer Vision
Diffusion Model Attribution via Spectral Coupling of Denoiser Responses
Pragati Shuddhodhan Meshram and Varun Chandrasekaran
Attributing a generated image to its source
diffusion model is a fundamental challenge in provenance verification and intellectual property protection. This problem is particularly difficult because
diffusion models trained on different
datasets can converge to similar score functions and thus similar output distributions, making the generated images themselves unreliable as attribution evidence. Existing non-invasive methods either fail on architecturally similar variants or rely on signals that vanish when models share the same autoencoder. We propose Spectral Denoising Signatures (SDS), a non-invasive attribution method that identifies the source model by fingerprinting each candidate model's denoising behavior. Our key insight is that a model's denoising score function exhibits a distinctive spectral geometry, reflected in how it redistributes energy across spatial frequency bands during denoising. By probing this behavior with frequency-controlled perturbations, SDS extracts a stable signature that is intrinsic to the model, requiring only standard forward passes with no inversion, optimization, or generation-time enrollment. Our results demonstrate that SDS achieves approximately 99.9% accuracy across eight diverse
diffusion models and 96.2% under cross-domain
prompt shift, outperforming non-invasive baselines across variations in training data, architecture, and training procedure, establishing spectral geometry as a principled and practical basis for
diffusion model attribution. Code is available at: https://github.com/Pragati-Meshram/SGS
Read the full abstract ▾
Computer Vision
RPM-Distill: Physiology-guided Adaptive Cross-modal Distillation for Robust Remote Physiological Measurement
Jiyao Wang, Qingyong Hu, Duoxun Tang et al. (+3 more)
Video-based remote physiological measurement (RPM) is highly accessible but remains fragile under varying illumination, skin tones, and motion.
Radio frequency (RF) radar is largely invariant to illumination and appearance, providing complementary cardio-respiratory micro-motion cues; however, requiring radar at inference is often impractical due to its limited ubiquity and deployment overhead. We propose RPM-Distill, a physiology-guided cross-modal
distillation framework that leverages synchronized radar only during training while retaining
video-only inference. Our key observation is that although RGB and RF waveforms differ in sensing physics and time-domain morphology, they share similar latent periodic rhythm in the frequency domain. We thus distill physiology-structured spectral evidence to improve robustness, via losses that (i) anchor the fundamental peak, (ii) match the off-peak background distribution, and (iii) preserve spectral morphology and sharpness. To avoid negative transfer under sample-level teacher quality and
alignment uncertainty, a spectral policy network predicts sample-level
distillation gates and component weights from the student–teacher spectral relation map, learned with a meta bilevel objective on a small labeled validation split. Through extensive experiments in challenging conditions and cross-
dataset settings, RPM-Distill brings 81% MAE and 21% correlation improvement over unimodal baselines. Code is at https://github.com/WJULYW/RPM-Distill.
Read the full abstract ▾
All authors Jiyao Wang, Qingyong Hu, Duoxun Tang, Xiao Yang, Kaishun Wu, Jiangbo Yu
Computer Vision
STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition
Nandani Sharma, Varun Sharma and Dinesh Singh
Micro-expression recognition is challenging due to subtle and short-lived facial muscle movements. Existing methods rely heavily on apex-onset frames, overlook fine-grained inter-frame dynamics, and separately model spatial and temporal information, limiting generalization across
datasets. To address these challenges, we propose STAG, a dynamic ROI-AU-coupled spatial-temporal network that jointly models motion flow and adaptive facial connectivity. The framework extracts optical flow from discriminative frames using magnitude-based selection and temporal
attention. A dual-branch architecture combines an enhanced graph
attention network for structured spatial
reasoning with a
transformer encoder for temporal modeling. A bidirectional cross-
attention module enables mutual refinement of spatial and temporal features, while AU-guided dynamic connectivity adapts facial region interactions according to muscle activation patterns. The
transformer captures subtle temporal dynamics beyond apex-based approaches, improving semantic consistency and interpretability for explainable micro-expression recognition. The fused representation is optimized using focal loss and evaluated on CASME II, 4DME, DFME, NaME, SAMM, and SMIC-HS. Extensive experiments demonstrate improved robustness, generalization, interpretability, and computational efficiency, confirming the effectiveness of adaptive relational
reasoning, AU-guided dynamic connectivity, and deep spatial-temporal feature fusion for accurate cross-
dataset micro-expression recognition.
Read the full abstract ▾
Computer Vision
TextDS: Parameter-Efficient Representation Alignment for Scene Text Detection under Distribution Shifts
Boyuan Chen, Zichen Dang, Chuang Yang et al. (+2 more)
In real-world deployments, scene text detectors inevitably face distribution shifts beyond the training distribution. Prior work often depends on large-scale scene-text pretraining, yet evaluation under cross-domain changes and real-world imaging degradations remains limited. We propose TextDS, an efficient framework for scene text detection under distribution shifts. First, we propose a data-efficient dual-encoder design with visual foundation models, eliminating the reliance on large-scale scene-text pretraining. Second, we introduce Step-wise LoRA adaptation (SWLoRA), which performs progressive low-rank refinement with a dynamic early-exit mechanism for effective feature adaptation. Third, we propose Common Subspace Fusion (CSF) to align and fuse the two branches in a shared subspace while retaining complementary, shift-robust information. Finally, we construct adverse-condition scene text detection
datasets to address the gap in evaluating under imaging degradation. Experiments show that TextDS achieves competitive performance in scene text detection, demonstrating robustness across domains and adverse imaging conditions with only 4.9M trainable parameters.
Read the full abstract ▾
All authors Boyuan Chen, Zichen Dang, Chuang Yang, Lap-Pui Chau, Yi Wang
Computer Vision
ReScene: Structured Indoor Scene Reconstruction from Multi-View Captures
Haoran Xu, Lechao Zhang, Daoguo Dong et al. (+2 more)
Constructing simulation-ready 3D scenes from multi-view captures is a key bottleneck for
Embodied Artificial Intelligence, as downstream tasks require object-level structure, explicit inter-object relations, and physical plausibility. Existing approaches either rely on specialized capture hardware, suffer from single-view bias in object reconstruction, or yield layouts that are geometrically reasonable but physically inconsistent. We identify that the problem is not single-object reconstruction but cross-view relation fusion and physically plausible scene assembly. To address this challenge, we present ReScene, a framework that threads multi-view geometry throughout the pipeline as a unifying prior. Our method consists of two main components: HierView prioritizes reconstruction views based on semantic consistency and 3D coverage completeness, replacing the largest-mask heuristic that conflates image occupancy with object coverage; and Relation-Aware Assembly fuses multi-frame relation predictions from a
vision-
language model with
geometric and room-shell priors into a confidence-weighted scene graph, enabling physically consistent scene assembly. ReScene sets a new state of the art across geometry, rendering, and perceptual quality on a set of ScanNet scenes, achieving a 17% reduction in Chamfer Distance and 26% in LPIPS over the strongest prior baseline, while running up to 10x faster than prior multi-view methods. Based on the reconstructed scenes, we also generate an
embodied visual question answering
dataset, on which fine-tuned Qwen-VL approaches the performance of strong closed-source models on several spatial
reasoning tasks.
Read the full abstract ▾
All authors Haoran Xu, Lechao Zhang, Daoguo Dong, Yan Gao, Xin Tan
Computer Vision
AirGroundBench: Probing Spatial Intelligence in Multimodal Large Models under Heterogeneous Multi-View Embodied Collaboration
Haotian Li, Yida Wang, Leyuan Wang et al. (+7 more)
In recent years,
multimodal large
language models (MLLMs) have shown strong potential for
embodied intelligence, yet their ability to maintain geometrically consistent spatial understanding across heterogeneous views remains under-evaluated. Existing
benchmarks largely focus on single-
agent, single-view perception, leaving a gap in the systematic assessment of collaborative air-ground settings, where multi-scale observations are complementary but introduce scale mismatch, asymmetric occlusion, and reference-frame inconsistencies. We present AirGroundBench, a diagnostic
benchmark for evaluating multi-view spatial intelligence in heterogeneous
UAV-UGV collaboration. AirGroundBench is built from 11 high-fidelity simulated environments with 1,021 synchronized air-ground observation pairs, yielding approximately 62,000 dual-view, four-option single-choice visual question answering instances and 115 closed-loop
vision-
language navigation episodes. It covers 10 task types organized into four progressively demanding capability dimensions: spatial perception, cross-view
alignment, spatial transformation and
reasoning, and
embodied decision-making. To support geometry-grounded evaluation and analysis, we provide structured spatial annotations, including cross-view object identities and metric 2D and 3D bounding boxes. Evaluations of 13 representative MLLMs under
UAV-only, UGV-only, and dual-view input settings reveal consistent bottlenecks: models perform relatively well on spatial perception but struggle with cross-view
alignment and transformation-intensive
reasoning, and these deficits propagate to sequential decision-making in
vision-
language navigation. Although dual-view inputs provide measurable gains over single-view variants, a persistent gap from human performance remains, highlighting
geometric consistency as a key limitation of current
embodied MLLMs.
Read the full abstract ▾
All authors Haotian Li, Yida Wang, Leyuan Wang, Jinshan Lai, Keyang Wang, Zonghao Guo, Qiang Ma, Liuyu Xiang, Jianwei Hu, Zhaofeng He
Computer Vision
Mind the Gap: Quantifying the Domain Gap in Cross-Sensor Diffusion Super-Resolution
Dawid Kopeć, Katarzyna Jabłońska, Wojciech Kozłowski et al. (+1 more)
Demand for high-resolution satellite imagery has increased interest in super-resolution (SR) to bridge the spatial resolution gap between freely available missions such as Sentinel-2 and commercial systems like PlanetScope. Because no sensor provides true paired low- and high-resolution observations, SR models are usually trained on synthetically degraded data, creating a domain gap on real cross-sensor imagery. In this work, we provide the first systematic study of how this synthetic-to-real mismatch affects the performance of modern
diffusion-based SR models. Using a large, geometrically and temporally aligned
dataset of Sentinel-2 and PlanetScope imagery, we evaluate five state-of-the-art
diffusion architectures under controlled experimental settings. We also introduce LPIPS-Sat, a domain-adapted perceptual metric based on Sentinel-2 self-supervised features. Our results show two persistent challenges: synthetically trained models degrade sharply on real pairs, while models trained on real cross-sensor data exhibit optimisation difficulties and struggle to adapt to the physical and radiometric diversity. These findings highlight a key limitation of current SR and motivate methods that disentangle super-resolution from domain adaptation.
Read the full abstract ▾
All authors Dawid Kopeć, Katarzyna Jabłońska, Wojciech Kozłowski, Maciej Zięba
Computer Vision · Cross-listed from eess.IV
MLVC: Multi-platform Learned Video Codec for Real-World Deployment
Tanel Pärnamaa, Martin Lumiste, Ardi Loot et al. (+3 more)
Neural
video codecs have surpassed classical codecs in coding efficiency but remain impractical for deployment due to cross-platform incompatibility and high computational cost. Existing quantization-based solutions fail to produce deterministic results across diverse hardware platforms, leading to catastrophic decoding failures. We introduce MLVC, a hardware-robust neural
video codec designed for practical cross-platform inference. The key idea is to explicitly transmit scale parameters through the hyperprior, which guarantees
entropy coding consistency across devices without requiring bit-exact arithmetic. While this increases bitrate overhead, we recover most of the coding efficiency through architectural improvements (gated
memory, ReGLU activation), a long-term reference recovery mechanism, and domain-specific perceptual training. On the VCD
video conferencing
benchmark, MLVC achieves >70% BD-rate (MOS) improvement over hardware HEVC, the strongest deployable baseline, while reaching subjective quality competitive with DCVC-RT, which cannot operate across diverse platforms. Both the encoder and decoder run at 100 FPS on average on commodity NPUs from Apple, Intel, and Qualcomm. MLVC is the first neural
video codec to combine competitive compression performance, real-time speed, and cross-platform robustness across diverse consumer devices, making it suitable for widespread deployment. Code is available at https://github.com/microsoft/mlvc.
Read the full abstract ▾
All authors Tanel Pärnamaa, Martin Lumiste, Ardi Loot, Evgenii Indenbom, Andrei Znobishchev, Ando Saabas
Computer Vision
EMOSH: Expressive Motion and Shape Disentanglement for Human Animation
Dongbin Zhang, Hao Liu, Binquan Dai et al. (+5 more)
High-fidelity and expressive controllable human animation is essential for content creation and digital avatar applications. However, existing methods face a dilemma between expressiveness and disentanglement. Mainstream 2D pose-conditioned approaches suffer from "motion-shape
entanglement", leading to the leakage of the driving subject's body shape. Conversely, methods relying on 3D priors (e.g., SMPL) achieve
geometric disentanglement but struggle to capture facial expressions and complex gestures, resulting in rigid animations. To this end, we propose EMOSH, a novel framework for high-fidelity controllable human
video generation. First, an Expressive Human Model (EHM) is introduced as the core control representation. By explicitly disentangling shape and pose parameters, we fundamentally resolve the body shape leakage issue. Alongside this, a robust motion tracker is designed to accurately estimate EHM parameters from
video. Second, we propose a Coarse-to-Fine Hybrid Motion Injection strategy, enabling more fine-grained control over expressions and gestures. Furthermore, we introduce a Spatially-Aligned Conditioning mechanism to bridge the domain gap between training and inference, improving identity consistency. Extensive experiments demonstrate that EMOSH outperforms previous methods in both self-driven and cross-driven scenarios, producing high-fidelity
videos with vivid expressions while maintaining shape disentanglement.
Read the full abstract ▾
All authors Dongbin Zhang, Hao Liu, Binquan Dai, Kangjie Chen, Chuming Wang, Chen Li, Jing Lyu, Haoqian Wang
Computer Vision
TempAct: Advancing Temporal Plausibility in Autoregressive Video Generation via Planner-Executor RL
Jing Wang, Xiangxin Zhou, Jiajun Liang et al. (+5 more)
Autoregressive (AR)
video diffusion models enable low-latency streaming generation by synthesizing
videos chunk by chunk with cached visual context, but this chunk-wise formulation makes temporal instruction following ambiguous. A single global
prompt does not specify which sub-event should be realized in each chunk, while naively switching to step-wise
prompts often leads to delayed reactions, blended step semantics, and error propagation across
prompt transitions. These failures are difficult to address with supervised fine-tuning or
distillation alone: SFT suffers from exposure bias, while rollout-based
distillation still optimizes low-level denoising or teacher-distribution matching rather than directly enforcing action ordering and
prompt-transition correctness. We address these challenges with TempAct, a planner–executor reinforcement learning framework that jointly optimizes temporal decomposition and step-conditioned execution for temporally plausible AR
video generation. TempAct uses an LLM planner to explore span-aware step
prompts that are executable by the
video model, and trains an AR
diffusion executor to follow these
prompts under its own generated histories. Its key mechanism is hierarchical group exploration: candidate plans form planning groups, and each plan induces an execution group of multiple continuations from a shared visual context, enabling plan-level credit assignment for long-horizon temporal outcomes and executor-level credit assignment for
prompt-switch behavior. We further design hierarchical rewards that combine plan-quality and full-
video temporal feedback for the planner with local transition-level step-following rewards, aesthetic regularization, and KL constraints for the executor. Experiments on Self-Forcing and LongLive show that TempAct improves temporal consistency while preserving overall visual quality.
Read the full abstract ▾
All authors Jing Wang, Xiangxin Zhou, Jiajun Liang, Kaiqi Liu, Wanyuan Pang, Zhenyu Xie, Tianyu Pang, Xiaodan Liang
Computer Vision
Curriculum-guided Change Detection Training: Toward Accurate Serac Fall Monitoring
Arthur Dérédel, Carlos Crispim-Junior, Pierre Lemaire et al. (+2 more)
Change Detection (CD) aims to identify semantic or structural changes from nearly registered multi-temporal images. While recent advances in training methodologies have largely focused on semi-supervised learning and consistency regularization, alternative training paradigms remain underexplored. In particular, most deep CD methods rely on uniform sampling during training, implicitly assuming that all training samples contribute equally to the optimization process. However, such naive sampling can introduce noisy gradients and hinder robust representation learning. To address this limitation, we propose a curriculum learning framework tailored for change detection. Our approach investigates two complementary difficulty measures: the
Solar Angular Gap (SAG), a physically grounded proxy for acquisition-condition variability, and the Structural Similarity Index Measure (SSIM), which evaluates appearance similarity between image pairs. Based on these criteria, the framework progressively introduces challenging samples during training, enabling models to learn robust representations in a coarse-to-fine manner. We evaluate our method on the challenging SeracFallDet
benchmark, where results demonstrate consistent improvements of the proposed approach over standard uniform-sampling strategies for both pixel-based and object-based approaches. These results highlight the potential of curriculum learning to improve robustness in deep change detection. Importantly, our training framework is orthogonal to existing CD architectures, making it readily applicable to a broad range of methods.
Read the full abstract ▾
All authors Arthur Dérédel, Carlos Crispim-Junior, Pierre Lemaire, Johan Berthet, Laure Tougne Rodet
Computer Vision
HumanMoveVQA: Can Video MLLMs reason about human movement in videos?
Pulkit Gera, Faegheh Sardari, Asmar Nadeem et al. (+4 more)
Despite the rapid advance of
Multimodal Large
Language Models (MLLMs) in high-level
video understanding, a fundamental bottleneck remains: these models collapse complex human motion into coarse semantic labels. Existing
benchmarks mostly focus on scene-centric events or local joint articulations, failing to probe global human motion in space over time (trajectory and orientation changes). We introduce HumanMoveVQA, the first comprehensive
benchmark designed to evaluate global trajectory and orientation
reasoning from an exocentric perspective. Our
benchmark utilizes a first-frame anchored world coordinate system, preserving translation and rotation relative to a fixed starting point. We propose a scalable, multi-stage pipeline that lifts 2D
video observations into world-consistent 3D motion tracks to generate over 10K structured question-answer pairs across seven
reasoning categories, including motion aggregation, sequential ordering, and trajectory-level inference. Our extensive evaluation reveals a critical capability gap in state-of-the-art proprietary models on deep human motion understanding. However, we demonstrate that this is a learnable problem; by fine-tuning an open-source baseline with our targeted, world-consistent supervision, we achieve a significant improvement. HumanMoveVQA establishes a rigorous
geometric foundation for developing next-generation, movement-aware
video understanding models.
Read the full abstract ▾
All authors Pulkit Gera, Faegheh Sardari, Asmar Nadeem, Valentina Bono, Padraig Boulton, Adrian Hilton, Armin Mustafa
Computer Vision
Latent Visual Diffusion Reasoning with Monte Carlo Tree Search
Xirui Teng, Nan Xi and Junsong Yuan
Analyzing fine-grained skill activities (e.g., sports, surgery) requires not only recognizing visual patterns but also performing step-by-step visual
reasoning that leads to the final judgment. While recent advances in action quality assessment have achieved remarkable progress in evaluating performance, existing models remain black boxes, where they lack the ability to explicitly reveal the
reasoning processes underlying their judgments. To address this limitation, we propose Latent Visual
Diffusion Reasoning (LVDR), a novel framework that integrates keypoint-guided Monte Carlo Tree Search (MCTS) to model and visualize the latent visual
reasoning process. LVDR not only produces more accurate skill assessments but also uncovers the critical visual
reasoning sequences that contribute to the final evaluation. Extensive experiments across four
datasets spanning diverse sports and
surgical domains demonstrate that LVDR achieves competitive quantitative performance while providing interpretable visual
reasoning trajectories leading to the final predictions. Source codes and models can be found through the following link: https://github.com/XiruiTeng/LVDR_Official.git.
Read the full abstract ▾
Computer Vision
Parallel Rollout Approximation for Pixel-Space Autoregressive Image Generation
Jiayi Xu, Di He and Guolin Ke
Pixel-space continuous-
token autoregressive (AR) generation directly models images as sequences of raw pixel patches, avoiding discrete tokenization or a separately pretrained tokenizer. However, it faces coupled challenges: high-dimensional patch generation causes large single-step errors, and teacher-forced training creates a train–inference gap that makes these errors accumulate across AR steps. Existing fixes such as $x$-prediction and input noise injection only partially mitigate these issues. Exact rollout training better matches inference-time conditions, but is impractical due to prohibitively slow sequential sampling. We propose Parallel Rollout Approximation (PRA), a scalable framework that addresses both challenges jointly. PRA generates low-dimensional intermediate states instead of high-dimensional pixel patches, then maps them back to pixel-space
tokens with a pixel decoder, preserving a pixel-in, pixel-out AR interface. It also constructs inference-like pixel inputs through the same intermediate-state-to-pixel path used at inference, independently across positions, approximating the pixel-feedback interface encountered during inference-time rollout while retaining parallel teacher-forced training. On class-conditional ImageNet-1K generation at $256\times256$ resolution, PRA-S with 135M parameters achieves an FID of 2.58, surpassing the previous billion-scale pixel-space AR result of 3.60. Scaling to PRA-L with 511M parameters further improves FID to 1.94, establishing a new state of the art among pixel-space AR models. Beyond generation, PRA achieves higher ImageNet classification probing accuracy than other AR and
diffusion baselines, suggesting its potential for unified pixel-space image generation and understanding.
Read the full abstract ▾
Computer Vision
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
ZhengXian Wu, Hangrui Xu, Kai Shi et al. (+8 more)
Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge. Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive during
reasoning. We propose ProMSA, a progressive
multimodal search
agent for KB-VQA. Given an image-question pair, the
agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant
retrieval. For training, we first use rejection-sampling SFT to learn valid tool-use formats, then optimize the
agent with TN-GSPO, a sequence-level RL objective that normalizes updates by both generation length and tool-interaction depth. Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and
agent baselines, and improved
retrieval and end-to-end accuracy. The code is available at https://github.com/DingWu1021/Promsa.
Read the full abstract ▾
All authors ZhengXian Wu, Hangrui Xu, Kai Shi, Zhuohong Chen, Yunyao Yu, Chuanrui Zhang, Zirui Liao, Jun Yang, Zhenyu Yang, Haonan Lu, Haoqian Wang
Computer Vision
Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control
Haoyuan Wang, Yabo Chen, Haibin Huang et al. (+2 more)
Building interactive world models requires generating realistic
videos while maintaining controllable dynamics over long horizons. Autoregressive
video generation offers a scalable foundation, but suffers from error accumulation and temporal degradation during extended rollouts. This issue is further amplified under heterogeneous controls such as human motion and camera trajectories, which may interfere and destabilize a pretrained
video prior, while existing methods often trade off controllability and visual quality. We propose "Directing the World", a fast autoregressive framework for controllable world-model
video generation with compositional human-motion and camera-trajectory control. Our key idea is to decouple control learning while preserving a unified autoregressive
video prior. We introduce a Fast-Slow
Memory training strategy to stabilize long-horizon rollout learning and improve convergence. For human motion control, we design a t-guided Dynamic Projection mechanism and a refined Motion-CFG strategy, enabling temporally smooth and accurate motion
alignment without degrading visual fidelity, and supporting multi-person control.After learning a robust motion prior, we introduce a second-stage camera-trajectory control module to compose human dynamics with viewpoint changes for coherent world exploration. We further construct a large-scale
dataset with synchronized
video, text, human-motion, and camera-trajectory annotations, organized into motion-centric and camera-centric subsets for decoupled training. Extensive experiments show stable long-horizon generation with precise controllability and high visual quality. See more at https://whydahuzi.github.io/Directing-the-World.github.io/.
Read the full abstract ▾
All authors Haoyuan Wang, Yabo Chen, Haibin Huang, Chi Zhang, Xuelong Li
Computer Vision
Understanding How MLLMs Describe Artworks Using Token Activation Maps
Nicola Fanelli, Pasquale De Marinis, Raffaele Scaringi et al. (+3 more)
Multimodal Large
Language Models (MLLMs) describe artworks with remarkable fluency, yet the visual
reasoning behind their outputs remains opaque. When an MLLM names a style, identifies a subject, or recognizes an iconographic symbol, does it ground each claim in the relevant region of the canvas, draw on an undifferentiated visual signal, or rely primarily on textual priors? We study this using the
Token Activation Map (TAM), which produces, for each generated
token, a heatmap isolating the visual evidence specific to that
token from prior-context interference. Applying TAM to a curated set of paintings spanning multiple periods and genres, we analyze grounding patterns across five semantically distinct
token categories: common visual objects, style descriptors, metadata, iconographic
tokens, and affective expressions. We find that visual grounding varies substantially with
token semantics. We further show that MLLMs attempt to identify artworks and artists, achieving higher accuracy in artist attribution than in title prediction, where
hallucinations are more frequent. Finally, we compare TAM with SAM~3 open-vocabulary
segmentation. To ensure reproducibility, we release our code, experimental configurations,
prompts, and qualitative results on the project page at https://nicolafan.github.io/tamart/.
Read the full abstract ▾
All authors Nicola Fanelli, Pasquale De Marinis, Raffaele Scaringi, Eva Cetinic, Gennaro Vessio, Giovanna Castellano