Computer Vision
PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models
Hisayuki Yokomizo, Taiki Miyanishi, Yan Gang et al. (+3 more)
Vision-
Language Models (VLMs) are increasingly applied to robotic perception and manipulation, yet their ability to infer physical properties required for manipulation remains limited. In particular, estimating the mass of real-world objects is essential for determining appropriate grasp force and ensuring safe interaction. However, current VLMs lack reliable mass
reasoning capabilities, and most existing
benchmarks do not explicitly evaluate physical quantity estimation under realistic sensing conditions. In this work, we propose PhysQuantAgent, a framework for real-world object mass estimation using VLMs, together with VisPhysQuant, a new
benchmark dataset for evaluation. VisPhysQuant consists of RGB-D
videos of real objects captured from multiple viewpoints, annotated with precise mass measurements. To improve estimation accuracy, we introduce three visual prompting methods that enhance the input image with object detection, scale estimation, and cross-sectional image generation to help the model comprehend the size and internal structure of the target object. Experiments show that visual prompting significantly improves mass estimation accuracy on real-world data, suggesting the efficacy of integrating spatial
reasoning with VLM knowledge for physical inference.
Read the full abstract ▾
All authors Hisayuki Yokomizo, Taiki Miyanishi, Yan Gang, Shuhei Kurita, Nakamasa Inoue, Yusuke Iwasawa
Computer Vision
WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation
Jisu Nam, Yicong Hong, Chun-Hao Paul Huang et al. (+9 more)
Recent advances in
video diffusion transformers have enabled interactive gaming world models that allow users to explore generated environments over extended horizons. However, existing approaches struggle with precise action control and long-horizon 3D consistency. Most prior works treat user actions as abstract conditioning signals, overlooking the fundamental
geometric coupling between actions and the 3D world, whereby actions induce relative camera motions that accumulate into a global camera pose within a 3D world. In this paper, we establish camera pose as a unifying
geometric representation to jointly ground immediate action control and long-term 3D consistency. First, we define a physics-based continuous action space and represent user inputs in the Lie
algebra to derive precise 6-DoF camera poses, which are injected into the generative model via a camera embedder to ensure accurate action
alignment. Second, we use global camera poses as spatial indices to retrieve relevant past observations, enabling geometrically consistent revisiting of locations during long-horizon
navigation. To support this research, we introduce a large-scale
dataset comprising 3,000 minutes of authentic human gameplay annotated with camera trajectories and textual descriptions. Extensive experiments show that our approach substantially outperforms state-of-the-art interactive gaming world models in action controllability, long-horizon visual quality, and 3D spatial consistency.
Read the full abstract ▾
All authors Jisu Nam, Yicong Hong, Chun-Hao Paul Huang, Feng Liu, JoungBin Lee, Jiyoung Kim, Siyoon Jin, Yunsung Lee, Jaeyoon Jung, Suhwan Choi, Seungryong Kim, Yang Zhou
Computer Vision
Ruisi Wang, Zhongang Cai, Fanyi Pu et al. (+11 more)
Recent advances in
video generation have revealed an unexpected phenomenon:
diffusion-based
video models exhibit non-trivial
reasoning capabilities. Prior work attributes this to a Chain-of-Frames (CoF) mechanism, where
reasoning is assumed to unfold sequentially across
video frames. In this work, we challenge this assumption and uncover a fundamentally different mechanism. We show that
reasoning in
video models instead primarily emerges along the
diffusion denoising steps. Through qualitative analysis and targeted probing experiments, we find that models explore multiple candidate solutions in early denoising steps and progressively converge to a final answer, a process we term Chain-of-Steps (CoS). Beyond this core mechanism, we identify several emergent
reasoning behaviors critical to model performance: (1) working
memory that supports tasks requiring consistent reference, such as object permanence; (2) self-correction and enhancement, allowing recovery from incorrect intermediate solutions; and (3) perception before action, where early steps establish semantic grounding and later steps perform structured manipulation. Moreover, analysis of
Diffusion Transformer layers shows that middle layers conduct key
reasoning procedures. Motivated by these insights, we present a simple Training-Free Ensemble (TFE) as a proof-of-concept, demonstrating how
reasoning can be improved by ensembling latent trajectories from identical models with different random seeds. Overall, our work provides the first systematic dissection of the mechanisms underlying
video reasoning, offering a foundation to guide future research in better exploiting the inherent
reasoning dynamics of
video models as a new substrate for intelligence.
Read the full abstract ▾
All authors Ruisi Wang, Zhongang Cai, Fanyi Pu, Junxiang Xu, Wanqi Yin, Maijunxian Wang, Ran Ji, Chenyang Gu, Bo Li, Ziqi Huang, Hokin Deng, Dahua Lin, Ziwei Liu, Lei Yang
Computer Vision
SegviGen: Repurposing 3D Generative Model for Part Segmentation
Lin Li, Haoran Feng, Zehuan Huang et al. (+8 more)
We introduce SegviGen, a framework that repurposes native 3D generative models for 3D part
segmentation. Existing pipelines either lift strong 2D priors into 3D via
distillation or multi-view mask aggregation, often suffering from cross-view inconsistency and blurred boundaries, or explore native 3D discriminative
segmentation, which typically requires large-scale annotated 3D data and substantial training resources. In contrast, SegviGen leverages the structured priors encoded in pretrained 3D generative model to induce
segmentation through distinctive part colorization, establishing a novel and efficient framework for part
segmentation. Specifically, SegviGen encodes a 3D asset and predicts part-indicative colors on active voxels of a geometry-aligned reconstruction. It supports interactive part
segmentation, full
segmentation, and full
segmentation with 2D guidance in a unified framework. Extensive experiments show that SegviGen improves over the prior state of the art by 40% on interactive part
segmentation and by 15% on full
segmentation, while using only 0.32% of the labeled training data. It demonstrates that pretrained 3D generative priors transfer effectively to 3D part
segmentation, enabling strong performance with limited supervision. See our project page at https://fenghora.github.io/SegviGen-Page/.
Read the full abstract ▾
All authors Lin Li, Haoran Feng, Zehuan Huang, Haohua Chen, Wenbo Nie, Shaohua Hou, Keqing Fan, Pan Hu, Sheng Wang, Buyu Li, Lu Sheng
Computer Vision
MessyKitchens: Contact-rich object-level 3D scene reconstruction
Junaid Ahmed Ansari, Ran Ding, Fabio Pizzati et al. (+1 more)
Monocular 3D scene reconstruction has recently seen significant progress. Powered by the modern neural architectures and large-scale data, recent methods achieve high performance in depth estimation from a single image. Meanwhile, reconstructing and decomposing common scenes into individual 3D objects remains a hard challenge due to the large variety of objects, frequent occlusions and complex object relations. Notably, beyond shape and pose estimation of individual objects, applications in robotics and animation require physically-plausible scene reconstruction where objects obey physical principles of non-penetration and realistic contacts. In this work we advance object-level scene reconstruction along two directions. First, we introduceMessyKitchens, a new
dataset with real-world scenes featuring cluttered environments and providing high-fidelity object-level ground truth in terms of 3D object shapes, poses and accurate object contacts. Second, we build on the recent SAM 3D approach for single-object reconstruction and extend it with Multi-Object Decoder (MOD) for joint object-level scene reconstruction. To validate our contributions, we demonstrate MessyKitchens to significantly improve previous
datasets in registration accuracy and inter-object penetration. We also compare our multi-object reconstruction approach on three
datasets and demonstrate consistent and significant improvements of MOD over the state of the art. Our new
benchmark, code and pre-trained models will become publicly available on our project website: https://messykitchens.github.io/.
Read the full abstract ▾
All authors Junaid Ahmed Ansari, Ran Ding, Fabio Pizzati, Ivan Laptev
Computer Vision
SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation
Jiongze Yu, Xiangbo Gao, Pooja Verlani et al. (+4 more)
Video Super-Resolution (VSR) aims to restore high-quality
video frames from low-resolution (LR) estimates, yet most existing VSR approaches behave like black boxes at inference time: users cannot reliably correct unexpected artifacts, but instead can only accept whatever the model produces. In this paper, we propose a novel interactive VSR framework dubbed SparkVSR that makes sparse keyframes a simple and expressive control signal. Specifically, users can first super-resolve or optionally a small set of keyframes using any off-the-shelf image super-resolution (ISR) model, then SparkVSR propagates the keyframe priors to the entire
video sequence while remaining grounded by the original LR
video motion. Concretely, we introduce a keyframe-conditioned latent-pixel two-stage training pipeline that fuses LR
video latents with sparsely encoded HR keyframe latents to learn robust cross-space propagation and refine perceptual details. At inference time, SparkVSR supports flexible keyframe selection (manual specification, codec I-frame extraction, or random sampling) and a reference-free guidance mechanism that continuously balances keyframe adherence and blind restoration, ensuring robust performance even when reference keyframes are absent or imperfect. Experiments on multiple VSR
benchmarks demonstrate improved temporal consistency and strong restoration quality, surpassing baselines by up to 24.6%, 21.8%, and 5.6% on CLIP-IQA, DOVER, and MUSIQ, respectively, enabling controllable, keyframe-driven
video super-resolution. Moreover, we demonstrate that SparkVSR is a generic interactive, keyframe-conditioned
video processing framework as it can be applied out of the box to unseen tasks such as old-film restoration and
video style transfer. Our project page is available at: https://sparkvsr.github.io/
Read the full abstract ▾
All authors Jiongze Yu, Xiangbo Gao, Pooja Verlani, Akshay Gadde, Yilin Wang, Balu Adsumilli, Zhengzhong Tu
Computer Vision
SOMA: Unifying Parametric Human Body Models
Jun Saito, Jiefeng Li, Michael de Ruyter et al. (+12 more)
Parametric human body models are foundational to human reconstruction, animation, and simulation, yet they remain mutually incompatible: SMPL, SMPL-X, MHR, Anny, and related models each diverge in mesh
topology, skeletal structure, shape parameterization, and unit convention, making it impractical to exploit their complementary strengths within a single pipeline. We present SOMA, a unified body layer that bridges these heterogeneous representations through three abstraction layers. Mesh
topology abstraction maps any source model's identity to a shared canonical mesh in constant time per vertex. Skeletal abstraction recovers a full set of identity-adapted joint transforms from any body shape, whether in rest pose or an arbitrary posed configuration, in a single closed-form pass, with no iterative optimization or per-model training. Pose abstraction inverts the skinning pipeline to recover unified skeleton rotations directly from posed vertices of any supported model, enabling heterogeneous motion
datasets to be consumed without custom retargeting. Together, these layers reduce the $O(M^2)$ per-pair adapter problem to $O(M)$ single-backend connectors, letting practitioners freely mix identity sources and pose data at inference time. The entire pipeline is fully differentiable end-to-end and GPU-accelerated via NVIDIA-Warp.
Read the full abstract ▾
All authors Jun Saito, Jiefeng Li, Michael de Ruyter, Miguel Guerrero, Edy Lim, Ehsan Hassani, Roger Blanco Ribera, Hyejin Moon, Magdalena Dadela, Marco Di Lucca, Qiao Wang, Xueting Li, Jan Kautz, Simon Yuen, Umar Iqbal
Computer Vision
M^3: Dense Matching Meets Multi-View Foundation Models for Monocular Gaussian Splatting SLAM
Kerui Ren, Guanghao Li, Changjian Jiang et al. (+7 more)
Streaming reconstruction from uncalibrated
monocular video remains challenging, as it requires both high-precision pose estimation and computationally efficient online refinement in dynamic environments. While coupling 3D foundation models with SLAM frameworks is a promising paradigm, a critical bottleneck persists: most multi-view foundation models estimate poses in a feed-forward manner, yielding pixel-level correspondences that lack the requisite precision for rigorous
geometric optimization. To address this, we present M^3, which augments the Multi-view foundation model with a dedicated Matching head to facilitate fine-grained dense correspondences and integrates it into a robust
Monocular Gaussian Splatting SLAM. M^3 further enhances tracking stability by incorporating dynamic area suppression and cross-inference intrinsic
alignment. Extensive experiments on diverse indoor and outdoor
benchmarks demonstrate state-of-the-art accuracy in both pose estimation and scene reconstruction. Notably, M^3 reduces ATE RMSE by 64.3% compared to VGGT-SLAM 2.0 and outperforms ARTDECO by 2.11 dB in PSNR on the ScanNet++
dataset.
Read the full abstract ▾
All authors Kerui Ren, Guanghao Li, Changjian Jiang, Yingxiang Xu, Tao Lu, Linning Xu, Junting Dong, Jiangmiao Pang, Mulin Yu, Bo Dai
Computer Vision
What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers
Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss et al. (+3 more)
Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable
segmentation of complex microscopy images.
Read the full abstract ▾
All authors Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss, Anja Bielefeld, Samuel J. Cooper, Ronan Docherty
Computer Vision
An assessment of data-centric methods for label noise identification in remote sensing data sets
Felix Kröber, Genc Hoxha and Ribana Roscher
Label noise in the sense of incorrect labels is present in many real-world data sets and is known to severely limit the generalizability of deep learning models. In the field of remote sensing, however, automated treatment of label noise in data sets has received little
attention to date. In particular, there is a lack of systematic analysis of the performance of data-centric methods that not only cope with label noise but also explicitly identify and isolate noisy labels. In this paper, we examine three such methods and evaluate their behavior under different label noise assumptions. To do this, we inject different types of label noise with noise levels ranging from 10 to 70% into two
benchmark data sets, followed by an analysis of how well the selected methods filter the label noise and how this affects task performances. With our analyses, we clearly prove the value of data-centric methods for both parts - label noise identification and task performance improvements. Our analyses provide insights into which method is the best choice depending on the setting and objective. Finally, we show in which areas there is still a need for research in the transfer of data-centric label noise methods to remote sensing data. As such, our work is a step forward in bridging the methodological establishment of data-centric label noise methods and their usage in practical settings in the remote sensing domain.
Read the full abstract ▾
Computer Vision
Deep Reinforcement Learning-driven Edge Offloading for Latency-constrained XR pipelines
Sourya Saha and Saptarshi Debroy
Immersive extended reality (XR) applications introduce latency-critical workloads that must satisfy stringent real-time responsiveness while operating on energy- and battery-constrained devices, making execution placement between end devices and nearby edge servers a fundamental systems challenge. Existing approaches to adaptive execution and computation offloading typically optimize average performance metrics and do not fully capture the sustained interaction between real-time latency requirements and device battery lifetime in closed-loop XR workloads. In this paper, we present a battery-aware execution management framework for edge-assisted XR systems that jointly considers execution placement, workload quality, latency requirements, and battery dynamics. We design an online decision mechanism based on a lightweight deep reinforcement learning policy that continuously adapts execution decisions under dynamic network conditions while maintaining high motion-to-
photon latency compliance. Experimental results show that the proposed approach extends the projected device battery lifetime by up to 163% compared to latency-optimal local execution while maintaining over 90% motion-to-
photon latency compliance under stable network conditions. Such compliance does not fall below 80% even under significantly limited network bandwidth availability, thereby demonstrating the effectiveness of explicitly managing latency-energy trade-offs in immersive XR systems.
Read the full abstract ▾
Computer Vision
WildDepth: A Multimodal Dataset for 3D Wildlife Perception and Depth Estimation
Muhammad Aamir, Naoya Muramatsu, Sangyun Shin et al. (+6 more)
Depth estimation and 3D reconstruction have been extensively studied as core topics in computer
vision. Starting from rigid objects with relatively simple
geometric shapes, such as vehicles, the research has expanded to address general objects, including challenging deformable objects, such as humans and animals. However, for the animal, in particular, the majority of existing models are trained based on
datasets without metric scale, which can help validate image-only models. To address this limitation, we present WildDepth, a
multimodal dataset and
benchmark suite for depth estimation, behavior detection, and 3D reconstruction from diverse categories of animals ranging from domestic to wild environments with synchronized RGB and
LiDAR. Experimental results show that the use of multi-modal data improves depth reliability by up to 10% RMSE, while RGB-
LiDAR fusion enhances 3D reconstruction fidelity by 12% in Chamfer distance. By releasing WildDepth and its
benchmarks, we aim to foster robust
multimodal perception systems that generalize across domains.
Read the full abstract ▾
All authors Muhammad Aamir, Naoya Muramatsu, Sangyun Shin, Matthew Wijers, Jia-Xing Zhong, Xinyu Hou, Amir Patel, Andrew Loveridge, Andrew Markham
Computer Vision · Cross-listed from cs.LG
Adaptive Moments are Surprisingly Effective for Plug-and-Play Diffusion Sampling
Christian Belardi, Justin Lovelace, Kilian Q. Weinberger et al. (+1 more)
Guided
diffusion sampling relies on approximating often intractable likelihood scores, which introduces significant noise into the sampling dynamics. We propose using adaptive moment estimation to stabilize these noisy likelihood scores during sampling. Despite its simplicity, our approach achieves state-of-the-art results on image restoration and class-conditional generation tasks, outperforming more complicated methods, which are often computationally more expensive. We provide empirical analysis of our method on both synthetic and real data, demonstrating that mitigating gradient noise through adaptive moments offers an effective way to improve
alignment.
Read the full abstract ▾
All authors Christian Belardi, Justin Lovelace, Kilian Q. Weinberger, Carla P. Gomes
Computer Vision
V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising
Han Lin, Xichen Pan, Zun Wang et al. (+4 more)
Pixel-space
diffusion has recently re-emerged as a strong alternative to latent
diffusion, enabling high-quality generation without pretrained autoencoders. However, standard pixel-space
diffusion models receive relatively weak semantic supervision and are not explicitly designed to capture high-level visual structure. Recent representation-
alignment methods (e.g., REPA) suggest that pretrained visual features can substantially improve
diffusion training, and visual co-denoising has emerged as a promising direction for incorporating such features into the generative process. However, existing co-denoising approaches often entangle multiple design choices, making it unclear which are truly essential. We therefore present V-Co, a systematic study of visual co-denoising in a unified JiT-based framework. This controlled setting allows us to isolate the ingredients that make visual co-denoising effective. Our study reveals two main ingredients. First, co-denoising benefits from preserving feature-specific computation while enabling flexible cross-stream interaction, which leads to a fully dual-stream architecture together with a structurally defined unconditional prediction for classifier-free guidance. Second, it requires both stronger semantic supervision and proper cross-stream calibration, which we realize through a perceptual-drifting hybrid loss and RMS-based feature rescaling. Together, these findings yield a simple recipe for visual co-denoising. Experiments on ImageNet-256 show that, at comparable model sizes, V-Co outperforms the underlying pixel-space
diffusion baseline and strong prior pixel-
diffusion methods while using fewer training epochs, offering practical guidance for future representation-aligned generative models.
Read the full abstract ▾
All authors Han Lin, Xichen Pan, Zun Wang, Yue Zhang, Chu Wang, Jaemin Cho, Mohit Bansal
Computer Vision
IOSVLM: A 3D Vision-Language Model for Unified Dental Diagnosis from Intraoral Scans
Huimin Xiong, Zijie Meng, Tianxiang Hu et al. (+3 more)
3D intraoral scans (IOS) are increasingly adopted in routine dentistry due to abundant
geometric evidence, and unified multi-disease diagnosis is desirable for clinical documentation and communication. While recent works introduce dental
vision-
language models (VLMs) to enable unified diagnosis and report generation on 2D images or multi-view images rendered from IOS, they do not fully leverage native 3D geometry. Such work is necessary and also challenging, due to: (i) heterogeneous scan forms and the complex IOS
topology, (ii) multi-disease co-occurrence with class imbalance and fine-grained morphological ambiguity, (iii) limited paired 3D IOS-text data. Thus, we present IOSVLM, an end-to-end 3D VLM that represents scans as point clouds and follows a 3D encoder-projector-LLM design for unified diagnosis and generative visual question-answering (VQA), together with IOSVQA, a large-scale multi-source IOS diagnosis VQA
dataset comprising 19,002 cases and 249,055 VQA pairs over 23 oral diseases and heterogeneous scan types. To address the distribution gap between color-free IOS data and color-dependent 3D pre-training, we propose a geometry-to-chromatic proxy that stabilizes fine-grained
geometric perception and cross-modal
alignment. A two-stage curriculum training strategy further enhances robustness. IOSVLM consistently outperforms strong baselines, achieving gains of at least +9.58% macro accuracy and +1.46% macro F1, indicating the effectiveness of direct 3D geometry modeling for IOS-based diagnosis.
Read the full abstract ▾
All authors Huimin Xiong, Zijie Meng, Tianxiang Hu, Chenyi Zhou, Yang Feng, Zuozhu Liu
Computer Vision
Dual Stream Independence Decoupling for True Emotion Recognition under Masked Expressions
Jinsheng Wei, Xiguang Zhang, Zheng Shi et al. (+1 more)
Recongnizing true
emotions from masked expressions is extremely challenging due to deliberate concealment. Existing paradigms recognize true
emotions from masked-expression clips that contain onsetframes just starting to disguise. However, this paradigm may not reflect the actual disguised state, as the onsetframe leaks the true emotional information without reaching a stable disguise state. Thus, this paper introduces a novel apexframe-based paradigm that classifies true
emotions from the apexframe with a stable disguised state. Furthermore, this paper proposes a novel dual stream independence decoupling framework that decouples true and disguised
emotion features, avoiding the interference of disguised
emotions on true
emotions. For efficient decoupling, we design a decoupling loss group, comprising two classification losses that learn true
emotion and disguised expression features, respectively, and a
Hilbert-Schmidt Independence loss that enhances the independence of two features. Experiments demonstrate that the apexframe-based paradigm is challenging, and the proposed decouple framework improves recogntion performances.
Read the full abstract ▾
All authors Jinsheng Wei, Xiguang Zhang, Zheng Shi, Guanming Lu
Computer Vision
SuCor: Susceptibility Distortion Correction via Parameter-Free and Self-Regularized Optimal Transport
Sreekar Chigurupati and Eleftherios Garyfallidis
We present SuCor, a method for correcting susceptibility induced
geometric distortions in echo planar imaging (EPI) using optimal transport (OT) along the phase encoding direction. Given a pair of reversed phase encoding EPI volumes, we model each column of the distortion field as a Wasserstein-2 barycentric displacement between the opposing-polarity intensity profiles. Regularization is performed in the spectral domain using a bending-energy penalty whose strength is selected automatically via the Morozov discrepancy principle, requiring no manual tuning. On a human connectome project (HCP)
dataset with left-right/right-left b0 EPI pairs and a co-registered T1 structural reference, SuCor achieves a mean volumetric mutual information of 0.341 with the T1 image, compared to 0.317 for FSL TOPUP, while running in approximately 12 seconds on a single CPU core.
Read the full abstract ▾
Computer Vision
Semi-supervised Latent Disentangled Diffusion Model for Textile Pattern Generation
Chenggong Hu, Yi Wang, Mengqi Xue et al. (+3 more)
Textile pattern generation (TPG) aims to synthesize fine-grained textile pattern images based on given clothing images. Although previous studies have not explicitly investigated TPG, existing image-to-image models appear to be natural candidates for this task. However, when applied directly, these methods often produce unfaithful results, failing to preserve fine-grained details due to feature confusion between complex textile patterns and the inherent non-rigid texture distortions in clothing images. In this paper, we propose a novel method, SLDDM-TPG, for faithful and high-fidelity TPG. Our method consists of two stages: (1) a latent disentangled network (LDN) that resolves feature confusion in clothing representations and constructs a multi-dimensional, independent clothing feature space; and (2) a semi-supervised latent
diffusion model (S-LDM), which receives guidance signals from LDN and generates faithful results through semi-supervised
diffusion training, combined with our designed fine-grained
alignment strategy. Extensive evaluations show that SLDDM-TPG reduces FID by 4.1 and improves SSIM by up to 0.116 on our CTP-HD
dataset, and also demonstrate good generalization on the VITON-HD
dataset.
Read the full abstract ▾
All authors Chenggong Hu, Yi Wang, Mengqi Xue, Haofei Zhang, Jie Song, Li Sun
Computer Vision
When the City Teaches the Car: Label-Free 3D Perception from Infrastructure
Zhen Xu, Jinsu Yoo, Cristian Bautista et al. (+7 more)
Building robust 3D perception for self-driving still relies heavily on large-scale data collection and manual annotation, yet this paradigm becomes impractical as deployment expands across diverse cities and regions. Meanwhile, modern cities are increasingly instrumented with roadside units (RSUs), static sensors deployed along roads and at intersections to monitor traffic. This raises a natural question: can the city itself help train the vehicle? We propose infrastructure-taught, label-free 3D perception, a paradigm in which RSUs act as stationary, unsupervised teachers for ego vehicles. Leveraging their fixed viewpoints and repeated observations, RSUs learn local 3D detectors from unlabeled data and broadcast predictions to passing vehicles, which are aggregated as pseudo-label supervision for training a standalone ego detector. The resulting model requires no infrastructure or communication at test time. We instantiate this idea as a fully label-free three-stage pipeline and conduct a concept-and-feasibility study in a CARLA-based multi-
agent environment. With CenterPoint, our pipeline achieves 82.3% AP for detecting vehicles, compared to a fully supervised ego upper bound of 94.4%. We further systematically analyze each stage, evaluate its scalability, and demonstrate complementarity with existing ego-centric label-free methods. Together, these results suggest that city infrastructure itself can potentially provide a scalable supervisory signal for autonomous vehicles, positioning infrastructure-taught learning as a promising orthogonal paradigm for reducing annotation cost in 3D perception.
Read the full abstract ▾
All authors Zhen Xu, Jinsu Yoo, Cristian Bautista, Zanming Huang, Tai-Yu Pan, Zhenzhen Liu, Katie Z Luo, Mark Campbell, Bharath Hariharan, Wei-Lun Chao
Computer Vision
Retrieving Counterfactuals Improves Visual In-Context Learning
Guangzhi Xiong, Sanchit Sinha, Zhenghao He et al. (+1 more)
Vision-
language models (VLMs) have achieved impressive performance across a wide range of
multimodal reasoning tasks, but they often struggle to disentangle fine-grained visual attributes and reason about underlying causal relationships. In-context learning (ICL) offers a promising avenue for VLMs to adapt to new tasks, but its effectiveness critically depends on the selection of demonstration examples. Existing
retrieval-augmented approaches typically rely on passive similarity-based
retrieval, which tends to select correlated but non-causal examples, amplifying spurious associations and limiting model robustness. We introduce CIRCLES (Composed Image
Retrieval for Causal Learning Example Selection), a novel framework that actively constructs demonstration sets by retrieving counterfactual-style examples through targeted, attribute-guided composed image
retrieval. By incorporating counterfactual-style examples, CIRCLES enables VLMs to implicitly reason about the causal relations between attributes and outcomes, moving beyond superficial correlations and fostering more robust and grounded
reasoning. Comprehensive experiments on four diverse
datasets demonstrate that CIRCLES consistently outperforms existing methods across multiple architectures, especially on small-scale models, with pronounced gains under information scarcity. Furthermore, CIRCLES retrieves more diverse and causally informative examples, providing qualitative insights into how models leverage in-context demonstrations for improved
reasoning. Our code is available at https://github.com/gzxiong/CIRCLES.
Read the full abstract ▾
All authors Guangzhi Xiong, Sanchit Sinha, Zhenghao He, Aidong Zhang
Computer Vision
World Reconstruction From Inconsistent Views
Lukas Höllein and Matthias Nießner
Video diffusion models generate high-quality and diverse worlds; however, individual frames often lack 3D consistency across the output sequence, which makes the reconstruction of 3D worlds difficult. To this end, we propose a new method that handles these inconsistencies by non-rigidly aligning the
video frames into a globally-consistent coordinate frame that produces sharp and detailed pointcloud reconstructions. First, a
geometric foundation model lifts each frame into a pixel-wise 3D pointcloud, which contains unaligned surfaces due to these inconsistencies. We then propose a tailored non-rigid iterative frame-to-model ICP to obtain an initial
alignment across all frames, followed by a global optimization that further sharpens the pointcloud. Finally, we leverage this pointcloud as initialization for 3D reconstruction and propose a novel inverse deformation rendering loss to create high quality and explorable 3D environments from inconsistent views. We demonstrate that our 3D scenes achieve higher quality than baselines, effectively turning
video models into 3D-consistent world generators.
Read the full abstract ▾
Computer Vision
Emotion-Aware Classroom Quality Assessment Leveraging IoT-Based Real-Time Student Monitoring
Hai Nguyen, Hieu Dao, Hung Nguyen et al. (+2 more)
This study presents high-throughput, real-time multi-
agent affective computing framework designed to enhance classroom learning through emotional state monitoring. As large classroom sizes and limited teacher student interaction increasingly challenge educators, there is a growing need for scalable, data-driven tools capable of capturing students' emotional and engagement patterns in real time. The system was evaluated using the Classroom
Emotion Dataset, consisting of 1,500 labeled images and 300 classroom detection
videos. Tailored for IoT devices, the system addresses load balancing and latency challenges through efficient real-time processing. Field testing was conducted across three educational institutions in a large metropolitan area: a primary school (hereafter school A), a secondary school (school B), and a high school (school C). The system demonstrated robust performance, detecting up to 50 faces at 25 FPS and achieving 88% overall accuracy in classifying classroom engagement states. Implementation results showed positive outcomes, with favorable feedback from students, teachers, and parents regarding improved classroom interaction and teaching adaptation. Key contributions of this research include establishing a practical, IoT-based framework for
emotion-aware learning environments and introducing the 'Classroom
Emotion Dataset' to facilitate further validation and research.
Read the full abstract ▾
All authors Hai Nguyen, Hieu Dao, Hung Nguyen, Nam Vu, Cong Tran
Computer Vision
Search2Motion: Training-Free Object-Level Motion Control via Attention-Consensus Search
Sainan Liu, Tz-Ying Wu, Hector A Valdez et al. (+1 more)
We present Search2Motion, a training-free framework for object-level motion editing in image-to-
video generation. Unlike prior methods requiring trajectories, bounding boxes, masks, or motion fields, Search2Motion adopts target-frame-based control, leveraging first-last-frame motion priors to realize object relocation while preserving scene stability without fine-tuning. Reliable target-frame construction is achieved through semantic-guided object insertion and robust background inpainting. We further show that early-step self-
attention maps predict object and camera dynamics, offering interpretable user feedback and motivating ACE-Seed (
Attention Consensus for Early-step Seed selection), a lightweight search strategy that improves motion fidelity without look-ahead sampling or external evaluators. Noting that existing
benchmarks conflate object and camera motion, we introduce S2M-DAVIS and S2M-OMB for stable-camera, object-only evaluation, alongside FLF2V-obj metrics that isolate object artifacts without requiring ground-truth trajectories. Search2Motion consistently outperforms baselines on FLF2V-obj and VBench.
Read the full abstract ▾
All authors Sainan Liu, Tz-Ying Wu, Hector A Valdez, Subarna Tripathi
Computer Vision · Cross-listed from cs.RO
vAccSOL: Efficient and Transparent AI Vision Offloading for Mobile Robots
Adam Zahir, Michele Gucciardom Falk Selker, Anastasios Nanos et al. (+4 more)
Mobile
robots are increasingly deployed for inspection, patrol, and search-and-rescue operations, relying on computer
vision for perception,
navigation, and autonomous decision-making. However, executing modern
vision workloads onboard is challenging due to limited compute resources and strict energy constraints. While some platforms include embedded accelerators, these are typically tied to proprietary software stacks, leaving user-defined workloads to run on resource-constrained companion computers. We present vAccSOL, a framework for efficient and transparent execution of AI-based
vision workloads across heterogeneous robotic and edge platforms. vAccSOL integrates two components: SOL, a neural network compiler that generates optimized inference libraries with minimal runtime dependencies, and vAccel, a lightweight execution framework that transparently dispatches inference locally on the
robot or to nearby edge infrastructure. This combination enables hardware-optimized inference and flexible execution placement without requiring modifications to
robot applications. We evaluate vAccSOL on a real-world testbed with a commercial quadruped
robot and twelve deep learning models covering image classification,
video classification, and semantic
segmentation. Compared to a PyTorch compiler baseline, SOL achieves comparable or better inference performance. With edge offloading, vAccSOL reduces
robot-side power consumption by up to 80% and edge-side power by up to 60% compared to PyTorch, while increasing
vision pipeline frame rate by up to 24x, extending the operating lifetime of battery-powered
robots.
Read the full abstract ▾
All authors Adam Zahir, Michele Gucciardom Falk Selker, Anastasios Nanos, Kostis Papazafeiropoulos, Carlos J. Bernardos, Nicolas Weber, Roberto Gonzalez
Computer Vision
HMAR: Hierarchical Modality-Aware Expert and Dynamic Routing Medical Image Retrieval Architecture
Aojie Yuan
Medical image
retrieval (MIR) is a critical component of computer-aided diagnosis, yet existing systems suffer from three persistent limitations: uniform feature encoding that fails to account for the varying clinical importance of anatomical structures, ambiguous similarity metrics based on coarse classification labels, and an exclusive focus on global image similarity that cannot meet the clinical demand for fine-grained region-specific
retrieval. We propose HMAR (Hierarchical Modality-Aware Expert and Dynamic Routing), an adaptive
retrieval framework built on a Mixture-of-Experts (MoE) architecture. HMAR employs a dual-expert mechanism: Expert0 extracts global features for holistic similarity matching, while Expert1 learns position-invariant local representations for precise lesion-region
retrieval. A two-stage contrastive learning strategy eliminates the need for expensive bounding-box annotations, and a sliding-window matching
algorithm enables dense local comparison at inference time. Hash codes are generated via Kolmogorov-Arnold Network (KAN) layers for efficient Hamming-distance search. Experiments on the RadioImageNet-CT
dataset (16 clinical patterns, 29,903 images) show that HMAR achieves mean Average Precision (mAP) of 0.711 and 0.724 for 64-bit and 128-bit hash codes, improving over the state-of-the-art ACIR method by 0.7% and 1.1%, respectively.
Read the full abstract ▾
Computer Vision
$x^2$-Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space
Ruishan Guo, Ciyu Ruan, Haoyang Wang et al. (+3 more)
Estimating dense 2D optical flow and 3D scene flow is essential for dynamic scene understanding. Recent work combines images,
LiDAR, and event data to jointly predict 2D and 3D motion, yet most approaches operate in separate heterogeneous feature spaces. Without a shared latent space that all modalities can align to, these systems rely on multiple modality-specific blocks, leaving cross-sensor mismatches unresolved and making fusion unnecessarily complex.Event cameras naturally provide a spatiotemporal edge signal, which we can treat as an intrinsic edge field to anchor a unified latent representation, termed the Event Edge Space. Building on this idea, we introduce $x^2$-Fusion, which reframes
multimodal fusion as representation unification: event-derived spatiotemporal edges define an edge-centric homogeneous space, and image and
LiDAR features are explicitly aligned in this shared representation.Within this space, we perform reliability-aware adaptive fusion to estimate modality reliability and emphasize stable cues under degradation. We further employ cross-dimension contrast learning to tightly couple 2D optical flow with 3D scene flow. Extensive experiments on both synthetic and real
benchmarks show that $x^2$-Fusion achieves state-of-the-art accuracy under standard conditions and delivers substantial improvements in challenging scenarios.
Read the full abstract ▾
All authors Ruishan Guo, Ciyu Ruan, Haoyang Wang, Zihang Gong, Jingao Xu, Xinlei Chen
Computer Vision · Cross-listed from cs.RO
Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation
Mutian Xu, Tianbao Zhang, Tianqi Liu et al. (+3 more)
Simulating
robot-world interactions is a cornerstone of
Embodied AI. Recently, a few works have shown promise in leveraging
video generations to transcend the rigid visual/physical constraints of traditional simulators. However, they primarily operate in 2D space or are guided by static environmental cues, ignoring the fundamental reality that
robot-world interactions are inherently 4D spatiotemporal events that require precise interactive modeling. To restore this 4D essence while ensuring the precise
robot control, we introduce Kinema4D, a new action-conditioned 4D generative robotic simulator that disentangles the
robot-world interaction into: i) Precise 4D representation of
robot controls: we drive a URDF-based 3D
robot via kinematics, producing a precise 4D
robot control trajectory. ii) Generative 4D modeling of environmental reactions: we project the 4D
robot trajectory into a pointmap as a spatiotemporal visual signal, controlling the generative model to synthesize complex environments' reactive dynamics into synchronized RGB/pointmap sequences. To facilitate training, we curated a large-scale
dataset called Robo4D-200k, comprising 201,426
robot interaction episodes with high-quality 4D annotations. Extensive experiments demonstrate that our method effectively simulates physically-plausible, geometry-consistent, and embodiment-agnostic interactions that faithfully mirror diverse real-world dynamics. For the first time, it shows potential zero-shot transfer capability, providing a high-fidelity foundation for advancing next-generation
embodied simulation.
Read the full abstract ▾
All authors Mutian Xu, Tianbao Zhang, Tianqi Liu, Zhaoxi Chen, Xiaoguang Han, Ziwei Liu
Computer Vision
Fast-WAM: Do World Action Models Need Test-time Future Imagination?
Tianyuan Yuan, Zibin Dong, Yicheng Liu et al. (+1 more)
World Action Models (WAMs) have emerged as a promising alternative to
Vision-
Language-Action (VLA) models for
embodied control because they explicitly model how visual observations may evolve under action. Most existing WAMs follow an imagine-then-execute paradigm, incurring substantial test-time latency from iterative
video denoising, yet it remains unclear whether explicit future imagination is actually necessary for strong action performance. In this paper, we ask whether WAMs need explicit future imagination at test time, or whether their benefit comes primarily from
video modeling during training. We disentangle the role of
video modeling during training from explicit future generation during inference by proposing Fast-WAM, a WAM architecture that retains
video co-training during training but skips future prediction at test time. We further instantiate several Fast-WAM variants to enable a controlled comparison of these two factors. Across these variants, we find that Fast-WAM remains competitive with imagine-then-execute variants, while removing
video co-training causes a much larger performance drop. Empirically, Fast-WAM achieves competitive results with state-of-the-art methods both on simulation
benchmarks (LIBERO and RoboTwin) and real-world tasks, without
embodied pretraining. It runs in real time with 190ms latency, over 4$\times$ faster than existing imagine-then-execute WAMs. These results suggest that the main value of
video prediction in WAMs may lie in improving world representations during training rather than generating future observations at test time. Project page: https://yuantianyuan01.github.io/FastWAM/
Read the full abstract ▾
All authors Tianyuan Yuan, Zibin Dong, Yicheng Liu, Hang Zhao
Computer Vision
Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation
Jiawei Mao, Hardy Chen, Haoqin Tu et al. (+7 more)
Large
vision-
language models (LVLMs) have become increasingly strong but remain prone to
hallucinations in
multimodal tasks, which significantly narrows their deployment. As training these LVLMs to avoid
hallucinations becomes prohibitively expensive for larger models, training-free methods offer a cheap and flexible solution to this problem, yet existing approaches based on decoding or tool use often bring limited gains and/or weak interpretability. We propose Kestrel, a training-free framework for LVLM
hallucination mitigation that combines an explicit visual-grounding
agent with evidence-verified self-refinement mechanism. In detail, Kestrel first collects explicit visual evidence and converts tool outputs into reusable and structured textual evidence. Second, to take full advantage of these evidence, Kestrel verifies them via an LVLM judge for evidence checking, then iteratively self-refine answers based on verified evidence to reduce the risk of over-correction. Extensive experiments show that Kestrel improves performance over strong baselines across
hallucination benchmarks (e.g., average +3.31% on POPE and +28.34 on MME-
Hallucination with Qwen3-VL), while providing transparent verification traces for
hallucination diagnosis and analysis – e.g., both the integrated self-refinement module and grounding
agent contributing an average +2.0% gain on POPE.
Read the full abstract ▾
All authors Jiawei Mao, Hardy Chen, Haoqin Tu, Yuhan Wang, Letian Zhang, Zeyu Zheng, Huaxiu Yao, Zirui Wang, Cihang Xie, Yuyin Zhou
Computer Vision
Spectral Property-Driven Data Augmentation for Hyperspectral Single-Source Domain Generalization
Taiqin Chen, Yifeng Wang, Xiaochen Feng et al. (+4 more)
While hyperspectral images (HSI) benefit from numerous spectral channels that provide rich information for classification, the increased dimensionality and sensor variability make them more sensitive to distributional discrepancies across domains, which in turn can affect classification performance. To tackle this issue, hyperspectral single-source domain generalization (SDG) typically employs data augmentation to simulate potential domain shifts and enhance model robustness under the condition of single-source domain training data availability. However, blind augmentation may produce samples misaligned with real-world scenarios, while excessive emphasis on realism can suppress diversity, highlighting a tradeoff between realism and diversity that limits generalization to target domains. To address this challenge, we propose a spectral property-driven data augmentation (SPDDA) that explicitly accounts for the inherent properties of HSI, namely the device-dependent variation in the number of spectral channels and the mixing of adjacent channels. Specifically, SPDDA employs a spectral diversity module that resamples data from the source domain along the spectral dimension to generate samples with varying spectral channels, and constructs a channel-wise adaptive spectral mixer by modeling inter-channel similarity, thereby avoiding fixed augmentation patterns. To further enhance the realism of the augmented samples, we propose a spatial-spectral co-optimization mechanism, which jointly optimizes a spatial fidelity constraint and a spectral continuity self-constraint. Moreover, the weight of the spectral self-constraint is adaptively adjusted based on the spatial counterpart, thus preventing over-smoothing in the spectral dimension and preserving spatial structure. Extensive experiments conducted on three remote sensing
benchmarks demonstrate that SPDDA outperforms state-of-the-art methods.
Read the full abstract ▾
All authors Taiqin Chen, Yifeng Wang, Xiaochen Feng, Zhilin Zhu, Hao Sha, Yingjian Li, Yongbing Zhang