Computer Vision
Cascading multi-agent anomaly detection in surveillance systems via vision-language models and embedding-based classification
Tayyab Rehman, Giovanni De Gasperis and Aly Shmahell
Intelligent anomaly detection in dynamic visual environments requires reconciling real-time performance with semantic interpretability. Conventional approaches address only fragments of this challenge. Reconstruction-based models capture low-level deviations without contextual
reasoning, object detectors provide speed but limited semantics, and large
vision-
language systems deliver interpretability at prohibitive computational cost. This work introduces a cascading multi-
agent framework that unifies these complementary paradigms into a coherent and interpretable architecture. Early modules perform reconstruction-gated filtering and object-level assessment, while higher-level
reasoning agents are selectively invoked to interpret semantically ambiguous events. The system employs adaptive escalation thresholds and a publish-subscribe communication backbone, enabling asynchronous coordination and scalable deployment across heterogeneous hardware. Extensive evaluation on large-scale monitoring data demonstrates that the proposed cascade achieves a threefold reduction in latency compared to direct
vision-
language inference, while maintaining high perceptual fidelity (PSNR = 38.3 dB, SSIM = 0.965) and consistent semantic labeling. The framework advances beyond conventional detection pipelines by combining early-exit efficiency, adaptive multi-
agent reasoning, and explainable anomaly attribution, establishing a reproducible and energy-efficient foundation for scalable intelligent visual monitoring.
Read the full abstract ▾
Computer Vision
QwenStyle: Content-Preserving Style Transfer with Qwen-Image-Edit
Shiwen Zhang, Haibin Huang, Chi Zhang et al. (+1 more)
Content-Preserving Style transfer, given content and style references, remains challenging for
Diffusion Transformers (DiTs) due to its internal entangled content and style features. In this technical report, we propose the first content-preserving style transfer model trained on Qwen-Image-Edit, which activates Qwen-Image-Edit's strong content preservation and style customization capability. We collected and filtered high quality data of limited specific styles and synthesized triplets with thousands categories of style images in-the-wild. We introduce the Curriculum Continual Learning framework to train QwenStyle with such mixture of clean and noisy triplets, which enables QwenStyle to generalize to unseen styles without degradation of the precise content preservation capability. Our QwenStyle V1 achieves state-of-the-art performance in three core metrics: style similarity, content consistency, and aesthetic quality.
Read the full abstract ▾
All authors Shiwen Zhang, Haibin Huang, Chi Zhang, Xuelong Li
Computer Vision · Cross-listed from cs.CR
Leveraging Membership Inference Attacks for Privacy Measurement in Federated Learning for Remote Sensing Images
Anh-Kiet Duong, Petra Gomez-Krämer, Hoàng-Ân Lê et al. (+1 more)
Federated Learning (FL) enables collaborative model training while keeping training data localized, allowing us to preserve
privacy in various domains including remote sensing. However, recent studies show that FL models may still leak sensitive information through their outputs, motivating the need for rigorous
privacy evaluation. In this paper, we leverage membership inference attacks (MIA) as a quantitative
privacy measurement framework for FL applied to remote sensing image classification. We evaluate multiple black-box MIA techniques, including
entropy-based attacks, modified
entropy attacks, and the likelihood ratio attack, across different FL
algorithms and communication strategies. Experiments conducted on two public scene classification
datasets demonstrate that MIA effectively reveals
privacy leakage not captured by accuracy alone. Our results show that communication-efficient FL strategies reduce MIA success rates while maintaining competitive performance. These findings confirm MIA as a practical metric and highlight the importance of integrating
privacy measurement into FL system design for remote sensing applications.
Read the full abstract ▾
All authors Anh-Kiet Duong, Petra Gomez-Krämer, Hoàng-Ân Lê, Minh-Tan Pham
Computer Vision
How Does India Cook Biryani?
Shubham Goel, Farzana S, C V Rishi et al. (+2 more)
Biryani, one of India's most celebrated dishes, exhibits remarkable regional diversity in its preparation, ingredients, and presentation. With the growing availability of online cooking
videos, there is unprecedented potential to study such culinary variations using computational tools systematically. However, existing
video understanding methods fail to capture the fine-grained,
multimodal, and culturally grounded differences in procedural cooking
videos. This work presents the first large-scale, curated
dataset of biryani preparation
videos, comprising 120 high-quality YouTube recordings across 12 distinct regional styles. We propose a multi-stage framework leveraging recent advances in
vision-
language models (VLMs) to segment
videos into fine-grained procedural units and align them with audio transcripts and canonical recipe text. Building on these aligned representations, we introduce a
video comparison pipeline that automatically identifies and explains procedural differences between regional variants. We construct a comprehensive question-answer (QA)
benchmark spanning multiple
reasoning levels to evaluate procedural understanding in VLMs. Our approach employs multiple VLMs in complementary roles, incorporates human-in-the-loop verification for high-precision tasks, and
benchmarks several state-of-the-art models under zero-shot and fine-tuned settings. The resulting
dataset, comparison methodology, and QA
benchmark provide a new testbed for evaluating VLMs on structured,
multimodal reasoning tasks and open new directions for computational analysis of cultural heritage through cooking
videos. We release all data, code, and the project website at https://farzanashaju.github.io/how-does-india-cook-biryani/.
Read the full abstract ▾
All authors Shubham Goel, Farzana S, C V Rishi, Aditya Arun, C V Jawahar
Computer Vision
Thinking with Map: Reinforced Parallel Map-Augmented Agent for Geolocalization
Yuxiang Ji, Yong Wang, Ziyu Ma et al. (+6 more)
The image geolocalization task aims to predict the location where an image was taken anywhere on Earth using visual clues. Existing large
vision-
language model (LVLM) approaches leverage world knowledge, chain-of-thought
reasoning, and agentic capabilities, but overlook a common strategy used by humans – using maps. In this work, we first equip the model Thinking with Map ability and formulate it as an
agent-in-the-map loop. We develop a two-stage optimization scheme for it, including agentic reinforcement learning (RL) followed by parallel test-time scaling (TTS). The RL strengthens the agentic capability of model to improve sampling efficiency, and the parallel TTS enables the model to explore multiple candidate paths before making the final prediction, which is crucial for geolocalization. To evaluate our method on up-to-date and in-the-wild images, we further present MAPBench, a comprehensive geolocalization training and evaluation
benchmark composed entirely of real-world images. Experimental results show that our method outperforms existing open- and closed-source models on most metrics, specifically improving Acc@500m from 8.0% to 22.1% compared to Gemini-3-Pro with Google Search/Map grounded mode.
Read the full abstract ▾
All authors Yuxiang Ji, Yong Wang, Ziyu Ma, Yiming Hu, Hailang Huang, Xuecai Hu, Guanhua Chen, Liaoni Wu, Xiangxiang Chu
Computer Vision
Multi-task Cross-modal Learning for Chest X-ray Image Retrieval
Zhaohui Liang, Sivaramakrishnan Rajaraman, Niccolo Marini et al. (+2 more)
CLIP and BiomedCLIP are examples of
vision-
language foundation models and offer strong cross-modal
embeddings; however, they are not optimized for fine-grained medical
retrieval tasks, such as retrieving clinically relevant radiology reports using chest X-ray (CXR) image queries. To address this shortcoming, we propose a multi-task learning framework to fine-tune BiomedCLIP and evaluate improvements to CXR image-text
retrieval. Using BiomedCLIP as the backbone, we incorporate a lightweight MLP projector head trained with a multi-task composite loss function that includes: (1) a binary cross-
entropy loss to distinguish normal from abnormal CXR studies, (2) a supervised contrastive loss to reinforce intra-class consistency, and (3) a CLIP loss to maintain cross-modal
alignment. Experimental results demonstrate that the fine-tuned model achieves more balanced and clinically meaningful performance across both image-to-text and text-to-image
retrieval tasks compared to the pretrained BiomedCLIP and general-purpose CLIP models. Furthermore, t-SNE visualizations reveal clearer semantic clustering of normal and abnormal cases, demonstrating the model's enhanced diagnostic sensitivity. These findings highlight the value of domain-adaptive, multi-task learning for advancing cross-modal
retrieval in biomedical applications.
Read the full abstract ▾
All authors Zhaohui Liang, Sivaramakrishnan Rajaraman, Niccolo Marini, Zhiyun Xue, Sameer Antani
Computer Vision
Sketch&Patch++: Efficient Structure-Aware 3D Gaussian Representation
Yuang Shi, Géraldine Morin, Simone Gasparini et al. (+1 more)
We observe that Gaussians exhibit distinct roles and characteristics analogous to traditional artistic techniques – like how artists first sketch outlines before filling in broader areas with color, some Gaussians capture high-frequency features such as edges and contours, while others represent broader, smoother regions analogous to brush strokes that add volume and depth. Based on this observation, we propose a hybrid representation that categorizes Gaussians into (i) Sketch Gaussians, which represent high-frequency, boundary-defining features, and (ii) Patch Gaussians, which cover low-frequency, smooth regions. This semantic separation naturally enables layered progressive streaming, where the compact Sketch Gaussians establish the structural skeleton before Patch Gaussians incrementally refine volumetric detail. In this work, we extend our previous method to arbitrary 3D scenes by proposing a novel hierarchical adaptive categorization framework that operates directly on the 3DGS representation. Our approach employs multi-criteria density-based clustering, combined with adaptive quality-driven refinement. This method eliminates dependency on external 3D line primitives while ensuring optimal parametric encoding effectiveness. Our comprehensive evaluation across diverse scenes, including both man-made and natural environments, demonstrates that our method achieves up to 1.74 dB improvement in PSNR, 6.7% in SSIM, and 41.4% in LPIPS at equivalent model sizes compared to uniform
pruning baselines. For indoor scenes, our method can maintain visual quality with only 0.5% of the original model size. This structure-aware representation enables efficient storage, adaptive streaming, and rendering of high-fidelity 3D content across bandwidth-constrained networks and resource-limited devices.
Read the full abstract ▾
All authors Yuang Shi, Géraldine Morin, Simone Gasparini, Wei Tsang Ooi
Computer Vision
EdgeLDR: Quaternion Low-Displacement Rank Neural Networks for Edge-Efficient Deep Learning
Vladimir Frants, Sos Agaian and Karen Panetta
Deploying deep neural networks on edge devices is often limited by the
memory traffic and compute cost of dense linear operators. While quaternion neural networks improve parameter efficiency by coupling multiple channels through Hamilton products, they typically retain unstructured dense weights; conversely, structured
matrices enable fast computation but are usually applied in the real domain. This paper introduces EdgeLDR, a practical framework for quaternion block-circulant linear and convolutional layers that combines quaternion channel mixing with block-circulant parameter structure and enables FFT-based evaluation through the complex adjoint representation. We present reference implementations of EdgeLDR layers and compare FFT-based computation against a naive spatial-domain realization of quaternion circulant products. FFT evaluation yields large empirical speedups over the naive implementation and keeps latency stable as block size increases, making larger compression factors computationally viable. We further integrate EdgeLDR layers into compact CNN and
Transformer backbones and evaluate accuracy-compression trade-offs on 32x32 RGB classification (CIFAR-10/100, SVHN) and hyperspectral image classification (Houston 2013, Pavia University), reporting parameter counts and CPU/GPU latency. The results show that EdgeLDR layers provide significant compression with competitive accuracy.
Read the full abstract ▾
Computer Vision
Ensemble of radiomics and ConvNeXt for breast cancer diagnosis
Jorge Alberto Garza-Abdala, Gerardo Alejandro Fumagal-González, Beatriz A. Bosques-Palomo et al. (+4 more)
Early diagnosis of breast cancer is crucial for improving survival rates. Radiomics and deep learning (DL) have shown significant potential in assisting radiologists with early cancer detection. This paper aims to critically assess the performance of radiomics, DL, and ensemble techniques in detecting cancer from screening mammograms. Two independent
datasets were used: the RSNA 2023 Breast Cancer Detection Challenge (11,913 patients) and a Mexican cohort from the TecSalud
dataset (19,400 patients). The ConvNeXtV1-small DL model was trained on the RSNA
dataset and validated on the TecSalud
dataset, while radiomics models were developed using the TecSalud
dataset and validated with a leave-one-year-out approach. The ensemble method consistently combined and calibrated predictions using the same methodology. Results showed that the ensemble approach achieved the highest area under the curve (AUC) of 0.87, compared to 0.83 for ConvNeXtV1-small and 0.80 for radiomics. In conclusion, ensemble methods combining DL and radiomics predictions significantly enhance breast cancer diagnosis from mammograms.
Read the full abstract ▾
All authors Jorge Alberto Garza-Abdala, Gerardo Alejandro Fumagal-González, Beatriz A. Bosques-Palomo, Mario Alexis Monsivais Molina, Daly Avedano, Servando Cardona-Huerta, José Gerardo Tamez-Pena
Computer Vision
MOSAIC-GS: Monocular Scene Reconstruction via Advanced Initialization for Complex Dynamic Environments
Svitlana Morkva, Maximum Wilder-Smith, Michael Oechsle et al. (+3 more)
We present MOSAIC-GS, a novel, fully explicit, and computationally efficient approach for high-fidelity dynamic scene reconstruction from
monocular videos using Gaussian Splatting.
Monocular reconstruction is inherently ill-posed due to the lack of sufficient multiview constraints, making accurate recovery of object geometry and temporal coherence particularly challenging. To address this, we leverage multiple
geometric cues, such as depth, optical flow, dynamic object
segmentation, and point tracking. Combined with
rigidity-based motion constraints, these cues allow us to estimate preliminary 3D scene dynamics during an initialization stage. Recovering scene dynamics prior to the photometric optimization reduces reliance on motion inference from visual appearance alone, which is often ambiguous in
monocular settings. To enable compact representations, fast training, and real-time rendering while supporting non-rigid deformations, the scene is decomposed into static and dynamic components. Each Gaussian in the dynamic part of the scene is assigned a trajectory represented as time-dependent Poly-
Fourier curve for parameter-efficient motion encoding. We demonstrate that MOSAIC-GS achieves substantially faster optimization and rendering compared to existing methods, while maintaining reconstruction quality on par with state-of-the-art approaches across standard
monocular dynamic scene
benchmarks.
Read the full abstract ▾
All authors Svitlana Morkva, Maximum Wilder-Smith, Michael Oechsle, Alessio Tonioni, Marco Hutter, Vaishakh Patil
Computer Vision
STResNet & STYOLO : A New Family of Compact Classification and Object Detection Models for MCUs
Sudhakar Sah and Ravish Kumar
Recent advancements in lightweight neural networks have significantly improved the efficiency of deploying deep learning models on edge hardware. However, most existing architectures still trade accuracy for latency, which limits their applicability on microcontroller and neural processing unit based devices. In this work, we introduce two new model families, STResNet for image classification and STYOLO for object detection, jointly optimized for accuracy, efficiency, and
memory footprint on resource constrained platforms. The proposed STResNet series, ranging from Nano to Tiny variants, achieves competitive ImageNet 1K accuracy within a four million parameter budget. Specifically, STResNetMilli attains 70.0 percent Top 1 accuracy with only three million parameters, outperforming MobileNetV1 and ShuffleNetV2 at comparable computational complexity. For object detection, STYOLOMicro and STYOLOMilli achieve 30.5 percent and 33.6 percent mean average precision, respectively, on the MS COCO
dataset, surpassing YOLOv5n and YOLOX Nano in both accuracy and efficiency. Furthermore, when STResNetMilli is used as a backbone with the Ultralytics training environment.
Read the full abstract ▾
Computer Vision
Coding the Visual World: From Image to Simulation Using Vision Language Models
Sagi Eppel
The ability to construct mental models of the world is a central aspect of understanding. Similarly, visual understanding can be viewed as the ability to construct a representative model of the system depicted in an image. This work explores the capacity of
Vision Language Models (VLMs) to recognize and simulate the systems and mechanisms depicted in images using the Im2Sim methodology. The VLM is given a natural image of a real-world system (e.g., cities, clouds, vegetation) and is tasked with describing the system and writing code that simulates and generates it. This generative code is then executed to produce a synthetic image, which is compared against the original. This approach is tested on various complex emergent systems, ranging from physical systems (waves, lights, clouds) to vegetation, cities, materials, and geological formations. Through analysis of the models and images generated by the VLMs, we examine their understanding of the systems in images. The results show that leading VLMs (GPT, Gemini) have the ability to understand and model complex, multi-component systems across multiple layers of abstraction and a wide range of domains. At the same time, the VLMs exhibit limited ability to replicate fine details and low-level arrangements of patterns in the image. These findings reveal an interesting asymmetry: VLMs combine high-level, deep visual understanding of images with limited perception of fine details.
Read the full abstract ▾
Computer Vision
Bi-Orthogonal Factor Decomposition for Vision Transformers
Fenil R. Doshi, Thomas Fel, Talia Konkle et al. (+1 more)
Self-
attention is the central computational primitive of
Vision Transformers, yet we lack a principled understanding of what information
attention mechanisms exchange between
tokens.
Attention maps describe where weight mass concentrates; they do not reveal whether queries and keys trade position, content, or both. We introduce Bi-orthogonal Factor Decomposition (BFD), a two-stage analytical framework: first, an ANOVA-based decomposition statistically disentangles
token activations into orthogonal positional and content factors; second, SVD of the query-key interaction
matrix QK^T exposes bi-orthogonal modes that reveal how these factors mediate communication. After validating proper isolation of position and content, we apply BFD to state-of-the-art
vision models and uncover three phenomena.(i)
Attention operates primarily through content. Content-content interactions dominate
attention energy, followed by content-position coupling. DINOv2 allocates more energy to content-position than supervised models and distributes computation across a richer mode spectrum. (ii)
Attention mechanisms exhibit specialization: heads differentiate into content-content, content-position, and position-position operators, while singular modes within heads show analogous specialization. (iii) DINOv2's superior holistic shape processing emerges from intermediate layers that simultaneously preserve positional structure while contextually enriching semantic content. Overall, BFD exposes how
tokens interact through
attention and which informational factors - positional or semantic - mediate their communication, yielding practical insights into
vision transformer mechanisms.
Read the full abstract ▾
All authors Fenil R. Doshi, Thomas Fel, Talia Konkle, George Alvarez
Computer Vision
Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video
Zeren Jiang, Chuanxia Zheng, Iro Laina et al. (+2 more)
We propose Mesh4D, a feed-forward model for
monocular 4D mesh reconstruction. Given a
monocular video of a dynamic object, our model reconstructs the object's complete 3D shape and motion, represented as a deformation field. Our key contribution is a compact latent space that encodes the entire animation sequence in a single pass. This latent space is learned by an autoencoder that, during training, is guided by the skeletal structure of the training objects, providing strong priors on plausible deformations. Crucially, skeletal information is not required at inference time. The encoder employs spatio-temporal
attention, yielding a more stable representation of the object's overall deformation. Building on this representation, we train a latent
diffusion model that, conditioned on the input
video and the mesh reconstructed from the first frame, predicts the full animation in one shot. We evaluate Mesh4D on reconstruction and novel view synthesis
benchmarks, outperforming prior methods in recovering accurate 3D shape and deformation.
Read the full abstract ▾
All authors Zeren Jiang, Chuanxia Zheng, Iro Laina, Diane Larlus, Andrea Vedaldi
Computer Vision
Quantum Implicit Neural Representations for Novel View Synthesis
Daniele Lizzio Bosco, Shuteng Wang, Giuseppe Serra et al. (+1 more)
Quantum implicit neural representations have recently emerged as compact function approximators for continuous visual signals. In parallel, Neural Radiance Fields (NeRFs) have become a dominant framework for novel-view synthesis by recovering continuous volumetric scene representations from posed 2D images, but often require substantial model capacity and intensive optimisation. We introduce 3D
Quantum Implicit Scene Representation (3D-QISR), the first hybrid
quantum-classical radiance-field model for novel-view synthesis from 2D observations. 3D-QISR replaces the classical NeRF backbone with parameterised
quantum circuits that encode spatial and view-dependent information through
quantum state transformations, enabling compact
quantum-enhanced implicit representations while preserving compatibility with standard volumetric rendering. We propose two architectures: Full 3D-QISR, which maximises the representational role of a unified
quantum state, and Dual-Branch 3D-QISR, which separates spatial and view-dependent
quantum embeddings to introduce a task-aligned inductive bias, reduce state-preparation complexity, and improve scalability and hardware compatibility. Experiments on moderate-resolution novel-view synthesis
benchmarks show that 3D-QISR executed on simulated
quantum hardware matches or outperforms classical baselines while using fewer than half the trainable parameters. These results position
quantum neural networks as compact and competitive building blocks for continuous volumetric scene representation, opening a new direction for
quantum-enhanced implicit neural rendering.
Read the full abstract ▾
All authors Daniele Lizzio Bosco, Shuteng Wang, Giuseppe Serra, Vladislav Golyanik
Computer Vision
RL-AWB: Deep Reinforcement Learning for Auto White Balance Correction in Low-Light Night-time Scenes
Yuan-Kang Lee, Kuan-Lin Chen, Chia-Che Chang et al. (+1 more)
Nighttime color constancy still remains a challenging problem in computational photography due to low-light noise and complex illumination conditions. We present RL-AWB, a novel framework combining statistical methods with deep reinforcement learning for nighttime white balance. Our method begins with a statistical
algorithm tailored for nighttime scenes, integrating salient gray pixel detection with novel illuminant estimation. Building on this foundation, we develop the first deep reinforcement learning approach for color constancy that leverages the statistical
algorithm as its core, mimicking professional AWB tuning experts by dynamically determining image-specific parameters at inference time, without requiring ground-truth illuminants or reference images. To further facilitate cross-sensor evaluation, we introduce the first multi-sensor nighttime
dataset. Experiment results demonstrate that our method achieves strong generalization capability across low-light and well-illuminated images. Project page: https://ntuneillee.github.io/research/rl-awb/
Read the full abstract ▾
All authors Yuan-Kang Lee, Kuan-Lin Chen, Chia-Che Chang, Yu-Lun Liu
Computer Vision
Pixel-Perfect Visual Geometry Estimation
Gangwei Xu, Haotong Lin, Hongcheng Luo et al. (+6 more)
Recovering clean and accurate geometry from images is essential for robotics and augmented reality. However, existing geometry foundation models still suffer severely from flying pixels and the loss of fine details. In this paper, we present pixel-perfect visual geometry models that can predict high-quality, flying-pixel-free point clouds by leveraging generative modeling in the pixel space. We first introduce Pixel-Perfect Depth (PPD), a
monocular depth foundation model built upon pixel-space
diffusion transformers (DiT). To address the high computational complexity associated with pixel-space
diffusion, we propose two key designs: 1) Semantics-Prompted DiT, which incorporates semantic representations from
vision foundation models to
prompt the
diffusion process, preserving global semantics while enhancing fine-grained visual details; and 2) Cascade DiT architecture that progressively increases the number of image
tokens, improving both efficiency and accuracy. To further extend PPD to
video (PPVD), we introduce a new Semantics-Consistent DiT, which extracts temporally consistent semantics from a multi-view geometry foundation model. We then perform reference-guided
token propagation within the DiT to maintain temporal coherence with minimal computational and
memory overhead. Our models achieve the best performance among all generative
monocular and
video depth estimation models and produce significantly cleaner point clouds than all other models.
Read the full abstract ▾
All authors Gangwei Xu, Haotong Lin, Hongcheng Luo, Haiyang Sun, Bing Wang, Guang Chen, Sida Peng, Hangjun Ye, Xin Yang
Computer Vision
GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation
Henghui Ding, Chang Liu, Shuting He et al. (+2 more)
Referring Expression
Segmentation (RES) and Comprehension (REC) respectively segment and detect the object described by an expression, while Referring Expression Generation (REG) generates an expression for the selected object. Existing
datasets and methods commonly support single-target expressions only, i.e., one expression refers to one object, not considering multi-target and no-target expressions. This greatly limits the real applications of REx (RES/REC/REG). This paper introduces three new
benchmarks called Generalized Referring Expression
Segmentation (GRES), Comprehension (GREC), and Generation (GREG), collectively denoted as GREx, which extend the classic REx to allow expressions to identify an arbitrary number of objects. We construct the first large-scale GREx
dataset gRefCOCO that contains multi-target, no-target, and single-target expressions and their corresponding images with labeled targets. GREx and gRefCOCO are designed to be backward-compatible with REx, facilitating extensive experiments to study the performance gap of the existing REx methods on GREx tasks. One of the challenges of GRES/GREC is complex relationship modeling, for which we propose a baseline ReLA that adaptively divides the image into regions with sub-instance clues and explicitly models the region-region and region-
language dependencies. The proposed ReLA achieves the state-of-the-art results on the both GRES and GREC tasks. The proposed gRefCOCO
dataset and method are available at https://henghuiding.github.io/GREx.
Read the full abstract ▾
All authors Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Yu-Gang Jiang
Computer Vision · Cross-listed from cs.RO
Generate, Transfer, Adapt: Learning Functional Dexterous Grasping from a Single Human Demonstration
Xingyi He, Adhitya Polavaram, Yunhao Cao et al. (+4 more)
Functional grasping with dexterous robotic hands is a key capability for enabling tool use and complex manipulation, yet progress has been constrained by two persistent bottlenecks: the scarcity of large-scale
datasets and the absence of integrated semantic and
geometric reasoning in learned models. In this work, we present CorDex, a framework that robustly learns dexterous functional grasps of novel objects from synthetic data generated from just a single human demonstration. At the core of our approach is a correspondence-based data engine that generates diverse, high-quality training data in simulation. Based on the human demonstration, our data engine generates diverse object instances of the same category, transfers the expert grasp to the generated objects through correspondence estimation, and adapts the grasp through optimization. Building on the generated data, we introduce a
multimodal prediction network that integrates visual and
geometric information. By devising a local-global fusion module and an importance-aware sampling mechanism, we enable robust and computationally efficient prediction of functional dexterous grasps. Through extensive experiments across various object categories, we demonstrate that CorDex generalizes well to unseen object instances and significantly outperforms state-of-the-art baselines.
Read the full abstract ▾
All authors Xingyi He, Adhitya Polavaram, Yunhao Cao, Om Deshmukh, Tianrui Wang, Xiaowei Zhou, Kuan Fang
Computer Vision
RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation
Boyang Wang, Haoran Zhang, Shujie Zhang et al. (+8 more)
The diversity, quantity, and quality of manipulation data are critical for training effective
robot policies. However, due to hardware and physical setup constraints, collecting large-scale real-world manipulation data remains difficult to scale across diverse environments. Recent work uses text-
prompt conditioned image
diffusion models to augment manipulation data by altering the backgrounds and tabletop objects in the visual observations. However, these approaches often overlook the practical need for multi-view and temporally coherent observations required by state-of-the-art policy models. Further, text
prompts alone cannot reliably specify the scene setup. To provide the
diffusion model with explicit visual guidance, we introduce visual identity prompting, which supplies exemplar images as conditioning inputs to guide the generation of the desired scene setup. To this end, we also build a scalable pipeline to curate a visual identity pool from large robotics
datasets. Using our augmented manipulation data to train downstream
vision-
language-action and visuomotor policy models yields consistent performance gains in both simulation and real-
robot settings.
Read the full abstract ▾
All authors Boyang Wang, Haoran Zhang, Shujie Zhang, Jinkun Hao, Mingda Jia, Qi Lv, Yucheng Mao, Zhaoyang Lyu, Jia Zeng, Xudong Xu, Jiangmiao Pang
Computer Vision
Plenoptic Video Generation
Xiao Fu, Shitao Tang, Min Shi et al. (+5 more)
Camera-controlled generative
video re-rendering methods, such as ReCamMaster, have achieved remarkable progress. However, despite their success in single-view setting, these works often struggle to maintain consistency across multi-view scenarios. Ensuring spatio-temporal coherence in hallucinated regions remains challenging due to the inherent stochasticity of generative models. To address it, we introduce PlenopticDreamer, a framework that synchronizes generative
hallucinations to maintain spatio-temporal
memory. The core idea is to train a multi-in-single-out
video-conditioned model in an autoregressive manner, aided by a camera-guided
video retrieval strategy that adaptively selects salient
videos from previous generations as conditional inputs. In addition, Our training incorporates progressive context-scaling to improve convergence, self-conditioning to enhance robustness against long-range visual degradation caused by error accumulation, and a long-
video conditioning mechanism to support extended
video generation. Extensive experiments on the Basic and Agibot
benchmarks demonstrate that PlenopticDreamer achieves state-of-the-art
video re-rendering, delivering superior view synchronization, high-fidelity visuals, accurate camera control, and diverse view transformations (e.g., third-person to third-person, and head-view to gripper-view in robotic manipulation). Project page: https://research.nvidia.com/labs/dir/plenopticdreamer/
Read the full abstract ▾
All authors Xiao Fu, Shitao Tang, Min Shi, Xian Liu, Jinwei Gu, Ming-Yu Liu, Dahua Lin, Chen-Hsuan Lin
Computer Vision
ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos
Rustin Soraki, Homanga Bharadhwaj, Ali Farhadi et al. (+1 more)
Humans can effortlessly anticipate how objects might move or change through interaction–imagining a cup being lifted, a knife slicing, or a lid being closed. We aim to endow computational systems with a similar ability to predict plausible future object motions directly from passive visual observation. We introduce ObjectForesight, a 3D object-centric dynamics model that predicts future 6-DoF poses and trajectories of rigid objects from short
egocentric video sequences. Unlike conventional world or dynamics models that operate in pixel or latent space, ObjectForesight represents the world explicitly in 3D at the object level, enabling geometrically grounded and temporally coherent predictions that capture object affordances and trajectories. To train such a model at scale, we leverage recent advances in
segmentation, mesh reconstruction, and 3D pose estimation to curate a
dataset of 2 million plus short clips with pseudo-ground-truth 3D object trajectories. Through extensive experiments, we show that ObjectForesight achieves significant gains in accuracy,
geometric consistency, and generalization to unseen objects and scenes, establishing a scalable framework for learning physically grounded, object-centric dynamics models directly from observation. objectforesight.github.io
Read the full abstract ▾
All authors Rustin Soraki, Homanga Bharadhwaj, Ali Farhadi, Roozbeh Mottaghi
Computer Vision · Cross-listed from cs.AI
Learning Latent Action World Models In The Wild
Quentin Garrido, Tushar Nagarajan, Basile Terver et al. (+3 more)
Agents capable of
reasoning and planning in the real world require the ability of predicting the consequences of their actions. While world models possess this capability, they most often require action labels, that can be complex to obtain at scale. This motivates the learning of latent action models, that can learn an action space from
videos alone. Our work addresses the problem of learning latent actions world models on in-the-wild
videos, expanding the scope of existing works that focus on simple robotics simulations,
video games, or manipulation data. While this allows us to capture richer actions, it also introduces challenges stemming from the
video diversity, such as environmental noise, or the lack of a common embodiment across
videos. To address some of the challenges, we discuss properties that actions should follow as well as relevant architectural choices and evaluations. We find that continuous, but constrained, latent actions are able to capture the complexity of actions from in-the-wild
videos, something that the common vector quantization does not. We for example find that changes in the environment coming from
agents, such as humans entering the room, can be transferred across
videos. This highlights the capability of learning actions that are specific to in-the-wild
videos. In the absence of a common embodiment across
videos, we are mainly able to learn latent actions that become localized in space, relative to the camera. Nonetheless, we are able to train a controller that maps known actions to latent ones, allowing us to use latent actions as a universal interface and solve planning tasks with our world model with similar performance as action-conditioned baselines. Our analyses and experiments provide a step towards scaling latent action models to the real world.
Read the full abstract ▾
All authors Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, Michael Rabbat
Computer Vision
FlowLet: Conditional 3D Brain MRI Synthesis using Wavelet Flow Matching
Danilo Danese, Angela Lombardi, Matteo Attimonelli et al. (+2 more)
Brain Magnetic Resonance Imaging (MRI) plays a central role in studying neurological development, aging, and diseases. One key application is
Brain Age Prediction (BAP), which estimates an individual's biological
brain age from MRI data. Effective BAP models require large, diverse, and age-balanced
datasets, whereas existing 3D MRI
datasets are demographically skewed, limiting
fairness and generalizability. Acquiring new data is costly and ethically constrained, motivating generative data augmentation. Current generative methods are often based on latent
diffusion models, which operate in learned low dimensional latent spaces to address the
memory demands of volumetric MRI data. However, these methods are typically slow at inference, may introduce artifacts due to latent compression, and are rarely conditioned on age, thereby affecting the BAP performance. In this work, we propose FlowLet, a conditional generative framework that synthesizes age-conditioned 3D MRIs by leveraging flow matching within an invertible 3D wavelet domain, helping to avoid reconstruction artifacts and reducing computational demands. Experiments show that FlowLet generates high-fidelity volumes with few sampling steps. Training BAP models with data generated by FlowLet improves performance for underrepresented age groups, and region-based analysis confirms preservation of anatomical structures.
Read the full abstract ▾
All authors Danilo Danese, Angela Lombardi, Matteo Attimonelli, Giuseppe Fasano, Tommaso Di Noia
Computer Vision
MoE3D: A Mixture-of-Experts Module for 3D Reconstruction
Zichen Wang, Ang Cao, Liam J. Wang et al. (+1 more)
We propose a simple yet effective approach to enhance the performance of feed-forward 3D reconstruction models. Existing methods often struggle near depth discontinuities, where standard regression losses encourage spatial averaging and thus blur sharp boundaries. To address this issue, we introduce a mixture-of-experts formulation that handles uncertainty at depth boundaries by combining multiple smooth depth predictions. A softmax weighting head dynamically selects among these hypotheses on a per-pixel basis. By integrating our mixture model into a pre-trained state-of-the-art 3D model, we achieve a substantial reduction of boundary artifacts and gains in overall reconstruction accuracy. Notably, our approach is highly compute efficient, delivering generalizable improvements even when fine-tuned on a small subset of training data while incurring only negligible additional inference computation, suggesting a promising direction for lightweight and accurate 3D reconstruction.
Read the full abstract ▾
All authors Zichen Wang, Ang Cao, Liam J. Wang, Jeong Joon Park
Computer Vision
William Rudman, Michal Golovanevsky, Dana Arad et al. (+4 more)
Large
vision-
language models (VLMs) are highly capable, yet often hallucinate by favoring textual
prompts over visual evidence. We study this failure mode in a controlled object-counting setting, where the
prompt overstates the number of objects in the image (e.g., asking a model to describe four waterlilies when only three are present). At low object counts, models often correct the overestimation, but as the number of objects increases, they increasingly conform to the
prompt regardless of the discrepancy. Through mechanistic analysis of three VLMs, we identify a small set of
attention heads whose ablation substantially reduces
prompt-induced
hallucinations (PIH) by at least 40% without additional training. Across models, PIH-heads mediate
prompt copying in model-specific ways. We characterize these differences and show that PIH ablation increases correction toward visual evidence. Our findings offer insights into the internal mechanisms driving
prompt-induced
hallucinations, revealing model-specific differences in how these behaviors are implemented.
Read the full abstract ▾
All authors William Rudman, Michal Golovanevsky, Dana Arad, Yonatan Belinkov, Ritambhara Singh, Carsten Eickhoff, Kyle Mahowald
Computer Vision
AgentCompress: Task-Aware Compression for Affordable Large Language Model Agents
Zuhair Ahmed Khan Taha, Mohammed Mudassir Uddin and Shahnawaz Alam
Large
language models hold considerable promise for various applications, but their computational requirements create a barrier that many institutions cannot overcome. A single session using a 70-billion-parameter model can cost around $127 in cloud computing fees, which puts these tools out of reach for organizations operating on limited budgets. We present AgentCompress, a framework that tackles this problem through task-aware dynamic compression. The idea comes from a simple observation: not all tasks require the same computational effort. Complex
reasoning, for example, is far more demanding than text reformatting, yet conventional compression applies the same reduction to both. Our approach uses a lightweight neural controller that looks at the first few
tokens of each request, estimates how complex the task will be, and sends it to an appropriately quantized version of the model. This routing step adds only about 12 milliseconds of overhead. We tested the framework on 290 multi-stage workflows from domains including computer science, physics, chemistry, and biology. The results show a 68.3% reduction in computational costs while preserving 96.2% of the original success rate. These findings suggest that routing queries intelligently can make powerful
language models substantially more affordable without sacrificing output quality
Read the full abstract ▾
Computer Vision
VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twice
Shuming Liu, Mingchen Zhuge, Changsheng Zhao et al. (+20 more)
Chain-of-thought (CoT)
reasoning has emerged as a powerful tool for
multimodal large
language models on
video understanding tasks. However, its necessity and advantages over direct answering remain underexplored. In this paper, we first demonstrate that for RL-trained
video models, direct answering often matches or even surpasses CoT performance, despite CoT producing step-by-step analyses at a higher computational cost. Motivated by this, we propose VideoAuto-R1, a
video understanding framework that adopts a reason-when-necessary strategy. During training, our approach follows a Thinking Once, Answering Twice paradigm: the model first generates an initial answer, then performs
reasoning, and finally outputs a reviewed answer. Both answers are supervised via verifiable rewards. During inference, the model uses the confidence score of the initial answer to determine whether to proceed with
reasoning. Across
video QA and grounding
benchmarks, VideoAuto-R1 achieves state-of-the-art accuracy with significantly improved efficiency, reducing the average response length by ~3.3x, e.g., from 149 to just 44
tokens. Moreover, we observe a low rate of thinking-mode activation on perception-oriented tasks, but a higher rate on
reasoning-intensive tasks. This suggests that explicit
language-based
reasoning is generally beneficial but not always necessary.
Read the full abstract ▾
All authors Shuming Liu, Mingchen Zhuge, Changsheng Zhao, Jun Chen, Lemeng Wu, Zechun Liu, Chenchen Zhu, Zhipeng Cai, Chong Zhou, Haozhe Liu, Ernie Chang, Saksham Suri, Hongyu Xu, Qi Qian, Wei Wen, Balakrishnan Varadarajan, Zhuang Liu, Hu Xu, Florian Bordes, Raghuraman Krishnamoorthi, Bernard Ghanem, Vikas Chandra, Yunyang Xiong
Computer Vision
CoV: Chain-of-View Prompting for Spatial Reasoning
Haoyu Zhao, Akide Liu, Zeyu Zhang et al. (+5 more)
Embodied question answering (EQA) in 3D environments often requires collecting context that is distributed across multiple viewpoints and partially occluded. However, most recent
vision–
language models (VLMs) are constrained to a fixed and finite set of input views, which limits their ability to acquire question-relevant context at inference time and hinders complex spatial
reasoning. We propose Chain-of-View (CoV) prompting, a training-free, test-time
reasoning framework that transforms a VLM into an active viewpoint reasoner through a coarse-to-fine exploration process. CoV first employs a View Selection
agent to filter redundant frames and identify question-aligned anchor views. It then performs fine-grained view adjustment by interleaving iterative
reasoning with discrete camera actions, obtaining new observations from the underlying 3D scene representation until sufficient context is gathered or a step budget is reached. We evaluate CoV on OpenEQA across four mainstream VLMs and obtain an average +11.56% improvement in LLM-Match, with a maximum gain of +13.62% on Qwen3-VL-Flash. CoV further exhibits test-time scaling: increasing the minimum action budget yields an additional +2.51% average improvement, peaking at +3.73% on Gemini-2.5-Flash. On ScanQA and SQA3D, CoV delivers strong performance (e.g., 116 CIDEr / 31.9 EM@1 on ScanQA and 51.1 EM@1 on SQA3D). Overall, these results suggest that question-aligned view selection coupled with open-view search is an effective, model-agnostic strategy for improving spatial
reasoning in 3D EQA without additional training. Code is available on https://github.com/ziplab/CoV .
Read the full abstract ▾
All authors Haoyu Zhao, Akide Liu, Zeyu Zhang, Weijie Wang, Feng Chen, Ruihan Zhu, Gholamreza Haffari, Bohan Zhuang
Computer Vision · Cross-listed from cs.GR
GenAI-DrawIO-Creator: A Framework for Automated Diagram Generation
Jinze Yu and Dayuan Jiang
Diagrams are crucial for communicating complex information, yet creating and modifying them remains a labor-intensive task. We present GenAI-DrawIO-Creator, a novel framework that leverages Large
Language Models (LLMs) to automate diagram generation and manipulation in the structured XML format used by draw.io. Our system integrates Claude 3.7 to reason about structured visual data and produce valid diagram representations. Key contributions include a high-level system design enabling real-time diagram updates, specialized
prompt engineering and error-checking to ensure well-formed XML outputs. We demonstrate a working prototype capable of generating accurate diagrams (such as network architectures and flowcharts) from natural
language or code, and even replicating diagrams from images. Simulated evaluations show that our approach significantly reduces diagram creation time and produces outputs with high structural fidelity. Our results highlight the promise of Claude 3.7 in handling structured visual
reasoning tasks and lay the groundwork for future research in AI-assisted diagramming applications.
Read the full abstract ▾