Computer Vision
Learning a Maximum Entropy Model for Visual Textures using Diffusion
Xinyuan Zhao and Eero P. Simoncelli
Visual textures – spatially homogeneous image regions containing repeated elements (e.g. a field of grass, the bark of a tree) – are ubiquitous in visual scenes and provide important cues for recognizing and analyzing materials, surface geometry and objects. A number of existing texture models extract essential statistics from a single texture image, and can then generate high-quality samples that are visually similar to the original by matching these statistics. However, their statistics are either hand-designed or based on a network pretrained for another purpose (e.g., object recognition). Here, we develop the first principled method for unsupervised learning of a set of statistics that are used to constrain a maximum
entropy probability model for texture. We leverage methods developed for generative
diffusion models to derive training and sampling procedures, and compare these to the traditional method of sampling via matching the statistics. Despite the compactness of our trained model (512 statistics), it generates texture images whose quality is as good as or better than the current state-of-the-art model (~177k statistics), both visually and by quantitative perceptual scoring using DreamSim. Finally, we show that images generated from parameters lying along a line between the parameter vectors corresponding to two different reference textures are homogeneous textures with features that smoothly transition between those of the two references.
Read the full abstract ▾
Computer Vision
Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation
Hongchao Shu, Roger D. Soberanis-Mukul, Hao Ding et al. (+5 more)
Accurate
vision-based
navigation in
monocular endoscopy is difficult due to limited depth cues, weak tissue texture, non-rigid deformation, and substantial appearance variation across domains, all of which complicate pose estimation, depth prediction, and image-to-anatomy
alignment. Although recent
vision foundation models have shown promise, their learned representations often remain insufficiently geometry-consistent, hindering stable feature correspondence and limiting their reliability for downstream
navigation tasks. We propose a unified framework for learning geometry-consistent and domain-robust image representations for
monocular endoscopy. The framework combines a synthetic data pipeline that provides accurate
geometric supervision with Hierarchy-Aware Geometry-Semantic Adaptation, a structured alternative to standard LoRA that inserts low-rank adapters selectively across the
transformer hierarchy and couples them with layer-wise training objectives to encourage
geometric correspondence in intermediate features and semantic consistency in deeper features. Experiments on public and proprietary
datasets show improved
geometric and semantic representation quality, leading to better performance on downstream
navigation tasks including pose estimation and
monocular depth estimation. The learned representations show favorable synthetic-to-real transfer on clinical bronchoscopy and provide a useful initialization for adaptation to sinus endoscopy and colonoscopy under limited supervision. The framework also shows favorable scaling with model size and training data. These results support hierarchy-aware, geometry-guided adaptation as a practical approach for endoscopic representation learning.
Read the full abstract ▾
All authors Hongchao Shu, Roger D. Soberanis-Mukul, Hao Ding, Morgan Ringel, Mali Shen, Saif Iftekar Sayed, Hedyeh Rafii-Tari, Mathias Unberath
Computer Vision
FATE: Pillar Encoding and Frequency-Aware Training for Event-Based Object Detection
Md Tawheedul Islam Bhuian and Kyoung-Don Kang
Event cameras are bio-inspired sensors that asynchronously capture logarithmic intensity changes, offering inherent advantages in high-speed and high-dynamic-range scenarios. However, the sparse and asynchronous nature of event streams poses a fundamental challenge for modern deep learning architectures. To enable compatibility with standard models, most existing approaches partition the accumulation window into fixed temporal sub-bins. While effective for spatial processing, this internal discretization discards fine-grained temporal structure and constrains inference to the low temporal frequencies imposed by training supervision. To address this limitation, we propose FATE, a unified framework built upon a novel Pillar Encoding (PE). While operating over discrete macro-accumulation windows dictated by the target frequency, PE avoids internal temporal sub-binning. It organizes events into spatial pillars and approximates their intra-window
evolution via projection onto a continuous-time orthogonal
polynomial basis. This formulation yields an L2-optimal representation that retains rich temporal dynamics in a dense pseudo-image, mitigating information loss under sparse event conditions. To fully leverage this representation, we introduce Frequency-Aware Training (FAT), a soft mean-teacher curriculum that generates temporally dense pseudo-labels, effectively bridging the mismatch between low-frequency supervision and high-frequency inference. Extensive experiments demonstrate that FATE generalizes across architectural paradigms and consistently outperforms strong baselines. It enables robust object detection at high temporal resolutions up to 200 Hz, while incurring minimal overhead in parameter count and inference latency
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
ProCUA-SFT Technical Report
Jaehun Jung, Ximing Lu, Brandon Cui et al. (+11 more)
Training computer-use
agents (CUAs) – models that interact with graphical desktops through screenshots and keyboard/mouse actions – requires large-scale, diverse trajectory data collected in full desktop environments. The largest public resource, AgentNet (22.5K human trajectories), leads to negative transfer when used for supervised fine-tuning (SFT): continuing training UI-TARS 7B on AgentNet causes OSWorld success rate to fall from 26.3% to 8-10%. We present ProCUA-SFT, a
dataset of 3.1M step-level SFT samples distilled from 93K synthetic trajectories across 2,484 application combinations. The
dataset is produced by a fully automated pipeline that (i) synthesizes grounded tasks on live desktops seeded with real-world content – 912 spreadsheets from SpreadsheetBench, approximately 10K permissively-licensed presentations from Zenodo10K, and multi-application OSWorld configs – and (ii) verifies each task's feasibility through binary precondition checking before rollout. A single VLM (Kimi-K2.5) serves as goal generator, precondition judge, and trajectory executor, eliminating planner-actor capability gaps. Each trajectory is expanded into step-prefix samples that exactly reproduce the context layout seen at inference time. Fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch yields 45.0% on OSWorld – an 18.7 percentage-point improvement over the base model and over 35% above AgentNet-trained counterparts. A subset of ProCUA was incorporated into the training data for the Nemotron 3 Nano Omni model, contributing to its computer-use capabilities.
Read the full abstract ▾
All authors Jaehun Jung, Ximing Lu, Brandon Cui, Muhammad Khalifa, Shaokun Zhang, Hao Zhang, Jin Xu, Amala Sanjay Deshmukh, Karan Sapra, Andrew Tao, Yejin Choi, Jan Kautz, Mingjie Liu, Yi Dong
Computer Vision
SierpinskiCam: Camera-Controlled Video Retaking with Sierpinski Triangle Pattern Cues
Suttisak Wizadwongsa, Hyelin Nam, Supasorn Suwajanakorn et al. (+1 more)
Generating novel renderings of a scene along user-defined camera trajectories from a single
monocular video, dubbed
video retaking, is a compelling but difficult problem in content creation and visual effects. Existing geometry-guided approaches reconstruct a 4D representation from the source
video and render it along the target trajectory to condition
video diffusion models. However, this guidance degrades as the target camera departs from the source trajectory, leaving newly revealed regions sparse or entirely missing. We propose SierpinskiCam, which addresses this limitation by augmenting geometry-based guidance with Sierpinski dome texture cues that contains rich trackable features even under large viewpoint changes. We further introduce a reference
video conditioning mechanism that appends source-
video tokens to the target-
token sequence and separates the two streams with negative RoPE indices, enabling appearance grounding without architectural modification or per-
video adaptation. Extensive experiments show that SierpinskiCam achieves significant gains in camera controllability,
geometric consistency, and
video quality across diverse and challenging retaking scenarios. Project page: https://hyelinnam.github.io/SierpinskiCam/.
Read the full abstract ▾
All authors Suttisak Wizadwongsa, Hyelin Nam, Supasorn Suwajanakorn, Jeong Joon Park
Computer Vision
Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins
Yiqing Shen, Hao Ding and Mathias Unberath
Text-to-
video retrieval in operating rooms (OR) is an enabling technology for OR safety, as it allows stakeholders to retrieve and inspect recordings of specific events. However, because the most safety-critical events may not follow the common structure, to unlock its full potential text-to-
video retrieval must be able to handle implicit queries that require
reasoning to identify the right
video (e.g., the step right before clipping). However, existing methods rely on global
embeddings that cannot reason over such queries. We propose OR3, a text-to-
video retrieval method that converts clips into action-driven digital twins (ActDTs), grouping concurrent subject-action-object triplets under non-overlapping temporal intervals. Moreover, rather than cross-modal matching through paired encoders, OR3 performs imagination-based
retrieval where an LLM generates hypothetical ActDTs from queries. This enables intra-modal matching via a single encoder trained with ActDT-tailored hard negatives. Finally, evidence-grounded refinement revises imagined ActDTs based on discrepancies with top candidates to capture procedure-specific patterns. We construct a
benchmark from MM-OR with 276 implicit queries across four
reasoning categories over 386 clips from robotic knee procedures. OR3 achieves 57.6 R@1 and 77.3 R@5, outperforming the strongest baseline. These results demonstrate that OR3 enables fine-grained discrimination between visually similar OR
video clips through temporal action
reasoning.
Read the full abstract ▾
Computer Vision
Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration
Xiwen Wei, Mark Nutter, Madhusudhanan Srinivasan et al. (+1 more)
Unified
multimodal models (UMMs) have recently emerged as a promising paradigm for integrating
multimodal understanding and generation within a single autoregressive
transformer. However, during
multimodal instruction tuning, these models often exhibit pronounced modality imbalance:
language gradients dominate optimization, thus leading to lower image generation quality, especially under parameter-efficient fine-tuning such as LoRA. In this work, we systematically analyze modality imbalance in LoRA-based fine-tuning of UMMs for interleaved text-image generation. We show that
vision modality performance degrades substantially more than text modality performance when compared to unimodal counterparts, and that modality-specific gradients can differ by orders of magnitude across various tasks and layers. Motivated by this observation, we reformulate the
multimodal instruction tuning as a bi-objective optimization problem and propose Pareto LoRA, a Pareto-optimal gradient integration strategy that balances the text and image objectives by modulating the gradient direction and strength. Experiments on the CoMM
benchmark with Emu2 demonstrate that Pareto LoRA consistently improves
multimodal generation balance, achieving up to 44.9% gains in perceptual image quality over vanilla LoRA while maintaining comparable text performance.
Read the full abstract ▾
All authors Xiwen Wei, Mark Nutter, Madhusudhanan Srinivasan, Radu Marculescu
Computer Vision · Cross-listed from eess.IV
Phenotyping TPF via Self-Supervised Learning: A Label-Agnostic Framework with Expert Validation
Miral Elnakib, Muhammad Saad and Ahmad Al-Kabbany
The full potential of artificial intelligence in tibial plateau fracture characterisation remains unrealised, constrained by a fundamental dependency on labelled
datasets whose consistency cannot be guaranteed: conventional classification schemes such as Schatzker and AO/OTA suffer from inter-observer variability, causing supervised models to learn human disagreement rather than stable fracture morphology. We design, implement, and validate a label-agnostic framework that eliminates this constraint by learning fracture representations directly from imaging data without observer-assigned labels. A RadImageNet-pretrained ResNet-50 encoder is fine-tuned on 154 cleaned knee radiographs using the SimCLR contrastive objective, preceded by a data cleaning protocol and followed by UMAP dimensionality reduction and k-means clustering to discover four imaging-derived phenotypes. Phenotype validity is assessed through a blinded expert review protocol administered to two independent clinicians. The four phenotypes demonstrate robust stability (bootstrap ARI = 0.319 +/- 0.041), strong internal cohesion (silhouette = 0.511), and coherence ratings of 3-5/5 from both reviewers under blinded conditions; one phenotype was unanimously identified as exhibiting comminution – a high-complexity feature isolated without any supervisory signal. Inter-partition comparison against Schatzker labels yields ARI = 0.013, confirming orthogonality to conventional classification boundaries. Notably, expert reviewers anchored to established classification vocabularies perceived imaging-derived groups as heterogeneous precisely where Schatzker
alignment was lowest, suggesting that Schatzker-trained perception and label-agnostic
embedding geometry measure orthogonal dimensions. These findings establish label-agnostic SSL phenotyping as a reproducible and clinically interpretable complement to conventional classification.
Read the full abstract ▾
Computer Vision
Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA
Yiqing Shen, Han Zhang and Mathias Unberath
Surgical video question answering requires multi-step
reasoning across semantic, spatial, and temporal dimensions. Existing methods architecturally compress
videos into discrete
token representations and couple visual perception with
reasoning. This approach fragments continuous spatial-temporal relationships and has been shown to restrict multi-step
reasoning capabilities. We introduce a reinforcement learning (RL) framework that trains large
language models (LLMs) to decouple perception from
reasoning by operating over digital twin representations constructed from
surgical foundation models. Additionally, we introduce hierarchical representations across frame, temporal window, and procedure levels with probabilistic uncertainty estimates. Finally, we propose a novel reward that combines format validation with accuracy assessment through clinical plausibility evaluation and uncertainty-aware calibration for training. To demonstrate the capabilities of this approach, we introduce REAL-Colon-Reason, a colonoscopic
benchmark with 2000 question-answer pairs across three complexity levels. We achieve state-of-the-art performance on REAL-Colon-Reason and two existing
surgical VideoQA
benchmarks REAL-Colon-VQA and EndoVis18-VQA.
Read the full abstract ▾
Computer Vision
Pulling The REINS: Training-Free Safety Alignment of Video Diffusion Models via Representation Steering
Rohit Kundu, Arindam Dutta, Sarosij Bose et al. (+2 more)
Open-weight
video diffusion models can generate photorealistic unsafe content, from violence to misinformation, yet existing defenses either require expensive safety fine-tuning that degrades general capability, or apply external filters that are trivially bypassed by
adversarial prompts. We present REINS (REpresentation-space INference-time Safety steering), a training-free method that aligns
video diffusion models at inference time by steering their internal representations toward safe generation. Our key finding is that safety-relevant structure is linearly encoded in the hidden-state activations of
video diffusion transformers, and a single direction, discovered via Supervised PCA on binary safety labels, suffices to separate safe from unsafe generation trajectories. At inference, adding this direction to hidden states at an intermediate
transformer layer redirects generation from harmful content to semantically related safe alternatives, with no weight updates, no concept enumeration, and negligible computational overhead. Through mechanistic analysis, we reveal that while safety information accumulates monotonically with
transformer depth, steering effectiveness peaks at intermediate layers (~50% depth), exposing a fundamental tradeoff between information availability and downstream propagation capacity. We evaluate REINS across 9
video diffusion models, multiple parameter scales (1.3B-5B), and both text-to-
video and image-to-
video generation, to our knowledge, the broadest safety evaluation suite in the
video generation literature.
Read the full abstract ▾
All authors Rohit Kundu, Arindam Dutta, Sarosij Bose, Athula Balachandran, Amit K. Roy-Chowdhury
Computer Vision · Cross-listed from cs.RO
Contrastive Action-Image Pre-training for Visuomotor Control
Yuvan Sharma, Dantong Niu, Anirudh Pai et al. (+16 more)
Existing
vision encoders for robotics face a fundamental bottleneck: robotic
datasets lack the scale necessary for large-scale pre-training. Prior work circumvents this data scarcity by turning to internet-scale image and
language data or
egocentric human
video. While these models show promise, neither paradigm learns from paired
vision and action data, which downstream visuomotor control policies require. However,
robot trajectories, the most direct source of this paired signal, are not available at pre-training scale, motivating us to extract action signals from abundant human
video instead. To this end, we introduce CAIP (Contrastive Action-Image Pre-training), a
vision encoder that treats human hand poses from large-scale
egocentric video as a proxy for end-effector actions. By extracting 3D hand keypoints, a representation that aligns naturally with downstream
robot action spaces, CAIP learns a unified action-image representation through a contrastive objective. Leveraging 32,041 hours of
egocentric human
video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art
vision encoders including DINOv2, SigLIP, MVP, and R3M. Evaluated on a challenging real-world dexterous manipulation setup using Dexmate Vega and Sharpa Wave hands, CAIP yields performance gains of more than 30% on tasks involving folding, pouring, and fine-grained manipulation. Our results show that our method of contrastive action-centric pre-training yields a scalable path to achieving robust visual representations better suited for physical interaction.
Read the full abstract ▾
All authors Yuvan Sharma, Dantong Niu, Anirudh Pai, Zekai Wang, Zhuoyang Liu, Baifeng Shi, Stefano Saravalle, Boning Shao, Ruijie Zheng, Jing Wang, Konstantinos Kallidromitis, Yusuke Kato, Fabio Galasso, Yuke Zhu, Danfei Xu, Linxi "Jim" Fan, Jitendra Malik, Trevor Darrell, Roei Herzig
Computer Vision
GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligence
Maram Hasan, Aman Verma, Savitra Roy et al. (+5 more)
Remote-sensing
vision-
language models (RS-VLMs) have advanced Earth-observation analysis toward visual interpretation and instruction-following, yet fall short of operational geo-intelligence, which demands tool-grounded spatial
reasoning and structured, evidence-backed decisions. We introduce GeoDisaster, an operational geospatial disaster
reasoning benchmark with 2,921 verified instances across 43 question types and five task families: deforestation monitoring, multi-hazard analysis, building-damage assessment, flood-safe routing, and Sentinel-1 SAR flood monitoring. Instances integrate heterogeneous EO/GIS evidence-optical and SAR imagery, raster masks, vector geometries, road networks, and exposure layers-spanning hazard detection, damage assessment, exposure estimation, and diagnostic report generation. Ground-truth answers are grounded in executable geospatial workflows and deterministic consistency checks, removing the need for
language-model annotation. We further propose an orchestrated multi-
agent framework with 18 disaster-oriented tools, where role-specialized
agents coordinate through explicit execution contracts, aligned via Role-Contract Expectation
Alignment (RCEA): failure-aware supervised fine-tuning combined with contract-grounded reinforcement learning over dense step-level signals. Experiments show that GeoDisaster challenges existing RS-VLMs and agentic systems, while RCEA improves tool use, evidence grounding, state consistency, and decision generation.
Read the full abstract ▾
All authors Maram Hasan, Aman Verma, Savitra Roy, Hariseetharam Gunduboina, Daksh Jain, Muhammad Haris Khan, Subhasis Chaudhuri, Biplab Banerjee
Computer Vision
Landsat-Sentinel-2 Algal Bloom Mapping Using Vision Transformers: Model Description, Implementation, and Examples
Thainara Lima and Vitor Martins
Coastal algal bloom monitoring requires frequent, spatially detailed, and globally consistent observations, provided by Landsat-8/9 and Sentinel-2 A/B/C. Together, these missions offer over a decade of medium-resolution multispectral imagery with near-global coverage every 2-3 days, enabling the detection of fragmented bloom structures not resolvable by coarse ocean-color sensors. However, their use in aquatic environments remains challenging due to limited spectral coverage and a lack of harmonized reflectance products. As an alternative to traditional bio-optical methods, deep learning-based image classification offers a data-driven approach that can overcome many of these limitations. This study presents the first successful implementation of
vision transformer-based coastal algal bloom mapping using 30-m Landsat-Sentinel-2 images. A globally distributed bloom patch
dataset was generated across bloom-prone coastal hotspots worldwide. Four
transformer-based architectures were compared against a standard convolutional baseline for fine-scale bloom detection, and assessed under different optical water types and atmospheric and surface conditions. All deep learning models showed strong capabilities in detecting floating bloom areas, with omission and commission errors of 8-65%. Under cloud and glint stress in a time series, the Swin
Transformer outperformed traditional spectral-index approaches, which produced widespread false positives, effectively avoiding cloud- and glint-affected pixels. Comparisons with MODIS-derived products further highlighted the benefits of higher spatial resolution in detecting fragmented and irregularly affected blooms. Our findings support deep learning as a reliable tool for medium-resolution, consistent monitoring of floating algal blooms in dynamic coastal environments.
Read the full abstract ▾
Computer Vision
Beyond Benchmarks: Continuous Edge Inference for Fine-Grained Roadside Perception
Aditya Mishra and Haroon Lone
Continuous AI inference on resource-constrained edge hardware introduces deployment effects that are largely invisible to conventional
benchmark evaluation, including temporal instability in streaming
video, thermal throttling under sustained load, and workload-dependent performance variability. We present Edge-TSR, a deployment-oriented continuous edge inference system for sustained roadside perception on the NVIDIA Jetson Orin Nano. Edge-TSR integrates detection, tracking, fine-grained classification, and a lightweight track-aware temporal stabilization mechanism that improves streaming inference consistency with negligible computational overhead. Our central finding is that
benchmark-centric evaluation systematically overstates deployed edge inference performance. Across three state-of-the-art baselines, we observe consistent 20-30% relative degradation when transitioning from static-image evaluation to real-world streaming deployment. Edge-TSR addresses this gap through temporal inference stabilization, recovering up to 10.16% classification accuracy over per-frame inference baselines while maintaining sustained real-time performance under continuous operation. We evaluate the complete system under diverse real-world deployment conditions, jointly characterizing inference quality, latency, throughput, and thermal behavior during long-duration operation. A 55-minute vehicular deployment over a 26 km route demonstrates sustained operation at 16.18 FPS within safe thermal limits on a single embedded device without cloud offload. Our findings show that deployment-aware evaluation and temporal inference stabilization are necessary components of continuously operating edge AI systems intended for real-world sensing deployments. We release a sample annotated streaming
video evaluation
dataset and full system implementation to support reproducible deployment-centric evaluation.
Read the full abstract ▾
Computer Vision
Quantum Enchanced Multi-Scale CNN with Bi-directional Mamba for Crop Field Analysis
Mohammad Salman Khan, Ehsan Atoofian and Saad B. Ahmed
Hyperspectral image (HSI) crop analysis is essential for precision agriculture because it captures rich spectral and spatial information for accurate crop monitoring and assessment. However, HSI classification remains challenging due to high spectral dimensionality, spatial complexity, class imbalance, and limited labeled samples. To address these challenges, this paper proposes a BiSpectral Mamba-based framework that combines multi-scale convolutional feature extraction, spectral
attention, bidirectional state-space modeling, and
quantum-inspired learning. A multi-scale CNN backbone first extracts hierarchical spatial-spectral representations through feature fusion across multiple resolutions. A spectral
attention mechanism then emphasizes informative bands while suppressing redundant and noisy channels. The refined features are processed by a BiSpectral Mamba module that captures long-range dependencies in both forward and backward directions by modeling hyperspectral feature maps as sequential
tokens. In addition, class-weighted optimization and feature fusion strategies are incorporated to improve training stability and mitigate class imbalance. Experimental evaluation on the UAVHSI-Crop
dataset demonstrates the effectiveness of the proposed framework, achieving an overall accuracy of 84.83%. The results show that integrating convolutional,
attention-based, and state-space modeling components enables robust spatial-spectral feature learning for crop classification. The proposed framework also shows potential for broader agricultural and remote sensing applications, including crop disease detection, yield prediction, and soil moisture estimation, while highlighting the effectiveness of structured state-space and
quantum-inspired architectures for hyperspectral image analysis.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priors
Vanshali Sharma, Andrea M. Bejar, Halil Ertugrul Aktas et al. (+4 more)
Recent advances in
multimodal learning, including large
language models (LLMs) and
vision-
language models (VLMs), have demonstrated strong adaptability to natural images. However, extending their use to the medical domain, particularly for volumetric (3D) images, is challenging due to high computational complexity, volumetric dependencies and the semantic gap between visual features and clinical terminology. Naively fine-tuning LLMs on limited medical data often leads to overfitting and clinical
hallucination, where linguistic fluency is prioritized over clinical factuality. In this study, we investigate parameter-efficient adaptation strategies for volumetric CT report generation and introduce RAD3D-Prefix, a lightweight diagnostic-prior conditioning framework that minimizes the need for extensive parameter training. This module integrates image
embeddings with multi-label diagnostic classification logits, preserving critical clinical details while bridging the semantic gap. By keeping the LLM frozen, our method requires minimal trainable parameters and mitigates the risk of overfitting on small, domain-specific
datasets. Through a systematic study spanning LLMs from 96.1M to 1.6B parameters, we find that fine-tuning is most beneficial for smaller LLMs, whereas freezing larger (~1B+ LLMs and training only lightweight projection layers provides a superior trade-off between performance, generalization, and computational efficiency. Across multiple automatic metrics and a clinical reader study, RAD3D-Prefix outperforms comparable parameter-efficient baselines and demonstrates strong out-of-domain generalization while using substantially fewer trainable parameters than fully fine-tuned alternatives.
Read the full abstract ▾
All authors Vanshali Sharma, Andrea M. Bejar, Halil Ertugrul Aktas, Quoc-Huy Trinh, Debesh Jha, Gorkem Durak, Ulas Bagci
Computer Vision
Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation
Prabhjot Singh, Bhushan Pawar, Madhu Reddiboina et al. (+1 more)
Current
multilingual evaluations for
Vision-
Language Models (VLMs) assume a one-to-one mapping between
language and orthography, overlooking billions of users of multi-script
languages. We introduce PuMVR (Punjabi
Multimodal Visual
Reasoning), a
benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts: Gurmukhi, Shahmukhi, and Roman. Evaluating 10 state-of-the-art VLMs, we expose a substantial and systematic Script Gap. Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%. Crucially, visual input boosts absolute performance uniformly yet does not close the orthographic gap. Furthermore, cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation. Supported by McNemar tests across all script pairs, our findings demonstrate that current "
multilingual" VLMs are not truly multi-script. We propose the Script Consistency Rate (SCR), which falls as low as 24.8% on our
benchmark, as a mandatory metric for script-agnostic evaluation to ensure equitable AI access. Data and code are available at: https://github.com/prabhjotschugh/Not-Truly-
Multilingual-PuMVR.
Read the full abstract ▾
All authors Prabhjot Singh, Bhushan Pawar, Madhu Reddiboina, Rajvee Sheth
Computer Vision · Cross-listed from cs.RO
Human Universal Grasping
Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu et al. (+5 more)
Humans can grasp objects effortlessly, whereas multi-fingered
robots are far from this level of generality. We argue that the most natural source of
robot grasping data is from humans, who pick up thousands of objects every day. We present HUG, a flow-matching model that generates diverse human grasps for any user-specified object in a single RGB-D image captured from a stereo camera. Using smart glasses, we first collect 1M-HUGs, an
egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. Next, to model the distribution of natural human grasps, our novel flow-matching model fuses RGB and depth observations to output a grasp parameterized by wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various
robot hands, enabling zero-shot grasping in everyday scenes. To standardize evaluation, we build a new simulated
benchmark, HUG-Bench, of 90 unseen objects from five
geometric categories and various sizes, with metric-scale 3D meshes. We evaluate HUG in the real world on the 30-object test set of HUG-Bench across multiple stereo cameras,
robot embodiments, and household environments. HUG outperforms the state-of-the-art grasping baselines by +23% and +34% on our challenging object set. Code, data,
benchmark, checkpoints, and an interactive demo are released on our website: https://grasping.io/
Read the full abstract ▾
All authors Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan, Irmak Guzey, David Fouhey, Dandan Shan, Lerrel Pinto
Computer Vision · Cross-listed from cs.CL
Context-Aware RL for Agentic and Multimodal LLMs
Peiyang Xu, Bangzheng Li, Sijia Liu et al. (+4 more)
Large
language models (LLMs) often fail when answering requires identifying a small but decisive piece of evidence within a long or complex context, such as a single line in a tool trace or a subtle detail in an image. We propose ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon
reasoning and
multimodal performance through an indirect auxiliary objective. Instead of supervising only the final answer, ContextRL presents the model with a query, an answer, and two highly similar contexts, and rewards it for selecting the context that supports the query–answer pair, thereby encouraging fine-grained grounding. We construct contrastive context data in two domains: for coding
agents, trajectories serve as contexts, yielding 1k pairs built via condition filtering; for
multimodal reasoning, images serve as contexts, yielding 7K pairs built via generative editing and similarity search. ContextRL achieves average gains of +2.2% over standard GRPO on 5 long-horizon
benchmarks, and +1.8% across 12 diverse visual question answering
benchmarks. To disentangle the effect of the proposed objective from that of additional data, we compare against data-augmentation baselines that repurpose the same contrastive contexts as standard query–context–answer examples. These baselines provide little to no improvement, showing that the gains arise from the proposed context-selection objective rather than from the contrastive data alone.
Read the full abstract ▾
All authors Peiyang Xu, Bangzheng Li, Sijia Liu, Karthik R. Narasimhan, Pramod Viswanath, Prateek Mittal, Xingyu Fu
Computer Vision
BRDFusion: Physics Meets Generation for Urban Scene Inverse Rendering
Yi-Ruei Liu, Jie-Ying Lee, Zheng-Hui Huang et al. (+2 more)
Inverse rendering of urban scenes from captured
videos enables numerous applications, including content creation and autonomous driving simulation. Physically-based rendering methods follow and control lighting physics, but suffer from reconstruction and rendering artifacts. While generative models produce realistic
videos, they offer limited consistency and controllability. We present BRDFusion, a unified framework that combines two complementary models for inverse and forward rendering. Specifically, BRDFusion recovers explicit, consistent scene properties with physical modeling and alleviates optimization ambiguity with generative priors. During forward rendering, the physical model provides controllable rendering from the scene configuration, and the generative model denoises and fixes artifacts. Therefore, our method produces high-quality
videos while allowing precise control, outperforming baselines in real and synthetic scenes. Moreover, BRDFusion supports novel-view relighting, night simulation, and dynamic object insertion/editing. Project page: https://shigon255.github.io/brdfusion-page/
Read the full abstract ▾
All authors Yi-Ruei Liu, Jie-Ying Lee, Zheng-Hui Huang, Yu-Lun Liu, Chih-Hao Lin
Computer Vision · Cross-listed from cs.LG
Exact Posterior Score Estimation for Solving Linear Inverse Problems
Abbas Mammadov, Ozgur Kara, Kaan Oktay et al. (+5 more)
Diffusion and flow-based models learn powerful data priors by training a denoiser to reverse Gaussian corruption. To use this prior to solve a linear inverse problem, one needs to sample from the posterior, but the score that the prior provides is the unconditional score, not the posterior score. Existing methods either steer a fixed pretrained denoiser with approximate measurement-matching corrections, or train a conditional restoration model that abandons the denoising structure of the prior. We derive the exact posterior score in closed form for linear Gaussian inverse problems under general Gaussian interpolants, and show that posterior sampling reduces to a denoising problem at an operator-dependent shifted pivot under an anisotropic noise covariance. We turn this identity into Exact Posterior Score (EPS), a denoising training objective that preserves the input/output structure of standard pretraining and can therefore be trained from scratch or fine-tuned from a pretrained denoiser. At inference, EPS uses the same sampler as the underlying backbone, with no likelihood gradients or projections. We evaluate EPS on five linear inverse problems across FFHQ and ImageNet, where it outperforms training-free and training-based baselines on fidelity, perceptual, and distributional metrics, while using roughly an order of magnitude fewer denoiser evaluations than gradient-based posterior samplers.
Read the full abstract ▾
All authors Abbas Mammadov, Ozgur Kara, Kaan Oktay, Iskander Azangulov, Adil Kaan Akan, Hyungjin Chung, James Matthew Rehg, Yee Whye Teh
Computer Vision · Cross-listed from cs.RO
Geometric Action Model for Robot Policy Learning
Jisang Han, Seonghu Jeon, Jaewoo Jung et al. (+7 more)
Generalist
robot policies must follow user instructions while
reasoning about how objects, cameras, and
robot actions interact in the 3D physical world. Recent
vision-
language-action models (VLAs) and
video world-action models (WAMs) inherit strong semantic or temporal priors from large-scale foundation models, but they still operate primarily on 2D image frames or 2D-derived latent spaces, leaving implicit the 3D geometry required for contact-rich manipulation. We propose the
Geometric Action Model (GAM), a
language-conditioned manipulation policy that directly repurposes a pretrained
geometric foundation model (GFM) as a shared substrate for perception, temporal prediction, and action decoding. GAM splits the GFM at an intermediate layer: the shallow layers serve as an observation encoder, and a causal future predictor inserted at the split layer forecasts future latent
tokens conditioned on
language, proprioception, and action history. The predicted future
tokens are then routed through the remaining GFM blocks for feature propagation and decoding, allowing a single backbone to produce both future geometry and actions. This design equips the GFM with
language-conditioned temporal world modeling through minimal architectural modification while preserving its rich
geometric priors. Across a broad suite of simulation and real-
robot manipulation
benchmarks, GAM is more accurate, more robust, faster, and lighter than current foundation-model-scale baselines.
Read the full abstract ▾
All authors Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, Sunghwan Hong
Computer Vision · Cross-listed from cs.RO
R2RDreamer: 3D-aware Data Augmentation for Spatially-generalized 2D Manipulation Policies
Xiuwei Xu, Haowen Sun, Angyuan Ma et al. (+7 more)
Spatial generalization is critical for imitation-learned manipulation policies, but achieving it typically requires scaling demonstrations across diverse object poses,
robot configurations, and camera viewpoints. Data augmentation from a few source demonstrations offers a practical alternative to costly real-world collection. Simulation-based augmentation can create controllable variation, but requires complex environment and object setup and may introduce a sim-to-real gap. Recent real-to-real methods avoid these issues by jointly editing 3D observations and action trajectories from real demonstrations, yet they still rely on strong 3D scene parsing and geometry completion, and often produce observations tailored to 3D pointcloud policies rather than RGB-based 2D policies. We propose R2RDreamer, a real-to-real demonstration augmentation framework that preserves the
geometric consistency of 3D action-observation editing while moving visual completion to 2D
video space. Specifically, R2RDreamer first performs lightweight 3D augmentation by editing incomplete object pointclouds and end-effector trajectories in a shared 3D frame; it then projects the edited scene into masked image-space control
videos with occlusion-aware
reasoning and uses a dense-control image-to-
video model to complete temporally coherent RGB observations. Experiments on spatially shifted manipulation tasks with both 2D
diffusion-style policies and
vision-
language-action policies show that R2RDreamer improves spatial generalization from limited source demonstrations, with analyses validating the contributions of 3D editing, occlusion-aware projection, and
video completion.
Read the full abstract ▾
All authors Xiuwei Xu, Haowen Sun, Angyuan Ma, Yiwei Zhang, Zhenyu Wu, Xiaofeng Wang, Bingyao Yu, Zheng Zhu, Jie Zhou, Jiwen Lu
Computer Vision
The Importance of Phase in Neural Representations: An Internal Oppenheim-Lim Test of Image Classifiers
Alper Y{\i}ld{\i}r{\i}m
Oppenheim and Lim (1981) showed that natural images stay recognizable when reconstructed from their
Fourier phase alone, while the magnitude carries little of their identity. We ask whether trained image classifiers reproduce this asymmetry inside their hidden layers, and we test it causally: given two images, we transplant the phase of one onto the magnitude of the other at a chosen layer and record which image the prediction follows. In PRISM2D, GFNet, and ViT-B/16 the prediction follows the phase or sign donor, and deleting all image-specific magnitude barely moves accuracy, so identity rides on phase while image-specific magnitude is largely dispensable to the readout. ResNet-50 at first seems to break the pattern, because transplanting sign after its ReLUs does nothing; a fair intervention before the ReLU reveals a strong latent sign code in the late blocks, and a DC-only control shows the readout consumes a channel-wise spatial average. Controls rule out the trivial case in which magnitude simply stops depending on the image. The architectures therefore share a phase/sign identity code but expose it in different bases, set by rectification and readout geometry, which gives a mechanistic account of the texture–shape gap between CNNs and
attention models.
Read the full abstract ▾
Computer Vision
Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation
Jie Zhang, Xiaoyue Chen, Anzhe Chen et al. (+36 more)
We introduce Qwen-RobotWorld, a
language-conditioned
video world model for
embodied intelligence. With natural
language as a unified action interface, it predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor
navigation, and human-to-
robot transfer. This unified formulation provides three promising application directions: synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and
language-guided planning signals for downstream
robot control. This is achieved through a three-part design: a) Double-Stream MMDiT with MLLM Action Encoding, where a 60-layer double-stream
diffusion transformer couples frozen Qwen2.5-VL semantics with
video-VAE latents through layer-wise joint
attention; b)
Embodied World Knowledge (EWK), an 8.6M
video-text corpus (200M+ frames) with action-
language mapping over 20+ embodiments and 500+ action categories; and c) General+Expert Progressive Curriculum, a two-stage training strategy that first learns general visual priors and then injects
embodied specialization under a shared
language interface. Extensive results show strong competitiveness: ranks 1st overall on EWMBench and DreamGen Bench, outperforms all open-source models on WorldModelBench and PBench. Additional zero-shot analyses on RoboTwin-IF
benchmark further support robust generalization and multi-view consistency.
Read the full abstract ▾
All authors Jie Zhang, Xiaoyue Chen, Anzhe Chen, Dayiheng Liu, Deqing Li, Gengze Zhou, Hale Yin, Haoqi Yuan, Haoyang Li, Jiahao Li, Jiazhao Zhang, Jingren Zhou, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Pei Lin, Qihang Peng, Shengming Yin, Tianhe Wu, Tianyi Yan, Xiao Xu, Yan Shu, Yanran Zhang, Ye Wang, Yi Wang, Yilei Chen, Yixian Xu, Yiyang Huang, Yuxiang Chen, Zekai Zhang, Zhendong Wang, Zixing Lei, Zhixuan Liang, Zihao Liu, Zikai Zhou, Chenxu Lv, Xiong-Hui Chen, Chenfei Wu
Computer Vision
MeshLoom: Feed-Forward Non-Rigid Registration of Mesh Sequences
Jianqi Chen, Jiraphon Yenphraphai, Xiangjun Tang et al. (+4 more)
We present MeshLoom, a feed-forward registration network that directly reconstructs vertex deformations across mesh sequences. Our approach advances non-rigid registration beyond existing models, which are typically constrained by costly per-instance optimization, narrow object categories, pairwise-only inputs, or merely intermediate outputs. The network is simple and efficient, registering multiple meshes within seconds. At its core lies a
topology-aware encoder–decoder design. Specifically, we first introduce a
topology-aware point representation that encodes the anchor (reference) mesh's
topology into its per-vertex features. This representation strengthens the network's understanding of the anchor-mesh geometry and disambiguates points that are Euclidean-close yet geodesically distant. We then propose a multi-modal encoder that fuses this anchor-mesh representation with complementary cues from each frame, such as shape latents and image features. These multi-source signals are compressed into a compact global motion
embedding that captures dense inter-frame correspondence. A lightweight decoder then queries this global
embedding with the anchor-mesh point representation, retrieving per-vertex deformations at target timestamps. Through extensive experiments across diverse motions and object categories, we show that MeshLoom achieves state-of-the-art results on non-rigid registration. In addition, we find that our global
embedding-then-query paradigm naturally enables the network to generate deformations at intermediate timestamps, which extends MeshLoom to motion interpolation and mesh morphing. Project page: https://meshloom.github.io/ .
Read the full abstract ▾
All authors Jianqi Chen, Jiraphon Yenphraphai, Xiangjun Tang, Sergey Tulyakov, Chaoyang Wang, Peter Wonka, Rameen Abdal
Computer Vision
FusionRS: A Large-Scale RGB-Infrared-Style Remote Sensing Dataset for Cross-Modal Vision-Language Learning
Jiaju Han, Ben Zhang, Xuemeng Sun et al. (+8 more)
Remote sensing
vision-
language models have advanced Earth observation, but available large-scale
vision-
language resources remain RGB-centered, leaving complementary infrared information underexplored. Infrared observations provide distinctive intensity structures, object boundaries, and illumination-invariant cues that complement conventional RGB imagery, yet large-scale RGB-infrared-text resources remain scarce. We introduce FusionRS, the first large-scale RGB-infrared-style-text
dataset for controlled dual-modal remote sensing
vision-
language learning. It contains 600,000 spatially aligned pairs created by translating diverse public RGB remote sensing images into infrared-style counterparts. Each pair retains a conventional scene caption, and a curated subset adds 45,913 IR-aware captions describing observable intensity, contrast, texture, and structure while preserving scene semantics. We train CLIP-style models for RGB-infrared-style-text
alignment and adapt a generative
vision-
language model with mixed task-conditioned caption supervision. Evaluation covers cross-modal
retrieval, scaling and supervision ablations, sensor-captured transfer, and strictly held-out captioning and VQA. FusionRS substantially improves RGB-infrared-style
alignment and infrared-to-text
retrieval over RGB-only and non-IR-aware settings. Ablations show that IR-aware captions improve task-conditioned infrared description, demonstrating the value of modality-specific supervision. FusionRS provides a scalable foundation for controlled RGB-infrared remote sensing
vision-
language learning.
Read the full abstract ▾
All authors Jiaju Han, Ben Zhang, Xuemeng Sun, Qike Zhang, Yuxian Dong, Dingyi Lu, Chengyin Hu, Luwei Yang, Fengyu Zhang, Yiwei Wei, Jiujiang Guo
Computer Vision
ActiveSAM: Fast and Accurate Open-Vocabulary Semantic Segmentation with Frozen SAM 3
Tran Dinh Tien and Zhiqiang Shen
Segment Anything Model 3 (SAM 3) provides a strong frozen backbone for concept-prompted
segmentation, but applying it directly to open-vocabulary semantic
segmentation (OVSS) is inefficient: full-resolution decoding is typically run over the entire
dataset vocabulary, whereas each image contains only a small active subset of classes. We introduce ActiveSAM, a training-free inference framework that turns SAM 3 into an active-vocabulary segmenter. ActiveSAM first canonicalizes and expands class
prompts, then uses evidence-proportional grounding to estimate an image-conditioned active set from a low-resolution presence preview. Only retained
prompts receive full-resolution mask prediction, using bucketed
prompt multiplexing with the frozen SAM 3 decoder. The preview stage uses only class-presence evidence and skips unnecessary
segmentation-head computation. To resolve overlapping concept responses, exclusive concept decoding compares each pixel's joint score vector with class signatures estimated once per vocabulary from unlabeled images. ActiveSAM requires no weight updates, no oracle class-presence labels and no per-
dataset hyperparameter tuning. Across eight OVSS
benchmarks, ActiveSAM improves the speed-accuracy tradeoff of training-free open-vocabulary semantic
segmentation, outperforming the current state-of-the-art SegEarth-OV3 by +2.1 mIoU on average while running much faster, with 7.3-12.2x speedups on large-vocabulary
datasets. ActiveSAM also achieves the highest accuracy under image corruptions that simulate real-world distribution shift, making it well-suited for deployment in noisy-input domains such as autonomous driving and
embodied AI. Code is available at https://github.com/VILA-Lab/ActiveSAM
Read the full abstract ▾
Computer Vision
DreamX-World 1.0: A General-Purpose Interactive World Model
DreamX Team, Yancheng Bai, Rui Chen et al. (+20 more)
DreamX-World 1.0 is a general-purpose interactive text/image-to-
video world model for controllable long-horizon generation. It supports camera
navigation, revisits to previously observed regions, and promptable events across photorealistic, game-style, and stylized domains. Our data engine combines camera-accurate Unreal Engine rendering, action-rich gameplay recordings, and real-world
videos with recovered camera geometry. For camera control, we introduce E-PRoPE, a lightweight variant of projective positional encoding that retains PRoPE's projective camera geometry while applying camera-aware
attention to spatially reduced
tokens. We convert a bidirectional
video generator into a few-step autoregressive world model using causal forcing, DMD-style
distillation, and long-rollout training. Training on self-generated long-horizon contexts exposes the model to its own generated history and reduces the style and color drift that accumulates across autoregressive chunks.
Memory-Conditioned Scene Persistence retrieves earlier views through camera-geometry-based
retrieval, while residual recycling makes the conditioning path less sensitive to imperfect
memory latents. Event Instruction Tuning adds composable event control, and reinforcement learning
alignment recovers camera control and visual quality after
distillation. With mixed-precision DiT execution, residual reuse, 75%-pruned VAE decoding, and asynchronous pipeline parallelism, DreamX-World 1.0 reaches up to 16\,FPS on eight RTX\,5090 GPUs. On our 5-second basic evaluation, DreamX-World 1.0 achieves a camera-control score of 73.75 and an overall score of 84.76, outperforming HY-WorldPlay 1.5 and LingBot-World in overall score, which achieve 80.79 and 80.45, respectively.
Read the full abstract ▾
All authors DreamX Team, Yancheng Bai, Rui Chen, Xiangxiang Chu, Rujing Dang, Hao Dou, Bingjie Gao, Qiwen Gu, Siyu Hong, Jiachen Lei, Geng Li, Jifan Li, Ruimin Lin, Qingfeng Shi, Bingze Song, Lei Sun, Jing Tang, Ruitian Tian, Jun Wang, Jiahong Wu, Pengfei Zhang, Shen Zhang, Jiashu Zhu
Computer Vision
A Multi-Center Benchmark for Abdominal Disease Diagnosis and Report Generation from Non-Contrast CT
Mariam Elbakry, Aliaa Sayed Sheha, Salma Hassan Tantawy et al. (+5 more)
Multiphasic contrast-enhanced CT (CECT) is widely used for abdominal lesion characterization, yet it carries inherent risks of contrast-induced nephropathy, escalates acquisition burden, and heavily contributes to radiologist workload. To address these challenges, we introduce a novel multi-center
benchmark for multi-organ abdominal disease diagnosis and automated radiology report generation, which learns to synthesize contrast-enhanced findings from single-phase non-contrast CT (NCCT). To support this, we curated a large-scale
dataset of paired NCCT-CECT studies and their corresponding contrast-enhanced radiology reports from two centers, partitioned into internal sets and an external validation cohort. Under a unified evaluation protocol, we benchmarked five contemporary deep learning architectures encompassing chest-specific, abdomen-specific, and general-purpose
multimodal domains. Extensive experiments demonstrate that NCCT retains diagnostic signals, achieving an average multi-organ AUC of 69.1% on the internal cohort and 63.1% on the external cohort, respectively. By releasing this
dataset and standardized
benchmark publicly, this study aims to catalyze future research into safer, resource-efficient, and globally accessible contrast-free abdominal imaging workflows. Code is available at: https://github.com/xmed-lab/TriALS-Report.
Read the full abstract ▾
All authors Mariam Elbakry, Aliaa Sayed Sheha, Salma Hassan Tantawy, Aya Yassin, Concetto Spampinato, Karim Lekadir, Xiaomeng Li, Marawan Elbatel