Computer Vision · Cross-listed from cs.AI
Reason Before You Retrieve: Agentic Planning for Multi-modal RAG
Tianyu Yang, Shir Simon, Zhenzhen Li et al. (+2 more)
Multimodal retrieval-augmented generation (mRAG) aims to answer image-text queries with external knowledge, but most existing systems still retrieve directly from raw
multimodal input over a flat evidence space. This design often struggles with two key challenges: the
retrieval target is under-specified because the question intent must be grounded to the correct visual referent, and the search space is weakly structured, forcing semantically distinct evidence to compete in a single global ranking step. We propose MM-R2, a
multimodal agentic
retrieval framework that reasons before
retrieval by explicitly modeling both what to retrieve and where to search. MM-R2 first constructs an intent-grounded
retrieval state from the image-question pair, capturing the information need, grounded referent, and
retrieval constraints. It then performs
retrieval over a structured KnowledgeMap, where the
agent selects relevant
retrieval units before issuing grounded queries within them. To enable this capability, we build MM-R2-Traj, a large-scale trajectory
dataset of multi-step
retrieval processes, and adopt a two-stage post-training strategy with supervised fine-tuning and GRPO. Experiments on Infoseek and Encyclopedic VQA
datasets show that MM-R2 substantially outperforms strong baselines on answer accuracy while also yielding more interpretable and verifiable
retrieval trajectories.
Read the full abstract ▾
All authors Tianyu Yang, Shir Simon, Zhenzhen Li, Minhao Cheng, Xiangliang Zhang
Computer Vision · Cross-listed from physics.ins-det
Direct Time-of-Flight Measurement Accuracy Improvement With Perimeter-Gated SPADs
Md Sakibur Sajal, Hunter Guthrie, Zexi Liu et al. (+1 more)
Direct time of flight (dToF) measurements are susceptible to errors because of system-level and
circuit-level timing jitters. In addition, device-level uncertainty stemming from the dark noise of single-
photon avalanche diode (SPAD) contributes to the aggregated error. We demonstrate that perimeter gating can help reduce the device-level detection inaccuracy for SPAD devices by reducing the dark noise probability. Specifically, in this work, we developed a general framework to accurately estimate the dToF jitters stemming from different source levels and analyzed a counter-based time to digital converter (TDC)
circuit that are commonly used in such systems. We have also measured dToFs using a perimeter-gated SPAD (pg-SPAD) detector fabricated in a 0.35 $\mu$m standard CMOS process. Experimental results show that pg-SPADs can improve measurement accuracy in both free-running and time-gated operations.
Read the full abstract ▾
All authors Md Sakibur Sajal, Hunter Guthrie, Zexi Liu, Marc Dandin
Computer Vision · Cross-listed from cs.AI
iFLYTEK-Embodied-Omni Technical Report
Yuan Zhang, Jingfei Ni, Guanchen Lu et al. (+12 more)
General-purpose
embodied agents must understand
multimodal instructions, anticipate how their environment will evolve, and produce precise control actions over extended horizons. Existing approaches typically specialize in visual-
language reasoning,
video-based world modeling, or action generation, while cascaded pipelines that first synthesize future observations and then infer actions can introduce interface bottlenecks and compound prediction errors. We present iFLYTEK-
Embodied-Omni, a unified
multimodal foundation model that jointly models
vision(
videos and images),
language, and action within a single Omni framework. Its modality-specific visual-
language,
video-generation, and action-generation components communicate through shared
multimodal self-
attention. This design establishes
brain-cerebellum collaboration: the
vision-
language modeland
video generation model form a high-level
brain for instruction understanding, task planning, progress tracking, and future visual-state prediction, whereas the action generation modelserves as a low-level cerebellum that directly converts planned subgoals and shared
multimodal context into executable action chunks. To develop these capabilities, we combine action-annotated and action-free
embodied videos from human demonstrations and
robot interactions with
embodied reasoning,
embodied perception, and general-purpose image-text data to construct a comprehensive
dataset. We further adopt a four-stage strategy that progressively trains the VLM, VGM, and AGM before jointly fine-tuning the complete model.
Read the full abstract ▾
All authors Yuan Zhang, Jingfei Ni, Guanchen Lu, Shiqi Zhang, Qingshan Xu, Chi Liu, Xin Nie, Wenjie Xu, Lin Gao, Zhiyuan Cheng, Mingxin Zhou, Jiajia Wu, Diyuan Liu, Jia Pan, Chao Ji
Computer Vision · Cross-listed from cs.LG
Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models
Davie Chen
Text-to-image and
multimodal generative models are increasingly used to produce scientific figures such as mechanism diagrams, experimental-design schematics, conceptual frameworks, and graphical abstracts. Yet existing image-generation
benchmarks (e.g., GenEval, T2I-CompBench, DPG-Bench) evaluate natural images and measure compositionality, object counting, or photorealism. None of them measure what makes a generated scientific figure usable: correct and legible text labels, faithful depiction of entities and their relations, coherent diagrammatic structure, and adherence to disciplinary drawing conventions. We introduce SciDraw-Bench, a
benchmark of 32 structured scientific-figure generation tasks spanning eight figure types and ten disciplines, where each task pairs a natural-
language prompt with a machine-checkable specification of required labels, relations, components, conventions, and negative constraints. We propose a four-dimensional evaluation protocol: Text Fidelity (OCR-based label recall and character error rate), Semantic Correctness (
vision-
language-model judging against the specification), Structural Quality, and Convention Adherence, together with a meta-evaluation protocol and a preliminary inter-judge reliability analysis (human-rating validation is ongoing). We evaluate a domain-specific system, SciDraw AI, against representative general-purpose text-to-image models, and outline a code-to-figure baseline as a planned extension. In a pilot over all eight figure types, the domain-specific system substantially outperforms the general-purpose baselines on every dimension and figure type, with the largest gaps on semantic correctness and convention adherence; text fidelity remains the hardest dimension for all systems.
Read the full abstract ▾
Computer Vision
Enhancing Layer Interaction Using Key-Correlated Layer Attention
Jianlong Xiong, ChuanBo Xie, Le Yu et al. (+2 more)
Recent advances in network architecture design have introduced layer
attention to enhance inter-layer interactions. In such frameworks, each layer queries all preceding layers to establish cross-layer connections. However, layer
attention results in quadratic computational complexity with respect to network depth. To mitigate this issue, prior works have proposed Recurrent Layer
Attention (RLA) and linear
attention mechanisms, which suffer from static information updates and limited long-range cross-layer dependency modeling. To overcome these limitations, we propose Key-Correlated Layer
Attention (KCLA), inspired by our observation that Key representations in layer
attention exhibit high cosine similarity. KCLA achieves linear computational complexity while preserving dynamic information updates, directly derived from the foundational definition of layer
attention. Furthermore, KCLA maintains long-range cross-layer connections and features a fixed spatial complexity, independent of network depth. Empirical evaluations demonstrate that KCLA delivers good performance across diverse tasks, including image recognition, object detection, and medical image
segmentation. The code is publicly available at https://github.com/bgx666/KCLA.
Read the full abstract ▾
All authors Jianlong Xiong, ChuanBo Xie, Le Yu, Quansong He, Tao He
Computer Vision
DCSNet: Multiscale Feature Aggregation for Small Medical Object Segmentation with Detection-guided Hierarchical Cropping
Shanfeng Zhang, Bo Gou, Yue Cao et al. (+3 more)
Small object
segmentation in medical imaging is primarily hindered by class imbalance and inherent boundary complexity. Consequently, conventional global networks frequently fail to detect sparse targets or suffer from severe edge degradation. To overcome these limitations, we propose the Detection-guided Cropping
Segmentation Network (DCSNet), an end-to-end framework that transforms global dense prediction into a localized refinement process. This framework integrates two core components, namely Detection-guided Hierarchical Cropping (DGHC) and Multiscale Feature Aggregation (MSFA). The DGHC module leverages region proposals to dynamically extract object-centric features, effdataectively filtering out massive background interference to mitigate class imbalance. Subsequently, the MSFA module operates strictly within these purified regions, synergizing a
Transformer encoder with a pixel-adaptive fusion strategy. This mechanism dynamically aggregates multiscale features to capture both semantic context and fine-grained details for sharp boundary delineation. Extensive experiments across three diverse medical
datasets demonstrate that DCSNet significantly outperforms existing state-of-the-art methods, yielding substantial improvements in boundary precision and offering a highly robust solution for clinical micro-lesion
segmentation.
Read the full abstract ▾
All authors Shanfeng Zhang, Bo Gou, Yue Cao, Lei Zhang, Zhang Yi, Tao He
Computer Vision
Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs
Yunhun Nam and Jongheon Jeong
Vision-
Language Models (VLMs) have shown strong performance in visual understanding, yet they still suffer from
hallucinations, generating content that is not grounded in the image. Preference
alignment is a promising approach to improve visual faithfulness, but its success depends heavily on how preference pairs are constructed. Existing methods exhibit two key limitations; (a) intervention-based methods often introduce significant deviation from the policy distribution, and (b) sampling-based methods often underuse visual information during the construction. In this paper, we propose ViPSy (
Vision-driven Preference Synthesis), a framework for constructing preference data that are both policy-aligned and visually grounded. Our framework consists of two stages; in the first stage, ViPSy derives a visual cue from recurring object-level content across semantically aligned image variants, so preference construction can rely on visual information rather than
language priors. In the second stage, ViPSy conditions the policy's own rollouts on this cue, allowing candidates to be guided by visually grounded content while staying close to the policy's response distribution. The resulting candidates remain close to the policy's response distribution while better leveraging visual information from the image. Experiments show that the resulting VLM, preference-aligned with ViPSy-constructed preference pairs, achieves a new state-of-the-art in
hallucination mitigation. Compared with the previous state-of-the-art method, it reduces
hallucination rates on AMBER and Object HalBench by 35.7% and 24.5%, respectively. The resulting model further improves on general visual grounding
benchmarks, e.g., MMStar, MMVP, and CV-Bench, while also yielding gains in semantic
segmentation and ImageNet linear probing, underscoring the effectiveness of our framework in enhancing the model's visual capabilities.
Read the full abstract ▾
Computer Vision
Meta-learning as a principle for human-like visual representations
Can Demircan, Marcel Binz, Alireza Modirshanechi et al. (+1 more)
The structure of human visual representations underpins our capacity for adaptive behaviour. While pretrained neural networks model human visual representations with unprecedented success, a large discrepancy remains. We propose one reason: these networks optimise a single fixed objective, whereas human representations must support open-ended tasks. We hypothesise this flexibility arises from meta-learning (learning to learn), a pressure shaping representations to acquire new tasks from few observations. To test this, we train a sequence model, without any supervision from human data, across thousands of semantically rich tasks mapping images to high-level concepts. Compared to their pretrained base encoders, meta-learned representations better predict human similarity judgements, semantic rule learning, and high-level visual cortex. Behavioural gains depend on disentangled, high-level task distributions, while
brain alignment is driven primarily by the learning-to-learn pressure. Our results suggest the flexibility of human visual representations reflects the functional demand to learn new semantic relationships on the fly.
Read the full abstract ▾
All authors Can Demircan, Marcel Binz, Alireza Modirshanechi, Eric Schulz
Computer Vision
Semantic-Aware Generative Image Transmission for Resource-Constrained Visual IoT Systems
Chenyang Zhang, Changwang Liu, Jinqi Zhu et al. (+4 more)
Resource-constrained visual Internet of Things (IoT) systems, such as edge cameras, unmanned sensing platforms, industrial inspection nodes, and remote monitoring sensors, often need to transmit task-relevant visual evidence over low-rate wireless links to an edge/cloud service. Existing image communication methods usually compress or transmit complete global representations, leaving limited room to exploit receiver-side generative restoration. This paper proposes a semantic-aware generative image transmission framework for edge-assisted visual IoT. The image captured by an IoT visual sensor is encoded into a discrete
token grid by a VQ encoder. At the IoT transmitter or nearby gateway,
token recoverability, estimated from prediction
entropy and local structure complexity, is fused with semantic importance obtained from instance
segmentation and category-aware scoring. A spatial dispersal sampler then selects the
tokens to be transmitted under a bitrate budget. The transmitter sends only the quantization indices of kept
tokens and a binary mask map, while the edge/cloud receiver recovers masked
tokens through MaskGIT with Halton sequence scheduling. Experiments on Kodak and VisDrone scenes under AWGN and Rayleigh channels show that the proposed method provides a flexible bitrate-quality tradeoff for narrowband visual IoT links. At 0.074 bpp, it uses 44.6% of the transmitted bits of the 0.167-bpp DeepJSCC/WITT reference while achieving 29.9 dB PSNR. A pseudo-GT downstream detection study on Kodak further shows that semantic-aware masking preserves task-relevant objects better than random masking at both 30% and 50% mask ratios.
Read the full abstract ▾
All authors Chenyang Zhang, Changwang Liu, Jinqi Zhu, Jiayi Chang, Yuxuan Wang, Shuqing He, Jia Guo
Computer Vision
Shaoxuan Li, Xiangyu Dong, Xiaoguang Ma et al. (+3 more)
Vision-
language navigation (VLN) has recently advanced with large
language and
multimodal models, enabling
agents to follow natural-
language instructions in unseen environments without training a task-specific
navigation policy. However, most existing VLN methods relying on large models still adopt an open-loop decision-execution approach, where candidate actions are generated from instructions and observations but are rarely verified or corrected before execution. This causes critical issues in aerial VLN, where minor errors in intermediate actions may quickly accumulate into large trajectory deviations and lead to target loss. To address this issue, we propose Closed-loop Self-verified
Retrieval-augmented
Reasoning (CLOSER), a training-policy-free method that sequentially performs action
reasoning, reliability verification, targeted
retrieval, and action correction in a closed-loop manner before executing concrete actions. We instantiate the CLOSER in aerial VLN tasks and develop a CLOSER-VLN framework, which is composed of three components: a hierarchical reasoner for generating candidate actions based on available information, a multidimensional action verifier for assessing the reliability of actions generated by the reasoner, and a verification-triggered
multimodal retriever for retrieving targeted exemplars from a
memory bank only when verification fails. We conduct experimental evaluations on the CityNav
benchmark, where CLOSER-VLN achieves 32.01% SR and 21.28% SPL on the test-unseen split, confirming the effectiveness of closed-loop
reasoning.
Read the full abstract ▾
All authors Shaoxuan Li, Xiangyu Dong, Xiaoguang Ma, Junfeng Chen, Haoran Zhao, Yaoming Zhou
Computer Vision
RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception
Emily Bejerano, Federico Tondolo, Devang Gupta et al. (+5 more)
Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar
datasets fail to generalize to new objects, environments, and sensing trajectories. We present RadarTwin, a framework for generating deployment-specific radar training data before real data collection. Given a 3D reconstruction of a target space (phone
LiDAR,
robot-mounted sensing, or RGB-to-3D), RadarTwin uses a
vision-
language model to infer radar-relevant surface materials and a physics-based ray tracer to synthesize raw frequency-modulated continuous-wave (FMCW) radar measurements with multi-bounce propagation. To study what transfers from simulation to reality, we collect a paired real-simulated
dataset spanning household objects, material classes, distances, rotations, translations, and mobile sensing trajectories. We show that simulated and real radar share the same object-discriminative shape and material features, and that modeling the environment's multipath is essential to matching real measurements. A representation trained on simulation alone recognizes real objects at 2.5 times chance with no real radar labels, and a few labeled examples raise this to 95.3% on a 12-way recognition task. RadarTwin enables training radar perception for a new space before any real radar data is collected there.
Read the full abstract ▾
All authors Emily Bejerano, Federico Tondolo, Devang Gupta, Aaron Mano Cherian, Taeyoo Kim, Ayaan Qayyum, Xiaofan Yu, Xiaofan Jiang
Computer Vision
JASPR: Joint Spatial Representation learning of histology and spatial genomics for improved virtual genomic screening and clinical prognostication
Marija Pizurica, Eric Zimmermann, Neil Tenenholtz et al. (+5 more)
Recent studies have shown that spatial properties of tumors are critical for understanding disease biology and predicting patient outcomes. These spatial properties are increasingly uncovered through complementary modalities: spatial transcriptomics (ST) captures spatially-resolved molecular states, while hematoxylin and eosin-stained whole slide images (HE) reveal tissue morphology. While approaches are emerging to fuse these modalities, effective methods that learn not only joint representations but also incorporate spatial context across modalities are lacking. Here, we present JASPR (Joint Spatial Representation learning), a self-supervised deep learning framework that integrates HE images and ST data through a cross-modal reconstruction objective that incorporates spatial context within HE images and ST profiles. It employs shared modules to capture universal spatial properties across modalities, while modality-specific experts encode features unique to morphological and genomic data. We train and validate JASPR on breast cancer
datasets, demonstrating that its learned joint representation substantially improves HE-based prediction of 9,248 genes and provides prognostic value for breast cancer outcomes.
Read the full abstract ▾
All authors Marija Pizurica, Eric Zimmermann, Neil Tenenholtz, James Hall, Olivier Gevaert, Ava P. Amini, Lorin Crawford, Kristen A. Severson
Computer Vision
Active Adversarial Perturbation-driven Associative Memory Retrieval for RGB-Event Visual Object Tracking
Xiao Wang, Xufeng Lou, Zikang Yan et al. (+5 more)
RGB-Event tracking improves localization robustness by fusing RGB appearance textures and dense temporal motion cues from event sensors. While this multi-modal scheme broadens tracking applicability, real-world scenes suffer diverse structured signal degradations that hinder traditional multi-modal fusion. In harsh environments, either modality can lose reliability drastically, and targets frequently appear incomplete due to occlusion, edge truncation and foreground clutter.To tackle the above challenges, we present a hierarchical perturbation and
retrieval framework tailored for RGB-Event tracking with robustness against partial target missing and modal degradation, termed APRTrack. To mimic real-world signal corruption, APRTrack constructs structured degradation via two
adversarial perturbation branches at the modality and spatial levels, which separately simulate full-modal failure and localized target region absence. A hierarchical routing mechanism is designed to disentangle the training pipelines of the two perturbation types, effectively eliminating feature collapse induced by superimposed degradation constraints. Furthermore, we devise Footprint-guided Channel-calibrated Hopfield
Retrieval (FCHR) for reliable historical information compensation. This module evaluates
retrieval confidence based on association footprints between queries and
memory banks, and calibrates the
retrieval metric space prior to Hopfield matching, realizing controllable historical feature compensation bounded to target regions. Extensive experiments on FE108, COESOT, VisEvent, and FELT
datasets demonstrate the effectiveness of our proposed strategies for the RGB-Event visual object tracking. The source code and pre-trained models will be released on https://github.com/Event-AHU/OpenEvTracking
Read the full abstract ▾
All authors Xiao Wang, Xufeng Lou, Zikang Yan, Lan Chen, Sibao Chen, Yaowei Wang, Yonghong Tian, Jin Tang
Computer Vision · Cross-listed from cs.RO
WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation
Baiqi Li, Ce Zhang, Yu Fang et al. (+4 more)
A
robot working alongside people must reason about what they have done, in what order, and with what intent.
Video carries the spatial layouts, object histories, and gestures that
language leaves underspecified, yet today's manipulation
benchmarks pair an instruction with a single current image, offering no way to evaluate
reasoning over observed human behavior. We introduce WatchAct, a
benchmark for
robot manipulation grounded in observed human behavior. Each instance pairs a real-world human-action
video and a
language instruction with an aligned simulator scene and an executable LIBERO task, enabling scalable and reproducible evaluation. WatchAct comprises 3,000 long-horizon instances across 14 tasks in four capability domains drawn from the cognitive demands of watching another
agent: parsing events (Event Grounding), recovering procedural structure (Procedural
Reasoning), inferring unstated intent (Implicit Intent Inference), and tracking how the scene was changed (Episodic
Reasoning). We further propose a disentangled evaluation protocol that separately measures (i)~
video-to-plan
reasoning by
vision-
language models, (ii)~policy execution under oracle plans, and (iii)~full task completion by integrated planner–policy pipelines. In both simulation and on a Franka Research 3
robot, current systems remain far from solving WatchAct. The best pipeline, Gemini-3.1-Pro with $\pi_{0.5}$, reaches only 16.3% Success Rate (SR) in simulation and 14.0% on the real
robot. Gemini-3.1-Pro attains just 36.8% Plan SR (vs. 97.1% for humans), while $\pi_{0.5}$ reaches only 21.5% Task SR under oracle plans and drops to 10.6% on out-of-domain scenarios.
Dataset and code are available at https://baiqi-li.github.io/watchact_page/.
Read the full abstract ▾
All authors Baiqi Li, Ce Zhang, Yu Fang, Yue Yang, Shangzhe Li, Mingyu Ding, Gedas Bertasius
Computer Vision · Cross-listed from eess.IV
Revealing Mammographic Phenotypes in Deep Learning Breast Cancer Risk Models
Ruiyu Jia, Yanqi Xu, Yuxuan Chen et al. (+2 more)
Mammogram-based deep learning models have improved breast cancer risk prediction, but the learned imaging patterns remain underexplored. Existing interpretability methods rely on single-image saliency maps, failing to identify recurring mammographic phenotypes across large patient cohorts. By clustering patch
embeddings from a pre-trained model, Mirai, we isolate recurring phenotypes linked to 5-year cancer risk. Analyses show risk-increasing phenotypes capture complex structures (e.g., dense tissue, microcalcifications) and shortcut artifacts (e.g., clips). These phenotypes correlate strongly with older age and higher BI-RADS density. Our framework connects tissue patterns to AI risk scores, revealing clinical signatures and potential latent model confounders.
Read the full abstract ▾
All authors Ruiyu Jia, Yanqi Xu, Yuxuan Chen, Yiqiu Shen, Laura Heacock
Computer Vision · Cross-listed from cs.LG
Rethinking Training & Inference for Forecasting: Linking Winner-Take-All back to GMMs
Qiyuan Wu, Katie Z Luo, Bharath Hariharan et al. (+2 more)
Trajectory
forecasting for autonomous driving has advanced rapidly, yet representative models often produce uninformative posteriors over forecast modes, causing problems for mode
pruning. We trace this to a modeling-training mismatch: forecasters are typically modeled as conditional Gaussian mixture models (GMMs) but trained with a winner-take-all (WTA) loss that assigns each sample to its nearest mode. We argue that this K-means-like hard assignment (one-hot), while preventing mode collapse, is the source of uninformative mode probabilities: it over-segments the trajectory space, ignores relatedness among nearby modes, and yields assignment instability under small perturbations. Guided by this lens, we introduce two post-hoc treatments: (1) test-time posterior-weighted merging that aggregates nearby candidate trajectories; and (2) a one-step expectation-maximization (EM) update that replaces hard labels with soft responsibilities, sharing probability mass across neighboring modes. Across several WTA-trained architectures, these lightweight steps produce more informative, faithfully ranked mode posteriors and strengthen final forecasts on popular displacement metrics – without retraining. Our analysis unifies recent design choices through a GMM-vs-K-means perspective and offers principled, practical corrections that better align training objectives with inference.
Read the full abstract ▾
All authors Qiyuan Wu, Katie Z Luo, Bharath Hariharan, Wei-Lun Chao, Mark Campbell
Computer Vision
Methane-Plume Segmentation From Hyperspectral Satellite Imagery Via Multimodal Deep Learning
Brayan Quintero, Jeferson Acevedo, Samuel Traslaviña et al. (+1 more)
Efficient detection of methane plumes is crucial for understanding and mitigating global warming, as accurately identifying and segmenting them in earth observation imagery remain essential for large-scale monitoring. In this work, we propose a
multimodal deep learning model that integrates a feature-guided methane enhancement (FGME) mechanism which injects physically meaningful methane cues into
transformer-based RGB representations at multiple semantic scales. Our method is evaluated on the MPDataset, where it outperforms the state-of-the-art with improvements of +0.92 in MIoU, +0.87 in MPrecision and +1.01 in Recall. Notably, these gains are obtained with a substantially lower computational cost than other high-performing architectures, resulting in a favorable accuracy-efficiency trade-off for large-scale methane monitoring. These results highlight the potential of efficient
multimodal fusion strategies for accurate and scalable methane plume
segmentation in real-world remote sensing applications.
Read the full abstract ▾
All authors Brayan Quintero, Jeferson Acevedo, Samuel Traslaviña, Hoover Rueda-Chacón
Computer Vision
Neural Voxel Dynamics: Learning Volumetric Feature Advection for 3D Physics in V-JEPA Latent Space
Zican Wang and Niloy Mitra
We present Neural Voxel Dynamics, a self-supervised framework for learning 3D latent dynamics from
monocular video. While generative
video models now produce visually compelling motion, their predominantly 2D representations provide limited
geometric structure for modeling and controlling physical interactions. Instead, we learn dynamics in a lifted volumetric latent space. Specifically, we unproject semantic
Video Joint-
Embedding Predictive Architecture (V-JEPA) features into a voxel grid using
monocular depth priors, producing a geometrically grounded representation that retains rich
video features. We then introduce Volumetric Feature Advection, an action-conditioned transition model that predicts the
evolution of these latent features directly in 3D. Unlike hybrid approaches that assume access to privileged physics-engine states or explicit simulation during training and/or inference, we learn only using
video-derived supervision and, if available, actions, allowing a single latent dynamics model to represent heterogeneous material-dependent phenomena, unifying rigid-body and fluid dynamics. We evaluate across multiple
datasets (synthetic and real) and unseen scenarios (e.g., new boundary conditions, OOD generalization), measuring both predictive fidelity and 3D
geometric consistency. We report physical plausibility with Physics-IQ using
videos decoded from predicted latent states and
videos generated from predicted optical flow. Our results show that geometrically lifting pretrained
video latent representations provides a scalable route toward dynamic world models that learn structured 3D physical dynamics without access to privileged simulator state.
Read the full abstract ▾
Computer Vision
DinoLink: A Token-Centric Representation Compression Framework for Bandwidth-Constrained Collaborative V2X Perception
Tianle Zhu, Haohua Que, Handong Yao et al. (+2 more)
High-precision remote perception is often hindered by the severe bandwidth constraints of Vehicle-to-Everything (V2X) networks. We propose DinoLink, a
token-centric compression framework that replaces raw pixel streaming with discrete semantic communication for vehicle-cloud collaborative inference. DinoLink employs a dual-sparsity architecture: a saliency-aware selector prunes redundant background
tokens, while a Residual Vector Quantization (RVQ) module collapses features into compact codebook indices. By transmitting only lightweight indices and positional priors, DinoLink achieves a $139\times$ bitrate reduction compared to uncompressed transmission while maintaining a competitive 32.8% mAP on the nuScenes
dataset. Deployment simulations further demonstrate a $34.5\times$ acceleration in narrow-band environments, such as LoRa. Our results substantiate DinoLink as a robust, bandwidth-efficient frontend for high-fidelity remote perception in constrained V2X scenarios. The code is publicly available at https://github.com/UGA-MOBILITY-LAB/dino_link.
Read the full abstract ▾
All authors Tianle Zhu, Haohua Que, Handong Yao, Hongyi Xu, Zhipeng Bao
Computer Vision
Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs
Xi Xiao, Chen Liu, Chih-Ting Liao et al. (+9 more)
Multimodal large
language models (MLLMs) extend large
language models (LLMs) with visual perception, enabling joint
reasoning over images and text. Despite inheriting strong
reasoning capabilities from LLMs, they remain prone to
hallucinations that contradict their visual inputs. Mechanistic studies indicate that this weakness stems from visual laziness: MLLMs encode the correct visual evidence internally, but overly rely on strong
language priors during response. Existing
alignment methods, such as direct preference optimization, primarily optimize outcome-level rewards based on text. This introduces an optimization bias toward linguistic shortcuts, leading to responses that often contradict the visual evidence. To address this, we propose Visual Information Gain In
aLignment (VIGIL), a reinforcement-learning (RL) post-training framework that shifts the focus from numerical reward fitting to causal visual grounding. VIGIL introduces a
geometric constraint that explicitly maximizes the mutual information between the visual input and the generated response. We achieve this by penalizing "blind confidence" instances where the model remains improperly certain even when textual-visual
attention is masked to create a counterfactual blind state. Extensive experiments show that VIGIL consistently outperforms recent
alignment methods across
hallucination and
reasoning benchmarks without compromising text-only capabilities. Our approach matches the full-data performance of state-of-the-art methods using only 25% of the preference data and even demonstrates emergent spatial grounding capabilities without explicit bounding box supervision.
Read the full abstract ▾
All authors Xi Xiao, Chen Liu, Chih-Ting Liao, Yunbei Zhang, Qizhen Lan, Yuxiang Wei, Lin Zhao, Janet Wang, Jianyang Gu, Muchao Ye, Tianyang Wang, Hao Xu
Computer Vision
What Do Deepfake Benchmarks Measure? An Audit Using Frozen Self-Supervised Representations
Samuel Pagon, Yixuan Shen, Vishal Asnani et al. (+1 more)
As deepfake generators approach perceptual indistinguishability, reliable detection becomes critical. Yet, detectors that score well on
benchmarks routinely fail in the wild. A concerning feedback loop has emerged:
benchmarks drive increasingly complex, engineered detectors, yet if those
benchmarks do not reflect real-world deepfakes, this complexity may be solving the wrong problem entirely. This raises a prior question: what are these
benchmarks actually measuring? We conduct an audit of
video, image, and audio deepfake
benchmarks using a deliberately simple diagnostic. If a linear probe on frozen, general-purpose self-supervised representations can approximate the performance of a bespoke detector, the
benchmark is largely rewarding general modality understanding rather than forensic understanding. This has two implications: the
benchmark may not reflect realistic threat models, and it raises the question of whether the bespoke detectors the probe approaches are truly learning forensic understanding. We observe, across three modalities, linear probes on general-purpose self-supervised representations closely approach the performance of bespoke detectors. We further show that generator-level difficulty is partly explained by Frechet geometry in the same representation space. Together, these results support a
benchmark-audit view of deepfake detection: before high scores are read as evidence of forensic understanding, it is worth asking how much of the
benchmark is already solved by general-purpose representations.
Read the full abstract ▾
All authors Samuel Pagon, Yixuan Shen, Vishal Asnani, Feng Liu
Computer Vision
Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models
Xi Xiao, Xingjian Li, Yunbei Zhang et al. (+7 more)
Visual
prompt tuning has emerged as a parameter-efficient fine-tuning approach for adapting large-scale
Vision Transformers (ViTs) to downstream tasks. As its learnable
prompts are applied in input and feature spaces, prior to jointly going through
attention in
transformer layers, the most commonly used scheme for fusing image and
prompt tokens is concatenation or addition. In this paper, we aim to study a fundamental yet essential problem in visual
prompt tuning: whether a single fusion scheme tends to yield better results, and whether that would be beneficial to develop a hybrid fusion scheme. To this end, we formulate the task as a bi-level optimization problem, and solve it leveraging differentiable architecture search. In this context, the learnable
prompts and their fusion schemes are jointly optimized. To enrich the search space in the architecture search, we propose two additional fusion schemes, namely, affine transformation and cross-
attention, in addition to concatenation and addition. Extensive experiments on 34
datasets spanning VTAB-1k, FGVC, and HTA show consistent gains over
prompt-tuning baselines. With a frozen ViT backbone, our method delivers a favorable accuracy–latency–parameter trade-off compared with VPT-Deep and recent variants. Our findings reveal that how
prompts fuse with image
tokens plays a significant role in visual
prompt tuning, and a hybrid fusion fashion can more effectively leverage layer semantics of ViTs, contributing a novel perspective for visual
prompt-tuning research.
Read the full abstract ▾
All authors Xi Xiao, Xingjian Li, Yunbei Zhang, Cheng Han, Tianming Liu, Tianyang Wang, Runmin Jiang, Jihun Hamm, Xiao Wang, Min Xu
Computer Vision · Cross-listed from quant-ph
Aldo Lamarre and Dominik Šafránek
Autoencoders transformed classical machine learning by solving the curse of dimensionality, enabling principled weight initialization and learning compact, structured representations. In this work, we extend this paradigm to
quantum machine learning by introducing a variational autoencoder framework that learns task-specific
quantum embeddings of classical data. We demonstrate that high-dimensional
datasets, including ImageNet, can be compressed into a 13-
qubit quantum representation while remaining reconstructable through a learned decoder. On MNIST (3 vs 5), our approach achieves 98.5% validation accuracy using a
circuit-centric
quantum classifier, within 1.2 percentage points of a classical neural network baseline (99.7%) and more than 30 percentage points above a naive amplitude-
embedding approach. Unlike amplitude
embeddings, which require full
quantum state
tomography for recovery, or angle
embeddings, which generally rely on
circuit inversion under restrictive assumptions, the proposed framework reconstructs the original data from only a
polynomial number of measurements. The framework was further validated on IBM
quantum hardware, confirming that the learned
embeddings remain stable and reconstructable under real device noise.
Read the full abstract ▾
Computer Vision
Beyond Aesthetics: Quantifying Information Loss in Turbid Scenes
Vasiliki Ismiroglou, Stefan H. Bengtson, Tasos Benos et al. (+2 more)
Visibility in
underwater environments degrades rapidly under turbid conditions, yet the effects on computer-
vision models remain unclear. This issue is compounded by reliance on synthetic turbidity
datasets, which may misrepresent real-world information loss. To address this gap, we introduce the Turbid
Underwater Baseline (TUB)
dataset, comprising 1,320 images captured under extreme turbidity and over 16,000 high-confidence ground-truth
segmentation masks. We additionally propose PCD, a metric derived from phase congruency maps that is invariant to contrast and aims to capture the loss of structural information in real turbidity. We show that PCD correlates strongly with the performance of instance
segmentation models on both real and synthetic turbid images, whereas common metrics in the field show weak to no correlation at all. The
dataset and relevant code can be found on the project page: https://vap.aau.dk/pcd
Read the full abstract ▾
All authors Vasiliki Ismiroglou, Stefan H. Bengtson, Tasos Benos, Thomas B. Moeslund, Malte Pedersen
Computer Vision
GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Models
Chaoxiang Cai, Minghe Weng, Jie Li et al. (+4 more)
With the increase in model parameters and training data, the instruction following and generalization capabilities of Large VisionLanguage Models (LVLMs) have been significantly improved. Based on the Mixture of Experts (MoE) architecture, LVLMs expand their parameter capacity while maintaining the inference cost. However, traditional MoE methods employ a Top-k static routing strategy, which fails to account for variations in the input and adaptively select the number of experts, resulting in suboptimal resource utilization. In this paper, we propose viewing
token routing as an information encoding task, framing dynamic routing as a Minimum Description Length (MDL) problem in encoding By validating the connection between MDL and gating
entropy in the MoE scenario, we introduce Gating
Entropy-based Uncertainty-aware Adaptive Routing (GeMoE) for MoE. Unlike traditional static or heuristic-based dynamic routing methods, GeMoE explicitly models the trade-off between model complexity and performance. By using gating
entropy to assess the complexity of
tokens, GeMoE adaptively determines the number of experts each
token should engage. On a wide range of backbones and
benchmarks, our method achieves 99.5% average performance retention compared to the original static routing, while improving average expert activation sparsity by 36.5%.
Read the full abstract ▾
All authors Chaoxiang Cai, Minghe Weng, Jie Li, Yibo Jiang, Longrong Yang, Zequn Qin, Xi Li
Computer Vision
Beyond Single-Source Cognitive Taskonomy:Multi-Source Task Relations through fMRI Transfer Learning
Junfeng Xia, Wendu Li, Mengjiao Zhang et al. (+1 more)
Cognitive tasks are organized by shared and specialized neural processes. Masked fMRI reconstruction provides a common self-supervised objective for quantifying transfer relations among task states, but existing reconstruction-based taskonomies mainly study one-to-one transfer from a single source task to a target. Here, we extend an fMRI cognitive taskonomy from single-source to multi-source transfer across 23 Human Connectome Project task states and use Boolean Integer Programming (BIP) to analyze budget-constrained task allocation. We train 1,127 task-specific and transfer models. Single-source transfer is directional and paradigm structured: motor states transfer well within the motor paradigm but provide limited support to most non-motor targets, consistent with a shared sensorimotor execution system and effector-specific representations. Multi-source transfer depends on the composition of the source set, suggesting that many-to-one task relations are not fully captured by pairwise taskonomy alone. Across supervision budgets, BIP repeatedly allocates direct supervision to several 0-back and 2-back working-
memory states, although these states are not consistently the strongest individual sources. This pattern may reflect the integration of perceptual, attentional, and executive processes in working-
memory tasks. Together, these findings reveal a cross-paradigm-limited motor cluster and working-
memory states with high priority under the specified global allocation objective. Our study extends reconstruction-based fMRI taskonomy from one-to-one transfer to many-to-one task relations and budget-constrained task dependencies.
Read the full abstract ▾
All authors Junfeng Xia, Wendu Li, Mengjiao Zhang, Jie Guo
Computer Vision
A multi-task spatiotemporal deep neural network for predicting penetration depth and morphology in laser welding
Sen Li, Haichao Cui, Chendong Shao et al. (+2 more)
In
laser penetration welding, the assessment of penetration state and weld seam morphology plays a crucial role in determining the weld quality. This paper presents a comprehensive introduction of the innovative muti-task deep learning model that has the capability to predict penetration state, depth, and weld seam morphology with high accuracy. The monitoring platform relies on weld pool images captured during the
laser welding process using a complementary metal-
oxide-
semiconductor camera. The proposed model integrates spatiotemporal features extracted from top weld pool images along with welding parameters, establishing a deep learning framework based on convolutional neural networks and state space models for more efficient extraction and processing of spatial-temporal information. Furthermore, a reliable method for constructing the
dataset is proposed to enhance both robustness and generalization capability of the developed model. Validation results on the test set demonstrate that prediction accuracy for penetration state can reach 99.35%, while prediction error for penetration depth is 1.79 millimeter, and accuracy of reconstructing the weld cross-section is 95.65%. This study provides new insights and methodologies for in-situ quality control strategies in
laser penetration welding systems.
Read the full abstract ▾
All authors Sen Li, Haichao Cui, Chendong Shao, Yaqi Wang, Xinhua Tang
Computer Vision · Cross-listed from eess.IV
Rendering Novel Views of MRI Using 3D Gaussian Splatting
Robin Y. Park, Mark C. Eid, Rhydian Windsor et al. (+4 more)
The objective of this paper is to improve radiological gradings measured on MRIs of spines, by resampling scans so that the new view planes are better aligned with the target anatomy than the original sparse images. To this end, we adapt 3D Gaussian Splatting to form a volumetric reconstruction starting from non-aligned MRIs to render imaging planes aligned with the anatomy relevant for clinical evaluation. The novel view plane is optimal for diagnostic radiological grading of the target anatomy, whereas the original MRI is not. The resampled scans are then used to predict ordinal severity grades of localised stenosis conditions in spinal MRIs. We compare our method against (1) Voxel Interpolation, which takes the average of inverse-distance weighted nearest neighbour intensities for each target coordinate, and (2) Cubic B-spline resampling, which estimates target points using a smooth piecewise-cubic function built from the source points. Experiments show that across all stenosis conditions, resampled scans using Gaussian Splatting produce more accurate stenosis gradings compared to the raw scans which do not include the complete anatomy in-plane, as well as images resampled using Voxel Interpolation or Cubic B-spline resampling.
Read the full abstract ▾
All authors Robin Y. Park, Mark C. Eid, Rhydian Windsor, Amir Jamaludin, Ana I.L. Namburete, João F. Henriques, Andrew Zisserman
Computer Vision · Cross-listed from cs.LG
Fast LeWorldModel
Yuntian Gao and Xiangyu Xu
Joint-
Embedding Predictive Architectures (JEPAs), including recent LeWorldModel (LeWM), have become a promising foundation for reconstruction-free visual world models. For visual planning, however, LeWM evaluates candidate action sequences by repeatedly applying a local one-step latent transition model. This autoregressive rollout makes planning computationally expensive and exposes the predicted trajectory to accumulated latent errors as the horizon grows. We propose Fast LeWorldModel (Fast-LeWM), a fast latent world model that replaces repeated local rollout with action-prefix prediction. Given the current latent and a candidate action sequence, Fast-LeWM encodes its prefixes and predicts the future latents reached after executing those prefixes in parallel. By making action prefixes the basic prediction unit, Fast-LeWM directly models action effects accumulated to different extents over multiple horizons. This prefix-level supervision forces the model to learn how states continuously evolve under different action prefixes, rather than only fitting one-step state transitions. During planning, the predictor can use the prefix
token from the encoded action sequence to evaluate the corresponding future latent without explicitly rolling through each intermediate imagined state. Across multiple tasks, Fast-LeWM improves average success over LeWM while substantially reducing planning time, achieving lower open-loop latent loss whose growth becomes significantly slower as the rollout horizon increases.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
TaskNPoint: How to Teach Your Humanoid to Hit a Backhand in Minutes
Blake Werner, Ilona Demler, Pietro Perona et al. (+1 more)
How do we learn to hit a tennis backhand? Not from a thousand hours of tennis tournaments on TV - we work with a coach and practice. We argue this is also the right recipe for teaching dynamic skills to
humanoid robots. This follows from a structural property of dynamic skills: the outcome is decided by a short, crucial portion of the trajectory - for a backhand, the ~20cm of racket travel around ball contact. Getting this interaction window right requires coordinating the whole motion, so that control, physics, and morphology act in concert. Learning thus reduces to mastering a handful of distinct actions and, for each, practicing until the window comes out right. To this end, we introduce TaskNPoint, a training protocol which makes the coach-learner division of labor explicit. The human coach contributes four inputs: a discrete set of skills (e.g. different shots), one demonstration per skill, identification of the interaction window, and the goal. Learning in a physically realistic simulation environment fills in each action trajectory and provides robustness to unmodeled events. Crucially, randomized target sampling during training lets a single demonstration generalize zero-shot to unseen goal locations. We test this approach on a Unitree G1
humanoid that hits forehands and backhands against balls thrown by a human, kicks incoming soccer balls, and picks and places boxes from novel locations. We find that learning is successful from short human
video demonstrations and under an hour of training on a single GPU, with no per-task reward tuning.
Read the full abstract ▾
All authors Blake Werner, Ilona Demler, Pietro Perona, Aaron D. Ames