Computer Vision
NAMD: Virtual Follow-up Computed Tomography Synthesis via Nodule-Aligned Multimodal Diffusion Models for Early Lung Cancer Diagnosis
James Song, Yifan Wang, Chuan Zhou et al. (+1 more)
Lung cancer remains the leading cause of cancer-related mortality worldwide, with survival outcomes critically dependent on early and accurate detection. When low-dose computed
tomography (LDCT) findings are indeterminate, clinicians typically defer diagnosis pending follow-up CT imaging obtained up to 12 months later, inevitably delaying treatment for patients with malignant nodules. To address this clinical gap, we propose Nodule-Aligned
Multimodal (Latent)
Diffusion (NAMD), a novel generative framework that synthesizes one-year follow-up nodule CT images conditioned on the baseline CT scan, quantitative nodule biomarkers, and patient-level Electronic Health Records (EHR), enabling timely prediction of nodule malignant progression without requiring actual follow-up scans. NAMD introduces two key contributions: (i) a nodule-aligned latent space regularized so that
embedding distances reflect clinically meaningful biomarker changes, and (ii) an LLM-driven
multimodal conditioning mechanism encoding heterogeneous EHR data into the
diffusion backbone. Evaluated on the National Lung Screening Trial (NLST), our method's synthetic follow-up images achieve an AUROC of 0.805 and an AUPRC of 0.346 for lung nodule malignancy prediction, outperforming both the baseline LDCT performance without virtual follow-up generation, and existing state-of-the-art conditional generation methods, while maintaining competitive image quality. These findings suggest that NAMD enables earlier and more accurate lung cancer diagnosis by capturing clinically meaningful features of nodule progression.
Read the full abstract ▾
All authors James Song, Yifan Wang, Chuan Zhou, Liyue Shen
Computer Vision
Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers
Siyu Zhang
Sparse neural networks are often hypothesized to be more interpretable than dense models, motivated by findings that weight sparsity can produce compact
circuits in
language models. However, it remains unclear whether structural sparsity itself leads to improved semantic interpretability. In this work, we systematically evaluate the relationship between weight sparsity and interpretability in
Vision Transformers using DeiT-III B/16 models pruned with Wanda. To assess interpretability comprehensively, we introduce IMPACT, a multi-level framework that evaluates interpretability across four complementary levels:
neurons, layer representations, task
circuits, and model-level attribution. Layer representations are analyzed using BatchTopK sparse autoencoders,
circuits are extracted via learnable node masking, and explanations are evaluated with
transformer attribution using insertion and deletion metrics. Our results reveal a clear structural effect but limited interpretability gains. Sparse models produce
circuits with approximately $2.5\times$ fewer edges than dense models, yet the fraction of active nodes remains similar or higher, indicating that
pruning redistributes computation rather than isolating simpler functional modules. Consistent with this observation, sparse models show no systematic improvements in
neuron-level selectivity, SAE feature interpretability, or attribution faithfulness. These findings suggest that structural sparsity alone does not reliably yield more interpretable
vision models, highlighting the importance of evaluation frameworks that assess interpretability beyond
circuit compactness.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Federated Learning for Privacy-Preserving Medical AI
Tin Hoang
This dissertation investigates
privacy-preserving
federated learning for Alzheimer's disease classification using three-dimensional MRI data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). Existing methodologies often suffer from unrealistic data partitioning, inadequate
privacy guarantees, and insufficient benchmarking, limiting their practical deployment in healthcare. To address these gaps, this research proposes a novel site-aware data partitioning strategy that preserves institutional boundaries, reflecting real-world multi-institutional collaborations and data heterogeneity. Furthermore, an Adaptive Local Differential
Privacy (ALDP) mechanism is introduced, dynamically adjusting
privacy parameters based on training progression and parameter characteristics, thereby significantly improving the
privacy-utility trade-off over traditional fixed-noise approaches. Systematic empirical evaluation across multiple client federations and
privacy budgets demonstrated that advanced
federated optimisation
algorithms, particularly FedProx, could equal or surpass centralised training performance while ensuring rigorous
privacy protection. Notably, ALDP achieved up to 80.4% accuracy in a two-client configuration, surpassing fixed-noise Local DP by 5-7 percentage points and demonstrating substantially greater training stability. The comprehensive ablation studies and benchmarking establish quantitative standards for
privacy-preserving collaborative medical AI, providing practical guidelines for real-world deployment. This work thereby advances the state-of-the-art in
federated learning for medical imaging, establishing both methodological foundations and empirical evidence necessary for future
privacy-compliant AI adoption in healthcare.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
AsgardBench – Evaluating Visually Grounded Interactive Planning Under Minimal Feedback
Andrea Tupini, Lars Liden, Reuben Tan et al. (+2 more)
With AsgardBench we aim to evaluate visually grounded, high-level action sequence generation and interactive planning, focusing specifically on plan adaptation during execution based on visual observations rather than
navigation or low-level manipulation. In the landscape of
embodied AI
benchmarks, AsgardBench targets the capability category of interactive planning, which is more sophisticated than offline high-level planning as it requires
agents to revise plans in response to environmental feedback, yet remains distinct from low-level execution. Unlike prior
embodied AI
benchmarks that conflate
reasoning with
navigation or provide rich corrective feedback that substitutes for perception, AsgardBench restricts
agent input to images, action history, and lightweight success/failure signals, isolating interactive planning in a controlled simulator without low-level control noise. The
benchmark contains 108 task instances spanning 12 task types, each systematically varied through object state, placement, and scene configuration. These controlled variations create conditional branches in which a single instruction can require different action sequences depending on what the
agent observes, emphasizing conditional branching and plan repair during execution. Our evaluations of leading
vision language models show that performance drops sharply without visual input, revealing weaknesses in visual grounding and state tracking that ultimately undermine interactive planning. Our
benchmark zeroes in on a narrower question: can a model actually use what it sees to adapt a plan when things do not go as expected?
Read the full abstract ▾
All authors Andrea Tupini, Lars Liden, Reuben Tan, Yu Wang, Jianfeng Gao
Computer Vision
EvoIQA - Explaining Image Distortions with Evolved White-Box Logic
Ruchika Gupta, Illya Bakurov, Nathan Haut et al. (+1 more)
Traditional Image Quality Assessment (IQA) metrics typically fall into one of two extremes: rigid, hand-crafted mathematical models or "black-box" deep learning architectures that completely lack interpretability. To bridge this gap, we propose EvoIQA, a fully explainable symbolic regression framework based on Genetic Programming that Evolves explicit, human-readable mathematical formulas for image quality assessment (IQA). Utilizing a rich terminal set from the VSI, VIF, FSIM, and HaarPSI metrics, our framework inherently maps structural, chromatic, and information-theoretic degradations into observable mathematical equations. Our results demonstrate that the evolved GP models consistently achieve strong
alignment between the predictions and human visual preferences. Furthermore, they not only outperform traditional hand-crafted metrics but also achieve performance parity with complex, state-of-the-art deep learning models like DB-CNN, proving that we no longer have to sacrifice interpretability for state-of-the-art performance.
Read the full abstract ▾
All authors Ruchika Gupta, Illya Bakurov, Nathan Haut, Wolfgang Banzhaf
Computer Vision
Self-supervised Disentanglement of Disease Effects from Aging in 3D Medical Shapes
Jakaria Rabbi, Nilanjan Ray and Dana Cobzas
Disentangling pathological changes from physiological aging in 3D medical shapes is crucial for developing interpretable biomarkers and patient stratification. However, this separation is challenging when diagnosis labels are limited or unavailable, since disease and aging often produce overlapping effects on shape changes, obscuring clinically relevant shape patterns. To address this challenge, we propose a two-stage framework combining unsupervised disease discovery with self-supervised disentanglement of implicit shape representations. In the first stage, we train an implicit neural model with signed distance functions to learn stable shape
embeddings. We then apply clustering on the shape latent space, which yields pseudo disease labels without using ground-truth diagnosis during discovery. In the second stage, we disentangle factors in a compact variational space using pseudo disease labels discovered in the first stage and the ground truth age labels available for all subjects. We enforce separation and controllability with a multi-objective disentanglement loss combining covariance and a supervised contrastive loss. On ADNI hippocampus and OAI distal femur shapes, we achieve near-supervised performance, improving disentanglement and reconstruction over state-of-the-art unsupervised baselines, while enabling high-fidelity reconstruction, controllable synthesis, and factor-based explainability. Code and checkpoints are available at https://github.com/anonymous-submission01/medical-shape-disentanglement
Read the full abstract ▾
Computer Vision
FEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understanding
Eadom Dessalene, Botao He, Michael Maynord et al. (+5 more)
We introduce FEEL (Force-Enhanced
Egocentric Learning), the first large-scale
dataset pairing force measurements gathered from custom piezoresistive gloves with
egocentric video. Our gloves enable scalable data collection, and FEEL contains approximately 3 million force-synchronized frames of natural unscripted manipulation in kitchen environments, with 45% of frames involving hand-object contact. Because force is the underlying cause that drives physical interaction, it is a critical primitive for physical action understanding. We demonstrate the utility of force for physical action understanding through application of FEEL to two families of tasks: (1) contact understanding, where we jointly perform temporal contact
segmentation and pixel-level contacted object
segmentation; and, (2) action representation learning, where force prediction serves as a self-supervised pretraining objective for
video backbones. We achieve state-of-the-art temporal contact
segmentation results and competitive pixel-level
segmentation results without any need for manual contacted object
segmentation annotations. Furthermore we demonstrate that action representation learning with FEEL improves transfer performance on action understanding tasks without any manual labels over EPIC-Kitchens, SomethingSomething-V2, EgoExo4D and Meccano.
Read the full abstract ▾
All authors Eadom Dessalene, Botao He, Michael Maynord, Yonatan Tussa, Pavan Mantripragada, Yianni Karabati, Nirupam Roy, Yiannis Aloimonos
Computer Vision · Cross-listed from astro-ph.CO
Spectral Hierarchy of the Cosmic Web
Francisco-Shu Kitaura and Francesco Sinigaglia
We introduce a spectral hierarchy of
cosmic-web classifications obtained by applying simple scale-weighting kernels to the density field before performing a standard
eigenvalue-based web classification. This unifies and extends several widely used web definitions within a single framework: the familiar potential/
tidal web (large-scale, nonlocal), a
curvature-based web (more local, peak- and ridge-sensitive), and additional higher-derivative levels that progressively emphasize smaller-scale structure. Because the classification is built from second derivatives of the filtered field, successive hierarchy levels align naturally with operator families that appear in renormalised bias and effective descriptions of large-scale structure, providing an explicit bridge between
cosmic-web environments and long- and short-range nonlocal bias ingredients. We quantify the information content of the hierarchy with a compact statistic: we map each
cell to one of four ordered web types (void, sheet, filament,
knot), construct a corresponding “web contrast” field, and measure its cross-correlation with
halos from the AbacusSummit simulation suite on a coarse mesh with $\Delta L\simeq 5.5\,h^{-1}\mathrm{Mpc}$. We find that the hierarchy retains significant tracer-relevant information from very large scales down to the mesh Nyquist limit, with the more local (
curvature/higher-derivative) levels dominating toward nonlinear scales. This makes the spectral hierarchy a practical, interpretable conditioning basis for fast mock-
galaxy production and field-level modelling, and a flexible tool for studying environment-dependent clustering and assembly bias.
Read the full abstract ▾
Computer Vision
Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation
Renjie Liang, Yiling Ma, Yang Xing et al. (+6 more)
Automated radiology report generation from 3D CT volumes often suffers from incomplete pathology coverage. We provide empirical evidence that this limitation stems from a representational bottleneck: contrastive 3D CT
embeddings encode discriminative pathology signals, yet exhibit severe dimensional concentration, with as few as 2 effective dimensions out of 512. Corroborating this, scaling the
language model yields no measurable improvement, suggesting that the bottleneck lies in the visual representation rather than the generator. This bottleneck limits both generation and
retrieval; naive static
retrieval fails to improve clinical efficacy and can even degrade performance. We propose AdaRAG-CT, an adaptive augmentation framework that compensates for this visual bottleneck by introducing supplementary textual information through controlled
retrieval and selectively integrating it during generation. On the CT-RATE
benchmark, AdaRAG-CT achieves state-of-the-art clinical efficacy, improving Clinical F1 from 0.420 (CT-
Agent) to 0.480 (+6 points); ablation studies confirm that both the
retrieval and generation components contribute to the improvement. Code is available at https://github.com/renjie-liang/Adaptive-RAG-for-3DCT-Report-Generation.
Read the full abstract ▾
All authors Renjie Liang, Yiling Ma, Yang Xing, Zhengkang Fan, Jinqian Pan, Chengkun Sun, Li Li, Kuang Gong, Jie Xu
Computer Vision
Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition
Salah Eddine Bekhouche, Hichem Telli, Azeddine Benlamoudi et al. (+3 more)
Ambivalence and hesitancy (A/H) are subtle affective states where a person shows conflicting signals through different channels – saying one thing while their face or voice tells another story. Recognising these states automatically is valuable in clinical settings, but it is hard for machines because the key evidence lives in the disagreements between what is said, how it sounds, and what the face shows. We present ConflictAwareAH, a
multimodal framework built for this problem. Three pre-trained encoders extract
video, audio, and text representations. Pairwise conflict features – element-wise absolute differences between modality
embeddings – serve as bidirectional cues: large cross-modal differences flag A/H, while small differences confirm behavioural consistency and anchor the negative class. This conflict-aware design addresses a key limitation of text-dominant approaches, which tend to over-detect A/H (high F1-AH) while struggling to confirm its absence: our
multimodal model improves F1-NoAH by +4.6 points over text alone and halves the class-performance gap. A complementary text-guided late fusion strategy blends a text-only auxiliary head with the full model at inference, adding +4.1 Macro F1. On the BAH
dataset from the ABAW10 Ambivalence/Hesitancy Challenge, our method reaches 0.694 Macro F1 on the labelled test split and 0.715 on the private leaderboard, outperforming published
multimodal baselines by over 10 points – all on a single GPU in under 25 minutes of training.
Read the full abstract ▾
All authors Salah Eddine Bekhouche, Hichem Telli, Azeddine Benlamoudi, Salah Eddine Herrouz, Abdelmalik Taleb-Ahmed, Abdenour Hadid
Computer Vision
ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation
Aditya Iyer, Jack Roberts and Nora Ayanian
Multi-View Multi-Object Tracking (MV-MOT) aims to localize and maintain consistent identities of objects observed by multiple sensors. This task is challenging, as viewpoint changes and occlusion disrupt identity consistency across views and time. Recent end-to-end approaches address this by jointly learning 2D Bird's Eye View (BEV) representations and identity associations, achieving high tracking accuracy. However, these methods offer no principled uncertainty accounting and remain tightly coupled to their training configuration, limiting generalization across sensor layouts, modalities, or
datasets without retraining. We propose ModTrack, a modular MV-MOT system that matches end-to-end performance while providing cross-modal, sensor-agnostic generalization and traceable uncertainty. ModTrack confines learning methods to just the Detection and Feature Extraction stage of the MV-MOT pipeline, performing all fusion, association, and tracking with closed-form analytical methods. Our design reduces each sensor's output to calibrated position-covariance pairs $(\mathbf{z}, R)$; cross-view clustering and precision-weighted fusion then yield unified estimates $(\hat{\mathbf{z}}, \hat{R})$ for identity assignment and temporal tracking. A feedback-coupled, identity-informed Gaussian Mixture Probability Hypothesis Density (GM-PHD) filter with HMM motion modes uses these fused estimates to maintain identities under missed detections and heavy occlusion. ModTrack achieves 95.5 IDF1 and 91.4 MOTA on WildTrack, surpassing all prior modular methods by over 21 points and rivaling the state-of-the-art end-to-end methods while providing deployment flexibility they cannot. Specifically, the same tracker core transfers unchanged to MultiviewX and RadarScenes, with only perception-module replacement required to extend to new domains and sensor modalities.
Read the full abstract ▾
Computer Vision
Feed-forward Gaussian Registration for Head Avatar Creation and Editing
Malte Prinzler, Paulo Gotardo, Siyu Tang et al. (+1 more)
We present MATCH (Multi-view Avatars from Topologically Corresponding Heads), a multi-view Gaussian registration method for high-quality head avatar creation and editing. State-of-the-art multi-view head avatar methods require time-consuming head tracking followed by expensive avatar optimization, often resulting in a total creation time of more than one day. MATCH, in contrast, directly predicts Gaussian splat textures in correspondence from calibrated multi-view images in just 0.5 seconds per frame, without requiring data preprocessing. The learned intra-subject correspondence across frames enables fast creation of personalized head avatars, while correspondence across subjects supports applications such as expression transfer, optimization-free tracking, semantic editing, and identity interpolation. We establish these correspondences end-to-end using a
transformer-based model that predicts Gaussian splat textures in the fixed UV layout of a template mesh. To achieve this, we introduce a novel registration-guided
attention block, where each UV-map
token attends exclusively to image
tokens depicting its corresponding mesh region. This design improves efficiency and performance compared to dense cross-view
attention. MATCH outperforms existing methods in novel-view synthesis, geometry registration, and head avatar generation, while making avatar creation 10 times faster than the closest competing baseline. The code and model weights are available on the project website.
Read the full abstract ▾
All authors Malte Prinzler, Paulo Gotardo, Siyu Tang, Timo Bolkart
Computer Vision
Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory
Ce Zhang, Jinxi He, Junyi He et al. (+2 more)
Multi-modal Large
Language Models (MLLMs) have achieved remarkable performance across a wide range of visual
reasoning tasks, yet their vulnerability to safety risks remains a pressing concern. While prior research primarily focuses on jailbreak defenses that detect and refuse explicitly unsafe inputs, such approaches often overlook contextual safety, which requires models to distinguish subtle contextual differences between scenarios that may appear similar but diverge significantly in safety intent. In this work, we present MM-SafetyBench++, a carefully curated
benchmark designed for contextual safety evaluation. Specifically, for each unsafe image-text pair, we construct a corresponding safe counterpart through minimal modifications that flip the user intent while preserving the underlying contextual meaning, enabling controlled evaluation of whether models can adapt their safety behaviors based on contextual understanding. Further, we introduce EchoSafe, a training-free framework that maintains a self-reflective
memory bank to accumulate and retrieve safety insights from prior interactions. By integrating relevant past experiences into current
prompts, EchoSafe enables context-aware
reasoning and continual
evolution of safety behavior during inference. Extensive experiments on various multi-modal safety
benchmarks demonstrate that EchoSafe consistently achieves superior performance, establishing a strong baseline for advancing contextual safety in MLLMs. All
benchmark data and code are available at https://echosafe-mllm.github.io.
Read the full abstract ▾
All authors Ce Zhang, Jinxi He, Junyi He, Katia Sycara, Yaqi Xie
Computer Vision
Parallelised Differentiable Straightest Geodesics for 3D Meshes
Hippolyte Verninas, Caner Korkmaz, Stefanos Zafeiriou et al. (+2 more)
Machine learning has been progressively generalised to operate within non-Euclidean domains, but geometrically accurate methods for learning on surfaces are still falling behind. The lack of closed-form Riemannian operators, the non-differentiability of their discrete counterparts, and poor parallelisation capabilities have been the main obstacles to the development of the field on meshes. A principled framework to compute the exponential map on Riemannian surfaces discretised as meshes is straightest geodesics, which also allows to trace geodesics and parallel-transport vectors as a by-product. We provide a parallel GPU implementation and derive two different methods for differentiating through the straightest geodesics, one leveraging an extrinsic proxy function and one based upon a geodesic finite differences scheme. After proving our parallelisation performance and accuracy, we demonstrate how our differentiable exponential map can improve learning and optimisation pipelines on general geometries. In particular, to showcase the versatility of our method, we propose a new geodesic convolutional layer, a new flow matching method for learning on meshes, and a second-order optimiser that we apply to centroidal Voronoi tessellation. Our code, models, and pip-installable library (digeo) are available at: circle-group.github.io/research/DSG.
Read the full abstract ▾
All authors Hippolyte Verninas, Caner Korkmaz, Stefanos Zafeiriou, Tolga Birdal, Simone Foti
Computer Vision
Domain Adaptation Without the Compute Burden for Efficient Whole Slide Image Analysis
Umar Marikkar, Muhammad Awais and Sara Atito
Computational methods on analyzing Whole Slide Images (WSIs) enable early diagnosis and treatments by supporting pathologists in detection and classification of tumors. However, the extremely high resolution of WSIs makes end-to-end training impractical compared to typical image analysis tasks. To address this, most approaches use pre-trained feature extractors to obtain fixed representations of whole slides, which are then combined with Multiple Instance Learning (MIL) for downstream tasks. These feature extractors are typically pre-trained on natural image
datasets such as ImageNet, which fail to capture domain-specific characteristics. Although domain-specific pre-training on histopathology data yields more relevant feature representations, it remains computationally expensive and fail to capture task-specific characteristics within the domain. To address the computational cost and lack of task-specificity in domain-specific pre-training, we propose EfficientWSI (eWSI), a careful integration of Parameter-Efficient-Fine-Tuning (PEFT) and Multiple Instance Learning (MIL) that enables end-to-end training on WSI tasks. We evaluate eWSI on seven WSI-level tasks over Camelyon16, TCGA and BRACS
datasets. Our results show that eWSI when applied with ImageNet feature extractors yields strong classification performance, matching or outperforming MILs with in-domain feature extractors, alleviating the need for extensive in-domain pre-training. Furthermore, when eWSI is applied with in-domain feature extractors, it further improves classification performance in most cases, demonstrating its ability to capture task-specific information where beneficial. Our findings suggest that eWSI provides a task-targeted, computationally efficient path for WSI tasks, offering a promising direction for task-specific learning in computational pathology.
Read the full abstract ▾
Computer Vision
CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning
Marcell Kegl, Andras Palffy, Csaba Benedek et al. (+1 more)
In this paper, we address extrinsic calibration for camera,
lidar, and 4D radar sensors. Accurate extrinsic calibration of radar remains a challenge due to the sparsity of its data. We propose CLRNet, a novel, multi-modal end-to-end deep learning (DL) calibration network capable of addressing joint camera-
lidar-radar calibration, or pairwise calibration between any two of these sensors. We incorporate equirectangular projection, camera-based depth image prediction, additional radar channels, and leverage
lidar with a shared feature space and loop closure loss. In extensive experiments using the View-of-Delft and Dual-Radar
datasets, we demonstrate superior calibration accuracy compared to existing state-of-the-art methods, reducing both median translational and rotational calibration errors by at least 50%. Finally, we examine the domain transfer capabilities of the proposed network and baselines, when evaluating across
datasets. The code will be made publicly available upon acceptance at: https://github.com/tudelft-iv.
Read the full abstract ▾
All authors Marcell Kegl, Andras Palffy, Csaba Benedek, Dariu M. Gavrila
Computer Vision · Cross-listed from cs.AR
GLANCE: Gaze-Led Attention Network for Compressed Edge-inference
Neeraj Solanki, Hong Ding, Sepehr Tabrizchi et al. (+4 more)
Real-time object detection in AR/VR systems faces critical computational constraints, requiring sub-10\,ms latency within tight power budgets. Inspired by biological foveal
vision, we propose a two-stage pipeline that combines differentiable weightless neural networks for ultra-efficient gaze estimation with
attention-guided region-of-interest object detection. Our approach eliminates arithmetic-intensive operations by performing gaze tracking through
memory lookups rather than multiply-accumulate computations, achieving an angular error of $8.32^{\circ}$ with only 393 MACs and 2.2 KiB of
memory per frame. Gaze predictions guide selective object detection on attended regions, reducing computational burden by 40-50% and energy consumption by 65%. Deployed on the Arduino Nano 33 BLE, our system achieves 48.1% mAP on COCO (51.8% on attended objects) while maintaining sub-10\,ms latency, meeting stringent AR/VR requirements by improving the communication time by $\times 177$. Compared to the global YOLOv12n baseline, which achieves 39.2%, 63.4%, and 83.1% accuracy for small, MEDium, and LARGE objects, respectively, the ROI-based method yields 51.3%, 72.1%, and 88.1% under the same settings. This work shows that
memory-centric architectures with explicit
attention modeling offer better efficiency and accuracy for resource-constrained wearable platforms than uniform processing.
Read the full abstract ▾
All authors Neeraj Solanki, Hong Ding, Sepehr Tabrizchi, Ali Shafiee Sarvestani, Shaahin Angizi, David Z. Pan, Arman Roohi
Computer Vision · Cross-listed from cs.LG
Transition Flow Matching
Chenrui Ma
Mainstream flow matching methods typically focus on learning the local velocity field, which inherently requires multiple integration steps during generation. In contrast, Mean Velocity Flow models establish a relationship between the local velocity field and the global mean velocity, enabling the latter to be learned through a mathematically grounded formulation and allowing generation to be transferred to arbitrary future time points. In this work, we propose a new paradigm that directly learns the transition flow. As a global quantity, the transition flow naturally supports generation in a single step or at arbitrary time points. Furthermore, we demonstrate the connection between our approach and Mean Velocity Flow, establishing a unified theoretical perspective. Extensive experiments validate the effectiveness of our method and support our theoretical claims.
Read the full abstract ▾
Computer Vision
Towards Generalizable Robotic Manipulation in Dynamic Environments
Heng Fang, Shangru Li, Shuhan Wang et al. (+3 more)
Vision-
Language-Action (VLA) models excel in static manipulation but struggle in dynamic environments with moving targets. This performance gap primarily stems from a scarcity of dynamic manipulation
datasets and the reliance of mainstream VLAs on single-frame observations, restricting their spatiotemporal
reasoning capabilities. To address this, we introduce DOMINO, a large-scale
dataset and
benchmark for generalizable dynamic manipulation, featuring 35 tasks with hierarchical complexities, over 110K expert trajectories, and a multi-dimensional evaluation suite. Through comprehensive experiments, we systematically evaluate existing VLAs on dynamic tasks, explore effective training strategies for dynamic awareness, and validate the generalizability of dynamic data. Furthermore, we propose PUMA, a dynamics-aware VLA architecture. By integrating scene-centric historical optical flow and specialized world queries to implicitly forecast object-centric future states, PUMA couples history-aware perception with short-horizon prediction. Results demonstrate that PUMA achieves state-of-the-art performance, yielding a 6.3% absolute improvement in success rate over baselines. Moreover, we show that training on dynamic data fosters robust spatiotemporal representations that transfer to static tasks. All code and data are available at https://github.com/H-EmbodVis/DOMINO.
Read the full abstract ▾
All authors Heng Fang, Shangru Li, Shuhan Wang, Xuanyang Xi, Dingkang Liang, Xiang Bai
Computer Vision
Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models
Yulin Luo, Hao Chen, Zhuangzhe Wu et al. (+10 more)
Vision-
Language-Action (VLA) models have recently emerged as a promising paradigm for robotic manipulation, in which reliable action prediction critically depends on accurately interpreting and integrating visual observations conditioned on
language instructions. Although recent works have sought to enhance the visual capabilities of VLA models, most approaches treat the LLM backbone as a black box, providing limited insight into how visual information is grounded into action generation. Therefore, we perform a systematic analysis of multiple VLA models across different action-generation paradigms and observe that sensitivity to visual
tokens progressively decreases in deeper layers during action generation. Motivated by this observation, we propose DeepVision-VLA, built on a
Vision-
Language Mixture-of-
Transformers (VL-MoT) framework. This framework enables shared
attention between the
vision foundation model and the VLA backbone, injecting multi-level visual features from the
vision expert into deeper layers of the VLA backbone to enhance visual representations for precise and complex manipulation. In addition, we introduce Action-Guided Visual
Pruning (AGVP), which leverages shallow-layer
attention to prune irrelevant visual
tokens while preserving task-relevant ones, reinforcing critical visual cues for manipulation with minimal computational overhead. DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively, providing new insights for the design of visually enhanced VLA models.
Read the full abstract ▾
All authors Yulin Luo, Hao Chen, Zhuangzhe Wu, Bowen Sui, Jiaming Liu, Chenyang Gu, Zhuoyang Liu, Qiuxuan Feng, Jiale Yu, Shuo Gu, Peng Jia, Pheng-Ann Heng, Shanghang Zhang
Computer Vision
GlyphPrinter: Region-Grouped Direct Preference Optimization for Glyph-Accurate Visual Text Rendering
Xincheng Shuai, Ziye Li, Henghui Ding et al. (+1 more)
Generating accurate glyphs for visual text rendering is essential yet challenging. Existing methods typically enhance text rendering by training on a large amount of high-quality scene text images, but the limited coverage of glyph variations and excessive stylization often compromise glyph accuracy, especially for complex or out-of-domain characters. Some methods leverage reinforcement learning to alleviate this issue, yet their reward models usually depend on text recognition systems that are insensitive to fine-grained glyph errors, so images with incorrect glyphs may still receive high rewards. Inspired by Direct Preference Optimization (DPO), we propose GlyphPrinter, a preference-based text rendering method that eliminates reliance on explicit reward models. However, the standard DPO objective only models overall preference between two samples, which is insufficient for visual text rendering where glyph errors typically occur in localized regions. To address this issue, we construct the GlyphCorrector
dataset with region-level glyph preference annotations and propose Region-Grouped DPO (R-GDPO), a region-based objective that optimizes inter- and intra-sample preferences over annotated regions, substantially enhancing glyph accuracy. Furthermore, we introduce Regional Reward Guidance, an inference strategy that samples from an optimal distribution with controllable glyph accuracy. Extensive experiments demonstrate that the proposed GlyphPrinter outperforms existing methods in glyph accuracy while maintaining a favorable balance between stylization and precision.
Read the full abstract ▾
All authors Xincheng Shuai, Ziye Li, Henghui Ding, Dacheng Tao
Computer Vision
Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion
Zhenghong Zhou, Xiaohang Zhan, Zhiqin Chen et al. (+8 more)
Recent
video diffusion models achieve strong visual quality and temporal coherence, but still lack coordinated control over scene layout, customized subject identity, and camera/subject motion. We study controllable
video generation from three
prompts: a first-frame scene image, 3D-aware multi-view subject references, and a motion-driving signal. This setting is challenging because background regions are often trackable under camera motion, while foreground subjects can rotate, self-occlude, and reveal new regions that require identity-consistent appearance. We introduce Tri-Prompting, a
video diffusion framework that combines multi-view subject conditioning with dual-conditioned motion control. Tri-Prompting uses XYZ tracking points for visible background motion and low-resolution RGB proxies for foreground subject pose, allowing the model to recover fine appearance from multi-view references while retaining flexibility for plausible subject-scene interactions. An inference-time ControlNet scale schedule further balances motion controllability and visual realism. We evaluate Tri-Prompting against specialized baselines: DaS for motion reconstruction and Phantom for subject-driven generation. Tri-Prompting achieves competitive or improved reconstruction quality, stronger multi-view identity preservation, and better 3D consistency, while enabling 3D-aware subject insertion and in-scene manipulation.
Read the full abstract ▾
All authors Zhenghong Zhou, Xiaohang Zhan, Zhiqin Chen, Soo Ye Kim, Nanxuan Zhao, Haitian Zheng, Qing Liu, He Zhang, Zhe Lin, Yuqian Zhou, Jiebo Luo
Computer Vision
HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions
Yukang Cao, Haozhe Xie, Fangzhou Hong et al. (+4 more)
We present HSImul3R, a unified framework for simulation-ready 3D reconstruction of human-scene interactions (HSI) from casual captures, including sparse-view images and
monocular videos. Existing methods suffer from a perception-simulation gap: visually plausible reconstructions often violate physical constraints, leading to instability in physics engines and failure in
embodied AI applications. To bridge this gap, we introduce a physically-grounded bi-directional optimization pipeline that treats the physics simulator as an active supervisor to jointly refine human dynamics and scene geometry. In the forward direction, we employ Scene-targeted Reinforcement Learning to optimize human motion under dual supervision of motion fidelity and contact stability. In the reverse direction, we propose Direct Simulation Reward Optimization, which leverages simulation feedback on gravitational stability and interaction success to refine scene geometry. We further present HSIBench, a new
benchmark with diverse objects and interaction scenarios. Extensive experiments demonstrate that HSImul3R produces the first stable, simulation-ready HSI reconstructions and can be directly deployed to real-world
humanoid robots.
Read the full abstract ▾
All authors Yukang Cao, Haozhe Xie, Fangzhou Hong, Long Zhuo, Zhaoxi Chen, Liang Pan, Ziwei Liu
Computer Vision
Fast SAM 3D Body: Accelerating SAM 3D Body for Real-Time Full-Body Human Mesh Recovery
Timing Yang, Sicheng He, Hongyi Jing et al. (+4 more)
SAM 3D Body (3DB) achieves state-of-the-art accuracy in
monocular 3D human mesh recovery, yet its inference latency of several seconds per image precludes real-time application. We present Fast SAM 3D Body, a training-free acceleration framework that reformulates the 3DB inference pathway to achieve interactive rates. By decoupling serial spatial dependencies and applying architecture-aware
pruning, we enable parallelized multi-crop feature extraction and streamlined
transformer decoding. Moreover, to extract the joint-level kinematics (SMPL) compatible with existing
humanoid control and policy learning frameworks, we replace the iterative mesh fitting with a direct feedforward mapping, accelerating this specific conversion by over 10,000x. Overall, our framework delivers up to a 10.9x end-to-end speedup while maintaining on-par reconstruction fidelity, even surpassing 3DB on
benchmarks such as LSPET. We demonstrate its utility by deploying Fast SAM 3D Body in a
vision-only teleoperation system that-unlike methods reliant on wearable IMUs-enables real-time
humanoid control and the direct collection of manipulation policies from a single RGB stream.
Read the full abstract ▾
All authors Timing Yang, Sicheng He, Hongyi Jing, Jiawei Yang, Zhijian Liu, Chuhang Zou, Yue Wang
Computer Vision · Cross-listed from cs.RO
From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation
Yibin Liu, Yaxing Lyu, Daqi Gao et al. (+5 more)
Accurate process supervision remains a critical challenge for long-horizon robotic manipulation. A primary bottleneck is that current
video MLLMs, trained primarily under a Supervised Fine-Tuning (SFT) paradigm, function as passive "Observers" that recognize ongoing events rather than evaluating the current state relative to the final task goal. In this paper, we introduce PRIMO R1 (Process
Reasoning Induced Monitoring), a 7B framework that transforms
video MLLMs into active "Critics". We leverage outcome-based Reinforcement Learning to incentivize explicit Chain-of-Thought generation for progress estimation. Furthermore, our architecture constructs a structured temporal input by explicitly anchoring the
video sequence between initial and current state images. Supported by the proposed PRIMO
Dataset and
Benchmark, extensive experiments across diverse in-domain environments and out-of-domain real-world
humanoid scenarios demonstrate that PRIMO R1 achieves state-of-the-art performance. Quantitatively, our 7B model achieves a 50% reduction in the mean absolute error of specialized
reasoning baselines, demonstrating significant relative accuracy improvements over 72B-scale general MLLMs. Furthermore, PRIMO R1 exhibits strong zero-shot generalization on difficult failure detection tasks. We establish state-of-the-art performance on RoboFail
benchmark with 67.0% accuracy, surpassing closed-source models like OpenAI o1 by 6.0%.
Read the full abstract ▾
All authors Yibin Liu, Yaxing Lyu, Daqi Gao, Zhixuan Liang, Weiliang Tang, Shilong Mu, Xiaokang Yang, Yao Mu
Computer Vision · Cross-listed from cs.SD
AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer
Pengjun Fang, Yingqing He, Yazhou Xing et al. (+3 more)
Existing
video-to-audio (V2A) generation methods predominantly rely on text
prompts alongside visual information to synthesize audio. However, two critical bottlenecks persist: semantic granularity gaps in training data, such as conflating acoustically distinct sounds under coarse labels, and textual ambiguity in describing micro-acoustic features. These bottlenecks make it difficult to perform fine-grained sound synthesis using text-controlled modes. To address these limitations, we propose AC-Foley, an audio-conditioned V2A model that directly leverages reference audio to achieve precise and fine-grained control over generated sounds. This approach enables fine-grained sound synthesis, timbre transfer, zero-shot sound generation, and improved audio quality. By directly conditioning on audio signals, our approach bypasses the semantic ambiguities of text descriptions while enabling precise manipulation of acoustic attributes. Empirically, AC-Foley achieves state-of-the-art performance for Foley generation when conditioned on reference audio, while remaining competitive with state-of-the-art
video-to-audio methods even without audio conditioning. Code and demo are available at: https://ff2416.github.io/AC-Foley-Page
Read the full abstract ▾
All authors Pengjun Fang, Yingqing He, Yazhou Xing, Qifeng Chen, Ser-Nam Lim, Harry Yang
Computer Vision
Grounding World Simulation Models in a Real-World Metropolis
Junyoung Seo, Hyunwook Choi, Minkyung Kwon et al. (+10 more)
What if a world simulation model could render not an imagined environment but a city that actually exists? Prior generative world models synthesize visually plausible yet artificial environments by imagining all content. We present Seoul World Model (SWM), a city-scale world model grounded in the real city of Seoul. SWM anchors autoregressive
video generation through
retrieval-augmented conditioning on nearby street-view images. However, this design introduces several challenges, including temporal misalignment between retrieved references and the dynamic target scene, limited trajectory diversity and data sparsity from vehicle-mounted captures at sparse intervals. We address these challenges through cross-temporal pairing, a large-scale synthetic
dataset enabling diverse camera trajectories, and a view interpolation pipeline that synthesizes coherent training
videos from sparse street-view images. We further introduce a Virtual Lookahead Sink to stabilize long-horizon generation by continuously re-grounding each chunk to a retrieved image at a future location. We evaluate SWM against recent
video world models across three cities: Seoul, Busan, and Ann Arbor. SWM outperforms existing methods in generating spatially faithful, temporally consistent, long-horizon
videos grounded in actual urban environments over trajectories reaching hundreds of meters, while supporting diverse camera movements and text-prompted scenario variations.
Read the full abstract ▾
All authors Junyoung Seo, Hyunwook Choi, Minkyung Kwon, Jinhyeok Choi, Siyoon Jin, Gayoung Lee, Junho Kim, JoungBin Lee, Geonmo Gu, Dongyoon Han, Sangdoo Yun, Seungryong Kim, Jin-Hwa Kim
Computer Vision · Cross-listed from cond-mat.mtrl-sci
Benchmarking Machine Learning Approaches for Polarization Mapping in Ferroelectrics Using 4D-STEM
Matej Martinc, Goran Dražič, Anton Kokalj et al. (+5 more)
Four-dimensional scanning transmission electron microscopy (4D-STEM) provides rich, atomic-scale insights into materials structures. However, extracting specific physical properties - such as polarization directions essential for understanding functional properties of ferroelectrics - remains a significant challenge. In this study, we systematically
benchmark multiple machine learning models, namely ResNet, VGG, a custom convolutional neural network, and PCA-informed k-Nearest Neighbors, to automate the detection of polarization directions from 4D-STEM diffraction patterns in ferroelectric potassium sodium niobate. While models trained on synthetic data achieve high accuracy on idealized synthetic diffraction patterns of equivalent thickness, the domain gap between simulation and experiment remains a critical barrier to real-world deployment. In this context, a custom made prototype representation training regime and PCA-based methods, combined with data augmentation and filtering, can better bridge this gap. Error analysis reveals periodic missclassification patterns, indicating that not all diffraction patterns carry enough information for a successful classification. Additionally, our qualitative analysis demonstrates that irregularities in the model's prediction patterns correlate with defects in the
crystal structure, suggesting that supervised models could be used for detecting structural defects. These findings guide the development of robust, transferable machine learning tools for electron microscopy analysis.
Read the full abstract ▾
All authors Matej Martinc, Goran Dražič, Anton Kokalj, Katarina Žiberna, Janina Roknić, Matic Poberžnik, Sašo Džeroski, Andreja Benčan Golob
Computer Vision
Severe Domain Shift in Skeleton-Based Action Recognition:A Study of Uncertainty Failure in Real-World Gym Environments
Aaditya Khanal and Junxiu Zhou
The practical deployment gap – transitioning from controlled multi-view 3D skeleton capture to unconstrained
monocular 2D pose estimation – introduces a compound domain shift whose safety implications remain critically underexplored. We present a systematic study of this severe domain shift using a novel Gym2D
dataset (style/viewpoint shift) and the UCF101
dataset (semantic shift). Our Skeleton
Transformer achieves 63.2% cross-subject accuracy on NTU-120 but drops to 1.6% under zero-shot transfer to the Gym domain and 1.16% on UCF101. Critically, we demonstrate that high Out-Of-Distribution (OOD) detection AUROC does not guarantee safe selective classification. Standard uncertainty methods fail to detect this performance drop: the model remains confidently incorrect with 99.6% risk even at 50% coverage across both OOD
datasets. While energy-based scoring (AUROC >= 0.91) and Mahalanobis distance provide reliable distributional detection signals, such high AUROC scores coexist with poor risk-coverage behavior when making decisions. A lightweight finetuned gating mechanism restores calibration and enables graceful abstention, substantially reducing the rate of confident wrong predictions. Our work challenges standard deployment assumptions, providing a principled safety analysis of both semantic and
geometric skeleton recognition deployment.
Read the full abstract ▾
Computer Vision
Panoramic Affordance Prediction
Zixin Zhang, Chenfei Liao, Hongfei Zhang et al. (+10 more)
Affordance prediction serves as a critical bridge between perception and action in
embodied AI. However, existing research is confined to pinhole camera models, which suffer from narrow Fields of View (FoV) and fragmented observations, often missing critical holistic environmental context. In this paper, we present the first exploration into Panoramic Affordance Prediction, utilizing 360-degree imagery to capture global spatial relationships and holistic scene understanding. To facilitate this novel task, we first introduce PAP-12K, a large-scale
benchmark dataset containing over 1,000 ultra-high-resolution (12k, 11904 x 5952) panoramic images with over 12k carefully annotated QA pairs and affordance masks. Furthermore, we propose PAP, a training-free, coarse-to-fine pipeline inspired by the human foveal visual system to tackle the ultra-high resolution and severe distortion inherent in panoramic images. PAP employs recursive visual routing via grid prompting to progressively locate targets, applies an adaptive gaze mechanism to rectify local
geometric distortions, and utilizes a cascaded grounding pipeline to extract precise instance-level masks. Experimental results on PAP-12K reveal that existing affordance prediction methods designed for standard perspective images suffer severe performance degradation and fail due to the unique challenges of panoramic
vision. In contrast, PAP framework effectively overcomes these obstacles, significantly outperforming state-of-the-art baselines and highlighting the immense potential of panoramic perception for robust
embodied intelligence.
Read the full abstract ▾
All authors Zixin Zhang, Chenfei Liao, Hongfei Zhang, Harold Haodong Chen, Kanghao Chen, Zichen Wen, Litao Guo, Bin Ren, Xu Zheng, Yinchuan Li, Xuming Hu, Nicu Sebe, Ying-Cong Chen