Computer Vision
Time Imprint: Learning Time-Aware Representations in Multi-Modal Knowledge Graphs
Pengyu Zhang, Klim Zaporojets, Congfeng Cao et al. (+2 more)
Multi-Modal Knowledge Graphs (MMKGs) enrich entities with multiple modalities such as text and images, yet entities with highly similar multi-modal features remain difficult to distinguish. Temporal information of an entity can serve as an additional modality to disambiguate such entities, but existing approaches rarely treat time as a separate modality alongside text and images due to two major challenges: (1) sparse temporal semantics, which hinder
alignment with richer modalities, and (2) multiple timestamps, which introduce noise or reduce robustness in representation learning. To address these challenges, we propose Time Imprint, a framework that treats time as an entity-level modality and jointly aligns temporal, textual, and visual representations via a three-view contrastive objective. Additionally, to mitigate multi-timestamp ambiguity, Time Imprint studies a compact timestamp subset selection design space and aggregates the selected timestamps into a discriminative temporal
embedding with
attention pooling, balancing temporal specificity and robustness. Experiments on three MMKG
benchmarks demonstrate that Time Imprint achieves state-of-the-art link prediction performance, improving Hits@1 by up to 6.07% overall and yielding up to 58% gains on the subset of the top-1% ambiguity samples. We further examine different fusion strategies and the sensitivity to timestamp availability and quality, clarifying when and why time-as-modality is most beneficial, while adding only modest training overhead. We release our code at https://anonymous.4open.science/r/Time-Imprint.
Read the full abstract ▾
All authors Pengyu Zhang, Klim Zaporojets, Congfeng Cao, Jia-Hong Huang, Paul Groth
Computer Vision · Cross-listed from cs.GR
GReFEM: Multimodal LLMs as Zero-Shot Semantic Assistants for Physics-Guided 3D Mesh Refinement
Kartik Bali, Mahish K. Guru, Christian J Cyron et al. (+1 more)
Adaptive volumetric finite element meshing traditionally requires iterative PDE solvers or heavily supervised, data-driven surrogates trained on large-scale simulation data. This work explores a significant question: can the high-level semantic understanding of off-the-shelf MLLMs serve as a viable, zero-shot
geometric proxy for finite element mesh refinement? We introduce $\textbf{GReFEM}$ (
Geometric Reasoning Enhanced
Multimodal LLMs for Finite Element Meshing), a framework that utilizes MLLMs to visually localize stress-critical regions based on physics-guided textual
prompts. To bridge the gap between 2D MLLM pre-training and 3D geometries, we introduce $\textit{orthoViews}$, a trained view-selection module that assists these MLLMs by maximizing the observability of key
geometric features. We evaluate $\textbf{GReFEM}$ by computing finite-element error at stress concentrations, on diverse CAD mesh geometries under multiple loading cases, with MLLMs and
prompt variants, against blind and load-informed
geometric heuristics, solver-informed Zienkiewicz–Zhu (ZZ) refinement, and a retrained learned adaptive-meshing baseline, all at a matched element budget. Without any FEM solve or simulation training data, $\textbf{GReFEM}$ lowers the critical-region peak von Mises stress error relative to a blind
geometric heuristic and to refinement driven by a coarse FEM solve, closing $58\%$ of the gap between the unrefined mesh and a reference-solve oracle, while a hand-derived analytical load-informed heuristic remains stronger and the learned baseline performs slightly better overall. With this we investigate their performance as semantic assistants in automated simulation workflows. Code is available at https://github.com/kbali1297/GReFEM
Read the full abstract ▾
All authors Kartik Bali, Mahish K. Guru, Christian J Cyron, Roland Aydin
Computer Vision
Beyond Thermal Imaging: Inferring Thermophysical Properties from Time-Resolved Thermal Observations
Chenghao Xu, Malcolm Mielle and Olga Fink
Inferring latent physical properties from sensory observations is a fundamental challenge in machine perception. Among available sensing modalities, thermal imaging is particularly promising because temperature
evolution is directly governed by heat-transfer physics and therefore encodes information about underlying thermophysical properties of a scene. Recovering spatially resolved thermophysical properties from thermal observations could transform applications ranging from digital twins and infrastructure monitoring to robotics and scientific imaging. However, existing thermal scene reconstruction methods can recover temperature fields in complex 3D environments without identifying the thermophyiscal properties that govern thermal
evolution, whereas inverse methods provide physically interpretable parameter estimation but typically rely on simplified geometries and controlled experimental conditions. Here we introduce ThermoField, a framework that unifies thermal scene reconstruction and thermophysical parameter estimation through differentiable heat-transfer simulation. The proposed framework represents these quantities as spatially varying neural fields and constrains them through scene geometry, governing heat-transfer physics, and temporal thermal observations. We demonstrate that ThermoField jointly reconstructs geometry, estimates spatially varying thermal diffusivity, and predicts thermal
evolution under previously unseen environmental conditions. By integrating neural scene representations with differentiable heat-transfer solver, the framework enables physically interpretable parameter inference in complex 3D scenes. Our results establish a bridge between thermal scene reconstruction and inverse heat-transfer analysis, providing a unified approach for geometry reconstruction, thermophysical property estimation, and predictive thermal simulation from thermal observations.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
Evaluating the Effect of Frame Rate in Sequence-Based Classification of Autism-Related Self-Stimulatory Hand Idiosyncrasies
Raunak Mondal and Peter Washington
Autism spectrum disorder (ASD) affects over 75 million individuals worldwide, yet scalable computational methods for remote behavioral screening remain limited. This study addresses two complementary challenges in automated detection of autism-related self-stimulatory behaviors from
video: (1) identifying the optimal sequence-based neural network architecture and temporal sampling rate, and (2) characterizing data augmentation strategies for training on small behavioral
datasets. For the first objective, long short-term
memory (LSTM) and gated recurrent unit (GRU) models were trained on pose-derived features from the Self-Stimulatory Behavior Diagnosis (SSBD)
dataset at frame sampling intervals of 1, 5, 15, 30, 45, and 90 frames. Both architectures exceeded prior convolutional neural network (CNN) baselines (62-76% accuracy), with peak accuracies of 97.5% (LSTM) and 98.75% (GRU) at a sampling interval of every 15 frames. For the second objective, ten data augmentation strategies were applied to an I3D transfer learning pipeline, with an ablation study quantifying the marginal contribution of each technique. Horizontal flip achieved the highest standalone accuracy (48.78%), while exclusion of upsampling from the augmentation pipeline produced the largest performance degradation, indicating its necessity for complex behavioral
video augmentation. A personalized machine learning approach, in which per-subject models were trained and tested on temporally split segments of each
video, produced consistent predictions (mean loss 1.84, SD 0.79). These results provide practitioners with concrete guidance on architecture selection, sampling rate, and augmentation strategy for
video-based behavioral classification in data-scarce clinical domains.
Read the full abstract ▾
Computer Vision
Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT
Giulia Marchiori Pietrosanti, Giulio Rossolini and Giorgio Buttazzo
Vision Transformers (ViTs) remain vulnerable to localized
adversarial attacks, e.g.,
adversarial patches, while recent test-time defenses mitigate them by suppressing image
tokens with abnormally high
attention scores. These defenses exploit a strong coupling between
attention and
adversarial effectiveness:
adversarial tokens often need to attract substantial
attention to influence the prediction. We introduce
adversarial decoys, independently optimized image patches that redirect the
attention, and therefore related defenses, toward selected target
tokens. Rather than jointly optimizing misclassifications and defense evasion, our approach decouples the two objectives: the original
adversarial region induces the incorrect prediction, while a separate decoy manipulates the
attention ranking used by the defense. A layer-wise objective increases target-
token attention and promotes these
tokens above competing non-target ones. Since the decoy is optimized independently of the underlying attack, the method is attack-agnostic and can be easily integrated with any existing
adversarial patch attack. Experiments on ImageNet across multiple ViT architectures and attacks show that decoys can redirect high
attention scores away from the true
adversarial region while preserving much of the attack effectiveness. These results reveal a fundamental limitation of using
attention magnitude as an indicator of
adversarial relevance.
Read the full abstract ▾
Computer Vision
3D Reconstruction of deciduous Trees using low-cost UAV- and Crane-based Photogrammetry for Monitoring Shoot Elongation across entire Canopies
Stephan Nebiker, Micha Tschanz, Nando Amport et al. (+1 more)
Tree growth determines how much CO2 is sequestered from the atmosphere and temporarily stored in woody biomass. At the same time tree growth is affected by increasing temperatures, more frequent drought periods, late frosts and other extreme events associated with
climate change. While continuous measurements of radial (secondary) tree growth using dendrometers are well established, monitoring of shoot elongation (primary growth) has largely been neglected because suitable measurement techniques are lacking. As a result, the effects of
climate change on primary tree growth remain insufficiently understood. This work aims at reconstructing native deciduous trees in 3D as a basis for measuring and monitoring shoot elongation over entire tree canopies. Here we explored the use of low-cost
UAV photogrammetry and of a multi-camera CraneCam system under real-world conditions. Data were collected in two study areas over an entire growing season. We present sensor evaluations, photogrammetric data acquisition and processing strategies. A special focus is placed on the analysis of the resulting photogrammetric 3D point clouds in terms of accuracy, resolution and completeness. Results demonstrate 3D point accuracies of 5-6 mm for entire trees using consumer-grade
UAVs weighing less than 250 g and a 3D reconstruction completeness between 92% and 98% depending on the
UAV type. The paper introduces a novel 3Dprinted ground-truth branch to evaluate the capability to reconstructing fine-detail structures such as thin tree shoots. Finally, we discuss operational challenges and initial experiments towards a skeletonization of entire trees based on photogrammetric point clouds.
Read the full abstract ▾
All authors Stephan Nebiker, Micha Tschanz, Nando Amport, Frederik Baumgarten
Computer Vision · Cross-listed from cs.RO
Time-to-Collision Based Dynamic Obstacle Avoidance Using Pretrained Vision Models for Robots in Unstructured Environments
Erik Jagnandan, Mulugeta Haile, Gregory Barber et al. (+1 more)
Dynamic obstacle avoidance in unstructured outdoor environments remains a critical challenge for autonomous mobile
robots, particularly when large-scale
robot-specific training data and simulation-based policies are impractical. We present a data-efficient, interpretable method for
vision-based dynamic obstacle avoidance that operates entirely on real-world data, avoiding the sim-to-real transfer problem inherent in simulation-trained policies. Our approach leverages UniDepth, a large pretrained
monocular depth estimation model, to produce dense depth maps from RGB
video without requiring stereo cameras or
LiDAR at inference time. Dynamic obstacle avoidance is achieved by extending the SuperPoint and SuperGlue feature correspondence pipeline to track keypoints across long frame sequences, projecting their 2D pixel-space positions into 3D using camera intrinsics and predicted depth, running bundle adjustment initialized from these 3D keypoints, and computing per-keypoint time-to-collision (TTC). A 2D motion primitive in the ground plane is then selected to move the
robot away from the closest point of approach of the minimum-TTC keypoint. Evaluated on real-world data from the M3ED
dataset, our pipeline achieves a precision of 0.49 and a recall of 0.38 in identifying frames with a ground truth TTC below 1 second, and correctly generates the evasive motion direction in 84% of true positive detections. Crucially, it detects at least one frame with TTC less than 1 second for 20 out of 22 unique physical obstacles present in our test sequences. Unlike end-to-end learned methods that demand thousands of hours of
robot-specific training data, our approach eliminates model training entirely, requiring only 74 seconds of data for hyperparameter tuning. This demonstrates exceptional data efficiency while preserving interpretable and generalizable behavior across diverse obstacle types.
Read the full abstract ▾
All authors Erik Jagnandan, Mulugeta Haile, Gregory Barber, Pratik Chaudhari
Computer Vision
GIRAF: Towards Generalizable Human Interactions with Articulated Objects
Xiaohan Zhang, Sebastian Starke, Alexander Winkler et al. (+3 more)
Synthesizing realistic full-body human interactions with articulated objects is a fundamental challenge for
embodied AI and graphics, with applications in robotics training and virtual
agents. Existing models remain limited: some focus on simple activities with static objects, while others restrict
attention to hand-only manipulation. This leaves open the problem of generating coordinated full-body motion that approaches, manipulates, and moves articulated objects in a realistic and generalizable way. The key difficulty lies in
reasoning jointly about locomotion, fine-grained contact, and object articulation. Models must capture subtle hand-object correspondences that transfer across object geometries, while also producing seamless transitions from
navigation to manipulation. At the same time, the scarcity of large-scale paired motion-scene data makes it difficult to generalize across diverse object positions and shapes. We introduce a text-conditioned
diffusion model that addresses these challenges through three core ideas: an object-centric representation that unifies hand-object contact with object surfaces, a mixed-domain training strategy that balances locomotion and interaction, and a contact-based augmentation scheme that expands training diversity. Through experiments, our method demonstrated strong generalization to unseen object configurations, surpassing current state-of-the-art methods.
Read the full abstract ▾
All authors Xiaohan Zhang, Sebastian Starke, Alexander Winkler, Federica Bogo, Samir Aroudj, Yuting Ye
Computer Vision · Cross-listed from eess.IV
False Confidence: Automated Labels Confound Fairness Audits in Cervical Spine Segmentation
Linus Juni, Aasa Feragen and Aditya Parikh
Automated
segmentation of cervical-spine MRI is increasingly used in clinical workflows, yet no
fairness audit exists for this anatomy. We show that
auditing these
segmentation tasks is complicated by a common property of modern
segmentation datasets: expert-annotated gold labels are expensive, so abundant machine-generated (silver) labels are added to limit annotation cost. This matters because the reference used to judge a model can itself be biased. In this study, we present the first
fairness audit of cervical-spine MRI
segmentation across sex, age, and race using the CSpineSeg
dataset. We observe that the deployed model is demographically fair, but the choice of reference label, however, is not neutral. Because a
dataset's silver labels are generated by a model trained on its gold labels, any new model trained on those same gold labels agrees more with the silver labels than with expert truth: scoring identical predictions against silver rather than gold overestimates performance by ~8 Dice points and turns the
fairness verdict for age from non-significant to significant - not by the gap inflation Parikh et al. report (which we term false magnitude) but by collapsing within-group variance (which we term false confidence). Reference-label provenance is thus a first-order confounder in
segmentation evaluation: performance and
fairness should be reported against expert labels, and any
fairness claim stated together with the provenance of its reference.
Read the full abstract ▾
Computer Vision
DreamCharacter-1: From 3D Generative Foundation Models to Product-Ready Character Generation
Weizhe Liu, Yunjie Wu, Xiangqian Shu et al. (+5 more)
We present DreamCharacter-1, a lightweight post-adaptation framework that calibrates pretrained 3D foundation models toward high-fidelity, production-ready 3D character generation. Building upon a 3D foundation backbone, our pipeline incorporates three task-oriented components: (1) geometry post-training, which enhances fine-grained surface details through
geometric preference optimization; (2) texture post-training, which synthesizes high-resolution textures and refines the appearance of occluded regions; and (3) inference acceleration, which enables scalable deployment. Extensive quantitative and qualitative experiments demonstrate that DreamCharacter-1 produces visually compelling and structurally robust 3D character assets, consistently surpassing state-of-the-art character generation methods.
Read the full abstract ▾
All authors Weizhe Liu, Yunjie Wu, Xiangqian Shu, Guangwei Wang, Xiangyu Xu, Peng Li, Yujie Li, Hengkai Guo
Computer Vision · Cross-listed from cs.LG
Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
Eric Zhu, Abhinav Shrivastava and Soumik Mukhopadhyay
Reinforcement learning from human feedback (RLHF) has emerged as a powerful paradigm for aligning generative models with human preferences. However, applying RLHF to
diffusion models remains highly feedback inefficient, as existing approaches typically require large amounts of human or reward model evaluations. This limitation reduces the practicality of
diffusion RLHF in realworld settings where feedback is the primary bottleneck. In this paper, we propose two complementary strategies that substantially improve the feedback efficiency of
diffusion RLHF while preserving generalization to unseen
prompts. Our key observation is that reward information in
diffusion trajectories is unevenly distributed: not all denoising timesteps or trajectories contribute equally to learning from a reward signal. By emphasizing informative timesteps and trajectories during optimization, we obtain more effective gradient updates. First, we introduce a per-timestep weighting scheme that reweights denoising steps during policy optimization. We theoretically connect this weighting to the optimal convergence properties of proximal policy optimization (PPO) and approximate the resulting weighting trend empirically. Second, we introduce a replay mechanism that prioritizes informative trajectories, enabling the model to reuse past samples instead of repeatedly querying new rewards. Together, these strategies significantly improve the feedback efficiency of
diffusion RLHF. Under identical hyperparameter settings, our approach achieves up to a 6$\times$ improvement in sample efficiency compared to widely used
diffusion RLHF baselines.
Read the full abstract ▾
Computer Vision
Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence
Shuailei Ma, Jiaqi Liao, Xinyang Wang et al. (+24 more)
Despite the recent promise in
robot control,
video generative models suffer from a domain mismatch due to their primary focus on content creation. For example, their design inherently prioritizes visual fidelity and creativity over computational efficiency and physical realism. In this work, we present LingBot-
Video, a DiT-based
video pretraining paradigm specifically tailored for
embodied intelligence. From the architecture perspective, we adopt the Mixture-of-Experts (MoE), instead of dense, framework to achieve a better trade-off between modeling capacity and inference efficiency, and manage to scale it up from scratch. From the data perspective, we construct a data profiling engine that augments standard internet
videos with extensive
robot-oriented footage, encompassing manipulation,
navigation, and
egocentric perspectives, to equip the base model with an intrinsic understanding of actions and world dynamics. From the training perspective, we develop a multi-dimensional reward system to enforce the
alignment regarding physical rationality and task completion, going beyond standard criteria such as aesthetics,
prompt-following, and motion consistency. Comprehensive evaluations validate its performance and efficiency as a
video foundation model. We contribute LingBot-
Video as the inaugural large-scale, open-source MoE
video foundation model to the community, in a pioneering effort to bridge digital creativity and physical actuation.
Read the full abstract ▾
All authors Shuailei Ma, Jiaqi Liao, Xinyang Wang, Jingjing Wang, Chaoran Feng, Zijing Hu, Chong Bao, Zichen Xi, Yuqi Gan, Weisen Wang, Yanhong Zeng, Qin Zhao, Zifan Shi, Wei Wu, Hao Ouyang, Qiuyu Wang, Shangzhan Zhang, Jiahao Shao, Yipengjing Sun, Liangxiao Hu, Lunke Pan, Nan Xue, Kecheng Zheng, Yinghao Xu, Xing Zhu, Yujun Shen, Ka Leong Cheng
Computer Vision
MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models
Hyunjae Kim, Dain Kim, Pan Xiao et al. (+25 more)
Medicine is inherently
multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of
multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical
multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and
alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived
dataset. Across 26
benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the
vision encoder in a
multimodal large
language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two
benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image
retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical
multimodal foundation models across
benchmark and clinical settings. We publicly release the framework, corpus,
benchmarks, and pretrained models.
Read the full abstract ▾
All authors Hyunjae Kim, Dain Kim, Pan Xiao, Serina S. Applebaum, Younjoon Chung, Xuguang Ai, Yu Yin, Roy Jiang, Yuexi Du, Yawen Wei, Yiming Kong, Tuo Guo, Zhiyuan Cao, Mengmeng Du, Yuelei Fu, Yan Hu, Rui Shi, Gui Yang, Kevin W. Jin, Yuntian Liu, Yuxuan Tian, Jonathan Marquez, Zhen Chen, Sheng Zhang, Hoifung Poon, Hua Xu, Jaewoo Kang, Qingyu Chen
Computer Vision · Cross-listed from cs.RO
Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Hongyu Qu, Jianzhe Gao, Xiaobin Hu et al. (+6 more)
Mainstream
Vision-
Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing
memory-augmented VLAs either expand the observation window or retrieve history from the
memory bank as auxiliary policy-side context. However, they leave
memory outside the native latent
embedding space of VLA
reasoning, preventing historical experience from being fluidly interleaved with
multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-
memory-native framework that reconstructs historical experience into latent
memory tokens and directly interweaves them with VLA
reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term
memory vaults; (ii) a seeker that queries both vaults using the
multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent
memory tokens; and (iv) a weaver that injects these
memory tokens with the current observation and instruction into one continuous
embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables
memory to directly participate in VLA
reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.
Read the full abstract ▾
All authors Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang, Xinlei Yu, Rui Yan, Wenguan Wang, Xiangbo Shu, Shuicheng Yan
Computer Vision
Cardiac MRI Through-Plane Super-Resolution Guided by Reference and Memory
Shaoming Pan, Chenchuhui Hu, Leon Axel et al. (+1 more)
Clinical cardiac MRI is commonly acquired with high in-plane resolution but coarse through-plane resolution to reduce scan time and accommodate breath-hold and cardiac-motion constraints, which limits 3D analysis and diagnostic accuracy. We propose STRMSR, a reference- and
memory-guided through-plane super-resolution (SR) framework that reconstructs high-resolution (HR) cardiac volumes by leveraging HR reference views acquired from the same subject and intermediate SR results as the
memory. Our method uses coarse-to-fine contextual matching to establish robust correspondence between low-resolution target and reference/
memory images under spatial misalignment. A learnable patch-wise dynamic feature aggregation module predicts content-adaptive mixture weights for each local patch, effectively fusing dynamic information while suppressing unreliable feature transfers. The intermediate SR results stored in the
memory bank ensure slice-to-slice consistency for the super-resolved 3D volume. Experiments on the WHS cardiac MRI
dataset under two reference protocols, orthogonal-plane views and long-axis chamber views, demonstrate consistent improvements over baselines at 4x and 8x upsampling factors.Code is available at https://github.com/030108ming/STRMSR
Read the full abstract ▾
All authors Shaoming Pan, Chenchuhui Hu, Leon Axel, Meng Ye
Computer Vision
Automatic Echocardiography Segmentation via Transition Probability Correlation for Stable Semantic Extraction
Xinran Chen, Xiyuan Wang, Guangquan Zhou et al. (+1 more)
While echocardiography is essential for cardiovascular diagnosis, inherent speckle noise and low signal-to-noise ratio often lead to ambiguous semantic features and fragmented boundaries. These limitations significantly hinder the
segmentation accuracy of deep learning models in complex clinical cases. Moreover, temporal motion of the heart plays a critical role in recognizing anatomical structures. To address these challenges, we designed a STLSF module which comprises a window-matching-based semantic correction component and a semantics-guided texture enhancement component. By leveraging local transition probability correlations to correct semantics and employing semantics-guided texture enhancement, the STLSF module effectively mitigates texture instability and ambiguous semantic interpretations caused by disadvantaged echocardiography quality. Additionally, to facilitate the encoder's adaptation to the intrinsic priors of ultrasound-specific imaging patterns, we propose a frequency-aware denoising pre-training method. The entire work builds a convolution-based network with locality inductive bias and long-range dependencies. Extensive experiments confirm our SOTA performance, achieving 93.87% Dice on CAMUS and 92.62% on EchoNet-Dynamic, with respective HD95 values of 3.29mm and 2.73mm.
Read the full abstract ▾
All authors Xinran Chen, Xiyuan Wang, Guangquan Zhou, Chuan Chen
Computer Vision
AA-ViT: Anatomically Aware Vision Transformer with Structural and Frequency Guidance for Contrast Enhanced Brain MRI Synthesis
Talha Meraj, Tom Flannery, Charlie Cummins et al. (+6 more)
Accurate tumour localization and diagnosis is a critical component of clinical care for
brain cancers.
Magnetic Resonance Imaging (MRI) is the most commonly used imaging modality due to its superior soft-tissue contrast. However, standard MRI often exhibits limited contrast and imaging artifacts, which necessitates the use of contrast
agents to enhance lesion visibility. The administration of chemical contrast
agents is not always feasible and may be contraindicated in patients with renal impairment or other health conditions. As a result, developing accurate and non-invasive contrast enhanced MRI (CEMRI) synthesis methods has clinical importance. In recent years, numerous approaches for CEMRI synthesis have been proposed, predominantly relying on generative artificial intelligence models. While these methods demonstrate promising performance, their dependence on implicit feature learning often limits their ability to preserve anatomical boundaries and tumour-specific fine structures. To address these challenges, we propose an anatomically aware frequency-and-structure-guided
vision transformer (AA-ViT), for CEMRI synthesis using pre-contrast MRI modalities (T1, T2, and FLAIR). Experiments on the BraTS 2021
dataset demonstrate that the proposed method preserves anatomical and lesion boundaries, achieving higher PSNR and SSIM than state-of-the-art approaches. Clinical evaluation by three neuroradiologists and a neurosurgeon on 19 randomly selected cases across diverse gliomas yielded a mean score of 3.94/5, providing preliminary clinical validation rarely seen in prior studies. Synthetic post-contrast scans from our model could lower scanning costs, shorten imaging time, and avoid the potential risks of using gadolinium-based contrast
agents.
Read the full abstract ▾
All authors Talha Meraj, Tom Flannery, Charlie Cummins, Matt Townend, Thomas C Booth, Peter Crossley, Michael McCann, Ian Overton, Saritha Unnikrishnan
Computer Vision
Face-Trace: Open-Set Attribution and Progressive Discovery of Synthetic Face Generators
Alessia Infantino, Claudio Schiavella and Irene Amerini
Recent advances in generative Artificial Intelligence have made synthetic face images increasingly realistic, creating new challenges for multimedia forensics. Source attribution methods should identify the generator of an image when the source is known, but also handle samples produced by unseen models. Most existing approaches, however, address synthetic face attribution in a closed-set setting, assuming that test samples can only originate from generators observed during training. This assumption does not hold in real-world scenarios, where new generators continuously appear and detecting an image as unknown is not sufficient, since rejected samples should also be organized according to their underlying sources. We introduce Face-Trace, a pipeline for open-set synthetic face source attribution that combines known generator classification, energy-based rejection, and unknown generator discovery. A classifier trained on frozen I-JEPA
embeddings attributes known generators, while rejected samples are represented by combining projected I-JEPA features with complementary forensic traces and grouped to identify coherent sets of samples produced by unknown generators. We also extend the discovery stage to an incremental scenario, where rejected samples arrive over time. Experiments on the WILD
dataset show 96.73% closed-set attribution accuracy, while rejection reaches 71.25% balanced accuracy and rejected samples are clustered into meaningful unknown-generator groups, with an Adjusted Rand Index of 0.81, a Normalized Mutual Information of 0.90, and an overall purity of 87.74%. In the incremental setting, the discovered generator space is progressively extended while maintaining a final purity of 99.23%, and cross-
dataset experiments suggest that the pipeline can operate beyond the original data distribution.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
SonoRank: Towards Calibration-Free Real-Time Finger Flexion Detection from Forearm Ultrasound Sequences
Dean Zadok, Alon Wolf, Alex M. Bronstein et al. (+1 more)
Powered prosthetic hands are frequently abandoned, largely due to the limited functionality of current devices that rely on surface electromyography (sEMG). Sonomyography (ultrasound) has emerged as a promising alternative, owing to its ability to observe muscle activity in real time and control a greater number of degrees of freedom. Yet, existing ultrasound-based methods require per-user fine-tuning, limiting their commercialization. We propose SonoRank, an important step towards calibration-free finger flexion detection from forearm ultrasound
video. SonoRank first learns to rank pairs of ultrasound sequences by their relative motion magnitude for each of the five fingers. The learned representations are then fine-tuned to classify whether each finger is actively flexing, using a rest reference that is captured at the beginning of the operation. Under 12-fold leave-one-subject-out cross-validation on a
dataset of twelve subjects with synchronized kinematics, SonoRank achieves a 28% improvement in F1 score over direct classification baselines that skip the ranking stage. These results establish pairwise ranking as an effective pretraining signal for subject-independent control, bringing ultrasound-based prosthetics closer to practical, calibration-free deployment.
Read the full abstract ▾
All authors Dean Zadok, Alon Wolf, Alex M. Bronstein, Oren Salzman
Computer Vision
Infinite Worlds with Versatile Interactions
Zelin Gao, Qiuyu Wang, Jiapeng Zhu et al. (+17 more)
We present LingBot-World 2.0 (also known as LingBot-World-Infinity), an advanced iteration of LingBot-World featuring four distinct upgrades. (1) Our model achieves an unbounded interaction horizon while maintaining consistent output quality, benefiting from a carefully crafted causal pretraining paradigm. (2) Through distilling a real-time variant from the base model, our system guarantees rapid response time, sufficient to drive 720p
video streams at 60 fps. (3) Compared to the previous version, this update introduces highly diverse interactive elements, comprising a broader spectrum of actions (e.g., attacking, archery, spell-casting, and shooting) alongside a richer variety of text-driven events. (4) We pioneer the integration of an agentic harness within the domain of world modeling, wherein a pilot
agent is tasked with planning and executing character behaviors, while a director
agent is responsible for synthesizing novel environmental elements as the scene progresses. Additionally, to facilitate a shared experience, we develop an interface that permits multiple players to simultaneously immerse themselves in this vivid world simulator. We pair our primary 14B model with a lightweight 1.3B counterpart, which supports effortless deployment on a single GPU.
Read the full abstract ▾
All authors Zelin Gao, Qiuyu Wang, Jiapeng Zhu, Jingye Chen, Zichen Liu, Qingyan Bai, Jiahao Wang, Yufeng Yuan, Hanlin Wang, Yichong Lu, Ka Leong Cheng, Haojie Zhang, Jian Gao, Tianrui Feng, Yuzheng Liu, Yao Yao, Yinghao Xu, Xing Zhu, Yujun Shen, Hao Ouyang
Computer Vision
Context-Aware Slum Mapping in Sub-Saharan Africa Using Sentinel-1 Texture and Local Climate Zones
Peterson Chepkilot, Babak Memar and Paolo Gamba
Accurate mapping of informal settlements remains a major challenge in Sub-Saharan African (SSA) cities because optical imagery often fails to distinguish Informal Settlements (defined here as LCZ 7) from spectrally similar formal Compact Low-Rise areas (LCZ 3). This study presents a context-aware, reproducible Optical-SAR framework that improves informal settlement delineation using Sentinel-2 spectral features and Sentinel-1 structural information within an adapted Local
Climate Zone (LCZ) taxonomy. We implement a three-tier SAR integration strategy: calibrated backscatter, GLCM textures, and a physics-guided feature engineered to capture the high structural disorder and weak radar return characteristic of SSA informal settlements. Using reference data across Nairobi and Eldoret (Kenya), we evaluate performance via a stratified hold-out protocol and a season-aware ablation study. Results show that SAR textures provide the dominant performance gain for LCZ 7 detection. The Optical-SAR model achieves overall accuracy of 0.816 (dry) and 0.807 (wet), significantly outperforming the WUDAPT baseline (OA 0.704) and reducing the critical LCZ 3 - LCZ 7 confusion to 7%. Seasonal analysis indicates that while optical-only separability varies with phenology, SAR-derived textures stabilize informal settlement mapping across seasons. These findings demonstrate that the incorporation of SAR-derived features yields consistent improvements for urban morphology mapping in data-scarce environments across seasons and across the evaluated source cities, while cross-city transfer remains limited without local adaptation strategies.
Read the full abstract ▾
Computer Vision
Learning to Unify Deformable Shape and Texture Representations for Cardiac Video Classification
Tonmoy Hossain and Miaomiao Zhang
Deformable shape representations have proven to be robust complements to texture features in cardiac image classification, offering
geometric priors that are invariant to imaging artifacts and intensity variations. However, existing deep networks perform simple concatenation to combine these distinct feature representations, which neither fully exploits their complementary nature nor learns cross-modal feature dependencies. Furthermore, this results in uniform
attention across all timepoints; hence ignoring the varying diagnostic importance across the cardiac phases. In this paper, we propose a novel cardiac
video classification model that, for the first time, learns temporal features in an integrated space of deformable shape and image texture representations. In particular, we design a bi-directional cross-
attention in the latent space to fuse latent deformable shape and image features, allowing each modality to adaptively weight the other based on spatio-temporal correspondence. In contrast to current methods that apply uniform weighting across all the cardiac phases, our approach learns to dynamically adjust the contributions of shape and texture representations, derived from images, over time. We demonstrate state-of-the-art classification performance on a cine cardiac
magnetic resonance (CMR)
video dataset, achieving improved interpretability from
attention mechanisms that identify diagnostically critical cardiac phases and modality contributions.
Read the full abstract ▾
Computer Vision
Feng He, Zhenting Wang, Qifan Wang et al. (+4 more)
Hallucinations in
vision language models (VLMs) are commonly treated as semantic errors, yet they often arise from partial or ambiguous visual evidence. Prior work mainly focuses on detecting or suppressing
hallucinations at generation time, leaving the subsequent
reasoning stage largely unexplored. In this work, we study Post
Hallucination Reasoning (PHR), the stage in which hallucinated semantics enter the model's inference context and influence downstream predictions. To systematically investigate PHR, we introduce HIVE,
Hallucination Inference and Verification Engine, an evaluation infrastructure that enables controlled comparisons between faithful and hallucinated captions. Across nine tasks and nine models, we observe structured modality dependent patterns: hallucinated captions often improve accuracy on
vision language tasks, while text only tasks exhibit limited or unstable effects. Further analyses show that hallucinated cues broaden semantic coverage and reshape
reasoning dynamics while preserving stable inference. These findings highlight that hallucinated semantics may influence downstream
reasoning once they enter the model's inference context. Understanding this post
hallucination stage is important for improving the reliability and interpretability of
multimodal reasoning systems. Code is publicly available at https://github.com/hefengcs/HIVE.
Read the full abstract ▾
All authors Feng He, Zhenting Wang, Qifan Wang, Qiang Guan, Dongfang Liu, Ruixiang Tang, Qiankun Li
Computer Vision
Discovering Geometric Biases in 3D Face Reconstruction: A Curvature-Aware Spectral Framework for Fairness Evaluation
Veronika Shilova, Emmanuel Malherbe, Giovanni Palma et al. (+3 more)
3D Morphable Models (3DMMs) remain the standard parametric shape priors for many state-of-the-art 3D face reconstruction
algorithms. However, as these models are derived from a finite number of 3D face samples, they inherit the morphological biases of their training data, potentially limiting their generalizability across diverse global populations. In this paper, we propose a novel framework to analyze 3DMM reconstructions through the lens of surface
curvature, with the objective to discover, quantify and visualize biases. While standard evaluation metrics often rely on Euclidean distances, our reconstruction error captures subtle surface nuances such as local
topology or undulations. To do so, we leverage the Laplace-Beltrami Operator (LBO) to generate high-resolution
curvature error maps, providing a localized and geometrically meaningful visualization of discrepancies between ground truth faces and reconstructed meshes. We derive from it an error metric that we validated through a user study, observing a significantly higher correlation to human perception compared to traditional methods. Furthermore, we conduct extensive experiments across several 3DMM bases and fitting
algorithms, uncovering systematic age-related biases and providing preliminary evidence of biases associated with gender and ethnicity. Our findings highlight the necessity of adopting
curvature-aware evaluation protocols to ensure demographic
fairness and
geometric precision in future 3D face reconstruction research.
Read the full abstract ▾
All authors Veronika Shilova, Emmanuel Malherbe, Giovanni Palma, Panagiotis-Alexandros Bokaris, Laurent Risser, Jean-Michel Loubes
Computer Vision
A Theory of Contrastive Learning with Natural Images
Antonio Torralba and Yair Weiss
Why does contrastive learning with simple images and augmentations yield useful representations for downstream tasks? We address this question by analytically computing the optimal representation in terms of a contrastive loss for a range of basic augmentations and any image
dataset with stationary statistics. We show that for certain augmentations the optimum can be attained by a CNN whose first layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. We also show that the optimal weights in such CNNs for more complicated augmentations are still sinusoids. The frequencies of the sinusoids and their weights can be computed using a simple waterfilling
algorithm given the
dataset's expected power spectrum. Experiments with different image
datasets and augmentations show that such CNNs trained with SGD empirically learn sinusoids in their first layer and to perform partial whitening
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AI
Xinjie Wang, Liu Liu, Taojun Ding et al. (+9 more)
We present EmbodiedGen V2, a generative 3D world engine for building executable policy-ready environments for
embodied intelligence. Sim-ready 3D asset generation has advanced rapidly, yet assembling such assets into policy-ready task environments remains largely manual, limiting scalable closed-loop learning. EmbodiedGen V2 addresses this gap through a unified sim-ready representation that connects cross-simulator assets, interaction affordances, task-driven worlds, large-scale multi-room scenes, and stateful Vibe Coding into a generative, editable, and reusable simulation pipeline. The generated environments support manipulation,
navigation, mobile manipulation, cross-simulator deployment, and
embodied policy training. In evaluation, the asset pipeline achieves 96.5% human acceptance and 98.6% collision success, and 83.3% of task-driven worlds are directly usable for downstream simulation without manual modification. Online reinforcement learning with generated environments further improves simulation success from 9.7% to 79.8%, and transfers to real
robots with task success increasing from 21.7% to 75.0%. These results establish EmbodiedGen V2 as scalable simulation infrastructure for training, evaluating, and deploying
embodied policies.
Read the full abstract ▾
All authors Xinjie Wang, Liu Liu, Taojun Ding, Andrew Choi, Chaodong Huang, Mengao Zhao, Ziang Li, Jackson Jiang, Chunlei Yu, Shengxiang Liu, Wei Xu, Zhizhong Su
Computer Vision
Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment
Kanglei Zhou, Ruizhi Cai, Xinning Wang et al. (+4 more)
Action Quality Assessment (AQA) aims to evaluate how well a person performs a movement, which is essential in applications such as sports scoring, skill assessment, and healthcare. However, unimodal approaches often struggle to capture subtle cues of movement quality in real-world settings. Although multi-modal inputs provide complementary information, existing methods still face two major challenges: heterogeneous modalities often lead to cross-modal misalignment and unstable fusion, and reliable multi-modal annotation is costly, resulting in limited
dataset diversity. To address these challenges, we propose DualAlign, a two-stage multi-modal fusion framework with adaptive
alignment. The framework first constructs a coherent visual representation by maximizing shared structural information across RGB
video, optical flow, and skeleton modalities. Textual semantics are then incorporated after visual stabilization, allowing high-level descriptions to complement rather than distort the underlying visual
manifold. To evaluate the framework under realistic multi-modal conditions, we introduce MM–JDM, a movement-quality assessment
dataset integrating RGB
videos, optical flow, skeleton sequences, and structured text. MM–JDM naturally exhibits modality noise, class imbalance, and label scarcity, making it a challenging
benchmark for studying multi-modal fusion and
alignment. Extensive experiments show that DualAlign improves average correlation on MM–JDM by 21.16% over the state-of-the-art methods and achieves gains of 3.53% and 5.95% on the RG and Fis-V
benchmarks, respectively. DualAlign also remains robust under missing-modality and label-scarce conditions.
Read the full abstract ▾
All authors Kanglei Zhou, Ruizhi Cai, Xinning Wang, Yijian Zheng, Liyuan Wang, Jianguo Li, Xiaohui Liang
Computer Vision
VCDP: Variation-Conditioned Distributional Proxy Learning for Semi-Supervised Medical Image Segmentation
Zimu Zhang, Yiheng Zhong, Zhuoru Zhang et al. (+4 more)
Semi-supervised 3D medical image
segmentation reduces the need for dense voxel-level annotations by exploiting unlabeled volumes. Although existing methods such as consistency regularization, pseudo-labeling, and co-training improve prediction-level robustness, they often provide insufficient feature-space organization for anatomically complex structures, especially small organs and ambiguous boundary regions with large intra-class variations. To address this issue, we propose Variation-Conditioned Distributional Proxy Learning (VCDP), a plug-and-play training-only regularization module for semi-supervised 3D medical image
segmentation. VCDP represents each class with a learnable Gaussian distribution for shared class semantics and multiple variation prototypes for fine-grained intra-class patterns. A unified variation-conditioned compatibility score is further formulated to fuse distributional similarity and soft variation aggregation, guiding voxel
embeddings to align with both global organ identity and local anatomical variations. VCDP is attached to decoder features during training and removed during inference, introducing no additional inference cost. Experiments on multi-organ
segmentation benchmarks show that VCDP improves most evaluated baselines, particularly for small, ambiguous, and highly variable organs. Our anonymous code is released at https://anonymous.4open.science/r/VCDP_code-41ED.
Read the full abstract ▾
All authors Zimu Zhang, Yiheng Zhong, Zhuoru Zhang, Yingzhen Hu, Yanan He, Fanliang Meng, Xiaofeng Liu
Computer Vision
Heterogeneity-Adaptive Diffusion Schrodinger Bridge for PET-Guided Whole-Body MRI Translation
Chengbo Wang, Jiacheng Yu, Linjie Bian et al. (+7 more)
While whole-body
multimodal medical imaging scanners have been increasingly recognized for more effective medical applications, the excessive long acquisition time in PET-MR scanning is a major obstacle in more efficient clinical practice. Deep learning-based MRI translation provides a potential solution to reduce scan duration. However, current models often focus on specific anatomical regions and face challenges for whole-body scans that consists of highly heterogeneous feature distributions mainly due to (1) different anatomical regions across whole-body, and (2) lesions or pathological tissues. This paper tackles the challenges through a novel Heterogeneity-Adaptive
Diffusion Schrodinger Bridge (HA-DSB) framework. By explicitly modeling translation as
stochastic transport between source and target distributions, HA-DSB incorporates region context
embeddings derived from a
vision-
language model (VLM) to enable region-specific modeling. To enhance fidelity of the pathological tissue, lesion-aware metabolic prior from PET is integrated directly into the bridge dynamics through a dual-stage guidance mechanism. Specifically, a PET-guided noise modulation module adaptively scales spatial
diffusion perturbations during the forward process, while PET features are leveraged during the reverse process to selectively amplify lesion-relevant structures via an
attention mechanism. Experiments demonstrate the superiority of our method across different body regions in whole-body MRI translation and show improved translation quality in lesion areas under PET guidance. Our code is available at Github.
Read the full abstract ▾
All authors Chengbo Wang, Jiacheng Yu, Linjie Bian, Ming Qi, Xiaosheng Liu, Tongtong Che, Jichang Zhang, Shuyu Li, Shaoli Song, Xiuying Wang
Computer Vision
When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs
Tanay Sodha, Aditya Sharma, Ramya Hebbalaguppe et al. (+2 more)
Reliable confidence estimation remains a key limitation of test-time adaptation in
vision-
language models (VLMs), where
prompt tuning improves zero-shot accuracy but often degrades calibration due to
entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph
Attention Network (GAT) using contrastive objectives to produce structurally informed
embeddings that capture inter-attribute dependencies. We introduce two attribute selection strategies: ARGTCA-DIV for intra-class diversity and ARGTCA-DISC for inter-class discrimination. Experiments across nine
benchmarks show that ARGTCA-DIV reduces average Expected Calibration Error (ECE) by approximately ~37% over baselines, while ARGTCA-DISC consistently performs as the second-best variant, reducing average ECE by approximately ~17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.
Read the full abstract ▾
All authors Tanay Sodha, Aditya Sharma, Ramya Hebbalaguppe, Vinti Agarwal, Pranav Murthy Yeluripaty