Computer Vision
Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models
Sihan Cao, Jianwei Zhang, Pengcheng Zheng et al. (+7 more)
Large
Vision-
Language Models (LVLMs) incur substantial inference costs due to the processing of a vast number of visual
tokens. Existing methods typically struggle to model progressive visual
token reduction as a multi-step decision process with sequential dependencies and often rely on hand-engineered scoring rules that lack adaptive optimization for complex
reasoning trajectories. To overcome these limitations, we propose TPRL, a reinforcement learning framework that learns adaptive
pruning trajectories through
language-guided sequential optimization tied directly to end-task performance. We formulate visual
token pruning as a sequential decision process with explicit state transitions and employ a self-supervised autoencoder to compress visual
tokens into a compact state representation for efficient policy learning. The
pruning policy is initialized through learning from demonstrations and subsequently fine-tuned using Proximal Policy Optimization (PPO) to jointly optimize task accuracy and computational efficiency. Our experimental results demonstrate that TPRL removes up to 66.7% of visual
tokens and achieves up to a 54.2% reduction in FLOPs during inference while maintaining a near-lossless average accuracy drop of only 0.7%. Code is released at \href{https://github.com/MagicVicCoder/TPRL}{\textcolor{mypink}{https://github.com/MagicVicCoder/TPRL}}.
Read the full abstract ▾
All authors Sihan Cao, Jianwei Zhang, Pengcheng Zheng, Jiaxin Yan, Caiyan Qin, Yalan Ye, Wei Dong, Peng Wang, Yang Yang, Chaoning Zhang
Computer Vision
Colony Grounded SAM2: Zero-shot detection and segmentation of bacterial colonies using foundation models
Daan Korporaal, Patrick de Kruijf, Ralph H.G.M. Litjens et al. (+1 more)
The detection and classification of bacterial colonies in images of agar-plates is important in microbiology, but is hindered by the lack of labeled
datasets. Therefore, we propose Colony Grounded SAM2, a zero-shot inference pipeline to detect and segment bacterial colonies in multiple settings without any further training. By utilizing the pre-trained foundation models Grounding DINO and Segment Anything Model 2, fine-tuned to the microbiological domain, we developed a model that is robust to data changes. Results showed a mean Average Precision of 93.1% and a $Dice@detection$ score of 0.85, showing excellent detection and
segmentation capabilities on out-of-distribution
datasets. The entire pipeline with model weights are shared open access to aid with annotation- and classification purposes in microbiology.
Read the full abstract ▾
All authors Daan Korporaal, Patrick de Kruijf, Ralph H.G.M. Litjens, Bas H.M. van der Velden
Computer Vision · Cross-listed from eess.IV
Comparative Analysis of Deep Learning Architectures for Multi-Disease Classification of Single-Label Chest X-rays
Ali M. Bahram, Saman Muhammad Omer and Hardi M. Mohammed
Chest X-ray imaging remains the primary diagnostic tool for pulmonary and cardiac disorders worldwide, yet its accuracy is hampered by radiologist shortages and inter-observer variability. This study presents a systematic comparative evaluation of seven deep learning architectures for multi-class chest disease classification: ConvNeXt-Tiny, DenseNet121, DenseNet201, ResNet50, ViT-B/16, EfficientNetV2-M, and MobileNetV2. A balanced
dataset of 18,080 chest X-ray images spanning five disease categories (Cardiomegaly, COVID-19, Normal, Pneumonia, and Tuberculosis) was constructed from three public repositories and partitioned at the patient level to prevent data leakage. All models were trained under identical conditions using ImageNet-pretrained weights, standardized preprocessing, and consistent hyperparameters. All seven architectures exceeded 90% test accuracy. ConvNeXt-Tiny achieved the highest performance (92.31% accuracy, 95.70% AUROC), while MobileNetV2 emerged as the most parameter-efficient model (3.5M parameters, 90.42% accuracy, 94.10% AUROC), completing training in 48 minutes. Tuberculosis and COVID-19 classification was near-perfect (AUROC >= 99.97%) across all architectures, while Normal, Cardiomegaly, and Pneumonia presented greater challenges due to overlapping radiographic features. Grad-CAM visualizations confirmed clinically consistent
attention patterns across disease categories. These findings demonstrate that high-accuracy multi-disease chest X-ray classification is achievable without excessive computational resources, with important implications for AI-assisted diagnosis in both resource-rich and resource-constrained healthcare settings.
Read the full abstract ▾
Computer Vision
WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubrics
Yuhong Dai, Yanlin Lai, Mitt Huang et al. (+9 more)
Existing web-generation
benchmarks rely on text
prompts or static screenshots as input. However,
videos naturally convey richer signals such as interaction flow, transition timing, and motion continuity, which are essential for faithful webpage recreation. Despite this potential,
video-conditioned webpage generation remains largely unexplored, with no dedicated
benchmark for this task. To fill this gap, we introduce WebVR, a
benchmark that evaluates whether MLLMs can faithfully recreate webpages from demonstration
videos. WebVR contains 175 webpages across diverse categories, all constructed through a controlled synthesis pipeline rather than web crawling, ensuring varied and realistic demonstrations without overlap with existing online pages. We also design a fine-grained, human-aligned visual rubric that evaluates the generated webpages across multiple dimensions. Experiments on 19 models reveal substantial gaps in recreating fine-grained style and motion quality, while the rubric-based automatic evaluation achieves 96% agreement with human preferences. We release the
dataset, evaluation toolkit, and baseline results to support future research on
video-to-webpage generation.
Read the full abstract ▾
All authors Yuhong Dai, Yanlin Lai, Mitt Huang, Hangyu Guo, Dingming Li, Hongbo Peng, Haodong Li, Yingxiu Zhao, Haoran Lyu, Zheng Ge, Xiangyu Zhang, Daxin Jiang
Computer Vision
Logit-Conditioned Diffusion Decoding for Frozen Discrete-Token VLMs
Ji Woo Hong, Gwanhyeong Koo, Hee Suk Yoon et al. (+5 more)
Pre-trained discrete-
token VLMs generate images by sampling sequences of codebook indices, but their visual fidelity is bounded by the VQ-VAE decoding stage. Prior work addresses this bottleneck by replacing the discrete tokenizer with continuous or hybrid visual
token representations, or by integrating
diffusion into the generation pipeline; both require foundation-scale VLM retraining or realignment. We show this is unnecessary: before each
token is sampled, discrete-
token VLMs already compute a full logit distribution over the codebook, but standard decoding discards it after selecting a single index. Under Otsu's threshold, the pre-sampling logits of capable VLMs assign statistically meaningful probability to at least two codebook entries per
token, and recovering this distribution can substantially close the fidelity gap. We propose DCDD, a post-hoc framework that conditions a
diffusion decoder on this discarded distribution while leaving the VLM frozen. At inference, DCDD converts the VLM's pre-sampling logits into distribution-weighted code vectors; during training, it uses VQ-VAE encoder-derived proxy logits calibrated to match the distributional statistics of the VLM's inference-time logits. The decoder maps these representations to high-fidelity images as an alternative to the native VQ-VAE decoder. Trained on ImageNet-1K for 50K steps, DCDD reduces reconstruction FID by 74% and Janus-Pro 7B's generation FID on MJHQ-30K by 34%, while semantic
alignment across GenEval, DPG-Bench, and WISE remain consistent. Controlled comparisons with
diffusion models from the same backbone family, applied either as an image-to-image refiner or as an identically trained decoder on the sampled codes, confirm that the gain stems from conditioning on the VLM's pre-sampling logit distribution rather than from the
diffusion model itself.
Read the full abstract ▾
All authors Ji Woo Hong, Gwanhyeong Koo, Hee Suk Yoon, Eunseop Yoon, SooHwan Eom, Qi Dai, Chong Luo, Chang D. Yoo
Computer Vision
VeloEdit: Training-Free Consistent and Continuous Instruction-Based Image Editing via Velocity Field Decomposition
Zongqing Li, Zhihui Liu, Yujie Xie et al. (+3 more)
Instruction-based image editing aims to modify source content according to textual instructions. However, existing methods built upon flow matching often struggle to maintain consistency in non-edited regions due to denoising-induced reconstruction errors that cause drift in preserved content. Moreover, they typically lack fine-grained control over edit strength. To address these limitations, we propose VeloEdit, a training-free method that enables highly consistent and continuously controllable editing. VeloEdit dynamically identifies editing regions by quantifying the discrepancy between the velocity fields responsible for preserving source content and those driving the desired edits. Based on this partition, we enforce consistency in preservation regions by substituting the editing velocity with the source-restoring velocity, while enabling continuous modulation of edit intensity in target regions via velocity interpolation. Unlike prior works that rely on complex
attention manipulation or auxiliary trainable modules, VeloEdit operates directly on the velocity fields. Extensive experiments on Flux.1 Kontext and Qwen-Image-Edit demonstrate that VeloEdit improves visual consistency and editing continuity with negligible additional computational cost. Code is available at https://github.com/xmulzq/VeloEdit.
Read the full abstract ▾
All authors Zongqing Li, Zhihui Liu, Yujie Xie, Shansiyuan Wu, Hongshen Lv, Songzhi Su
Computer Vision
Cylindrical Mechanical Projector for Omnidirectional Fringe Projection Profilometry
Mincheol Choi, Gaeun Kim and Jae-Sang Hyun
The demand for 360-degree 3D reconstruction has significantly increased in recent years across various domains such as the metaverse and 3D telecommunication. Accordingly, the importance of precise and wide-area 3D sensing technology has become emphasized. While the digital fringe projection method has been widely used due to its high accuracy and implementation flexibility, it suffers from fundamental limitations such as unidirectional projection and a restricted available light spectrum. To address these issues, this paper proposes a novel 3D reconstruction method based on a cylindrical mechanical projector. The proposed method consists of a rotational stage and a cylindrical pattern generator with ON/OFF slots at two distinct intervals, enabling omnidirectional projection of multi-frequency phase-shifted fringe patterns. By applying a multi-wavelength unwrapping
algorithm and a quasi-calibration technique, the system achieves high-accuracy 3D reconstruction using only a single camera. Experimental results, supported by repeatability and reproducibility analyses together with a measurement uncertainty evaluation, confirm reliable measurement performance and practical feasibility for omnidirectional 3D reconstruction. The expanded uncertainty of the reconstructed depth was evaluated as 0.215 mm.
Read the full abstract ▾
Computer Vision
Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers
Yuntao Shou, Xiangyong Cao, Qian Zhao et al. (+1 more)
Controllable pathology image synthesis requires reliable regulation of spatial layout, tissue morphology, and semantic detail. However, existing text-guided
diffusion models offer only coarse global control and lack the ability to enforce fine-grained structural constraints. Progress is further limited by the absence of large
datasets that pair patch-level spatial layouts with detailed diagnostic descriptions, since generating such annotations for gigapixel whole-slide images is prohibitively time-consuming for human experts. To overcome these challenges, we first develop a scalable multi-
agent LVLM annotation framework that integrates image description, diagnostic step extraction, and automatic quality judgment into a coordinated pipeline, and we evaluate the reliability of the system through a human verification process. This framework enables efficient construction of fine-grained and clinically aligned supervision at scale. Building on the curated data, we propose In-Context
Diffusion Transformer (IC-DiT), a layout-aware generative model that incorporates spatial layouts, textual descriptions, and visual
embeddings into a unified
diffusion transformer. Through hierarchical
multimodal attention, IC-DiT maintains global semantic coherence while accurately preserving structural and morphological details. Extensive experiments on five histopathology
datasets show that IC-DiT achieves higher fidelity, stronger spatial controllability, and better diagnostic consistency than existing methods. In addition, the generated images serve as effective data augmentation resources for downstream tasks such as cancer classification and survival analysis.
Read the full abstract ▾
All authors Yuntao Shou, Xiangyong Cao, Qian Zhao, Deyu Meng
Computer Vision
VisualLeakBench: Auditing the Fragility of Large Vision-Language Models against PII Leakage and Social Engineering
Youting Wang, Yuan Tang, Yitian Qian et al. (+1 more)
As Large
Vision-
Language Models (LVLMs) are increasingly deployed in
agent-integrated workflows and other deployment-relevant settings, their robustness against semantic visual attacks remains under-evaluated –
alignment is typically tested on explicit harmful content rather than
privacy-critical
multimodal scenarios. We introduce VisualLeakBench, an evaluation suite to audit LVLMs against OCR Injection and Contextual PII Leakage using 1,000 synthetically generated
adversarial images with 8 PII types, validated on 50 in-the-wild (IRL) real-world screenshots spanning diverse visual contexts. We evaluate four frontier systems (GPT-5.2, Claude~4, Gemini-3 Flash, Grok-4) with Wilson 95% confidence intervals. Claude~4 achieves the lowest OCR ASR (14.2%) but the highest PII ASR (74.4%), exhibiting a comply-then-warn pattern – where verbatim data disclosure precedes any safety-oriented
language. Grok-4 achieves the lowest PII ASR (20.4%). A defensive system
prompt eliminates PII leakage for two models, reduces Claude~4's leakage from 74.4% to 2.2%, but has no effect on Gemini-3 Flash on synthetic data. Strikingly, IRL validation reveals Gemini-3 Flash does respond to mitigation on real-world images (50% to 0%), indicating that mitigation robustness is template-sensitive rather than uniformly absent. We release our
dataset and code for reproducible robustness and safety evaluation of deployment-relevant
vision-
language systems.
Read the full abstract ▾
All authors Youting Wang, Yuan Tang, Yitian Qian, Chen Zhao
Computer Vision
Taming Vision Priors for Data Efficient mmWave Channel Modeling
Zhenlin An, Longfei Shangguan, John Kaewell et al. (+5 more)
Accurately modeling millimeter-wave (mmWave) propagation is essential for real-time AR and autonomous systems. Differentiable ray tracing offers a physics-grounded solution but still facing deployment challenges due to its over-reliance on exhaustive channel measurements or brittle, hand-tuned scene models for material properties. We present VisRFTwin, a scalable and data-efficient digital-twin framework that integrates
vision-derived material priors with differentiable ray tracing. Multi-view images from commodity cameras are processed by a frozen
Vision-
Language Model to extract dense semantic
embeddings, which are translated into initial estimates of permittivity and conductivity for scene surfaces. These priors initialize a Sionna-based differentiable ray tracer, which rapidly calibrates material parameters via gradient descent with only a few dozen sparse channel soundings. Once calibrated, the association between
vision features and material parameters is retained, enabling fast transfer to new scenarios without repeated calibration. Evaluations across three real-world scenarios, including office interiors, urban canyons, and dynamic public spaces show that VisRFTwin reduces channel measurement needs by up to 10$\times$ while achieving a 59% lower median delay spread error than pure data-driven deep learning methods.
Read the full abstract ▾
All authors Zhenlin An, Longfei Shangguan, John Kaewell, Philip Pietraski, Jelena Senic, Camillo Gentile, Nada Golmie, Kyle Jamieson
Computer Vision
DINOv3 with Test-Time Calibration for Automated Carotid Intima-Media Thickness Measurement on CUBS v1
Zhenpeng Zhang, Jinwei Lu, Yurui Dong et al. (+1 more)
Carotid intima-media thickness (CIMT) measured from B-mode ultrasound is an established vascular biomarker for atherosclerosis and cardiovascular risk stratification. Although a wide range of computerized methods have been proposed for carotid boundary delineation and CIMT estimation, robust and transferable deep models that jointly address
segmentation and measurement remain underexplored, particularly in the era of
vision foundation models. Motivated by recent advances in adapting DINOv3 to medical
segmentation and exploiting DINOv3 in test-time optimization pipelines, we investigate a DINOv3-based framework for carotid intima-media complex
segmentation and subsequent CIMT measurement on the Carotid Ultrasound Boundary Study (CUBS) v1
dataset. Our pipeline predicts the intima-media band at a fixed image resolution, extracts upper and lower boundaries column-wise, corrects for image resizing using the per-image calibration factor provided by CUBS, and reports CIMT in physical units. Across three patient-level test splits, our method achieved a mean test Dice of 0.7739 $\pm$ 0.0037 and IoU of 0.6384 $\pm$ 0.0044. The mean CIMT absolute error was 181.16 $\pm$ 11.57 $\mu$m, with a mean Pearson correlation of 0.480 $\pm$ 0.259. In a held-out validation subset ($n=28$), test-time threshold calibration reduced the mean absolute CIMT error from 141.0 $\mu$m at the default threshold to 101.1 $\mu$m at the measurement-optimized threshold, while simultaneously reducing systematic bias toward zero. Relative to the error ranges reported in the original CUBS
benchmark for classical computerized methods, these results place a DINOv3-based approach within the clinically relevant $\sim$0.1 mm measurement regime. Together, our findings support the feasibility of using
vision foundation models for interpretable, calibration-aware CIMT measurement.
Read the full abstract ▾
All authors Zhenpeng Zhang, Jinwei Lu, Yurui Dong, Bo Yuan
Computer Vision
DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding
Mingzhe Tao, Ruiping Liu, Junwei Zheng et al. (+6 more)
Fusing sensors with complementary modalities is crucial for maintaining a stable and comprehensive understanding of abnormal driving scenes. However,
Multimodal Large
Language Models (MLLMs) are underexplored for leveraging multi-sensor information to understand adverse driving scenarios in autonomous vehicles. To address this gap, we propose the DriveXQA, a
multimodal dataset for autonomous driving VQA. In addition to four visual modalities, five sensor failure cases, and five weather conditions, it includes $102,505$ QA pairs categorized into three types: global scene level, allocentric level, and ego-vehicle centric level. Since no existing MLLM framework adopts multiple complementary visual modalities as input, we design MVX-LLM, a
token-efficient architecture with a Dual Cross-
Attention (DCA) projector that fuses the modalities to alleviate information redundancy. Experiments demonstrate that our DCA achieves improved performance under challenging conditions such as foggy (GPTScore: $53.5$ vs. $25.1$ for the baseline).
Read the full abstract ▾
All authors Mingzhe Tao, Ruiping Liu, Junwei Zheng, Yufan Chen, Kedi Ying, M. Saquib Sarfraz, Kailun Yang, Jiaming Zhang, Rainer Stiefelhagen
Computer Vision
Learning to Assist: Physics-Grounded Human-Human Control via Multi-Agent Reinforcement Learning
Yuto Shibata, Kashu Yamazaki, Lalit Jayanti et al. (+3 more)
Humanoid robotics has strong potential to transform daily service and caregiving applications. Although recent advances in general motion tracking within physics engines (GMT) have enabled virtual characters and
humanoid robots to reproduce a broad range of human motions, these behaviors are primarily limited to contact-less social interactions or isolated movements. Assistive scenarios, by contrast, require continuous awareness of a human partner and rapid adaptation to their evolving posture and dynamics. In this paper, we formulate the imitation of closely interacting, force-exchanging human-human motion sequences as a multi-
agent reinforcement learning problem. We jointly train partner-aware policies for both the supporter (assistant)
agent and the recipient
agent in a physics simulator to track assistive motion references. To make this problem tractable, we introduce a partner policies initialization scheme that transfers priors from single-human motion-tracking controllers, greatly improving exploration. We further propose dynamic reference retargeting and contact-promoting reward, which adapt the assistant's reference motion to the recipient's real-time pose and encourage physically meaningful support. We show that AssistMimic is the first method capable of successfully tracking assistive interaction motions on established
benchmarks, demonstrating the benefits of a multi-
agent RL formulation for physically grounded and socially aware
humanoid control.
Read the full abstract ▾
All authors Yuto Shibata, Kashu Yamazaki, Lalit Jayanti, Yoshimitsu Aoki, Mariko Isogawa, Katerina Fragkiadaki
Computer Vision
ReDiff: Reliability-Guided Diffusion for Trustworthy Ultra-Low-Field to High-Field MRI Synthesis
Zhenxuan Zhang, Peiyuan Jing, Ruicheng Yuan et al. (+9 more)
Low-field to high-field MRI synthesis has emerged as a promising strategy to improve image quality when access to high-field scanners is limited. However, in ultra-low-field settings, the degradation of anatomical detail is spatially heterogeneous: structurally ambiguous regions are more susceptible to unstable high-frequency generation, which may produce anatomically inconsistent textures and boundaries. This issue is particularly problematic when synthesized images are used for downstream quantitative analysis. We therefore study how to make
diffusion-based LF-to-HF synthesis more spatially reliable, rather than only sharper on average. To this end, we propose a reliability-guided
diffusion framework (ReDiff) with two complementary inference-time mechanisms. First, a reliability-guided sampling strategy attenuates unstable reverse-
diffusion updates in regions with weak low-field support. Second, an uncertainty-aware candidate selection scheme aggregates multiple
stochastic reconstructions according to spatial consensus and predictive uncertainty. Beyond aggregate image quality, we test whether the uncertainty is itself a usable reliability signal. Experiments on paired 64mT$\rightarrow$3T MRI
datasets show that ReDiff attains the lowest LPIPS across three contrasts and two
datasets while remaining competitive on PSNR and SSIM, and downstream
segmentation analysis indicates better preservation of anatomical structure.
Read the full abstract ▾
All authors Zhenxuan Zhang, Peiyuan Jing, Ruicheng Yuan, Liwei Hu, Anbang Wang, Fanwen Wang, Yinzhe Wu, Kh Tohidul Islam, Zhaolin Chen, Zi Wang, Peter Lally, Guang Yang
Computer Vision
UNet-AF: Alias-free UNet architectures
Jérémy Scanvic, Quentin Barthélemy and Julián Tachella
The simplicity and effectiveness of UNet architectures make them ubiquitous in image restoration,
segmentation, and
diffusion models. They are often assumed to be equivariant to translations, yet they traditionally consist of layers that are known to be prone to aliasing, which hinders their equivariance in practice. To overcome this limitation, we show how to build sub-pixel translation-equivariant UNet architectures by appropriately choosing their main components (convolution, pooling, downsampling, activation, and normalization layers) to be alias-free. We evaluate the proposed equivariant architectures against non-equivariant baselines on image restoration tasks and observe competitive performance with a significant increase in measured equivariance. Through extensive ablation studies, we also demonstrate the importance of every architectural choice to achieve high equivariance. Our implementation is available at https://github.com/jscanvic/UNet-AF
Read the full abstract ▾
Computer Vision
UniCompress: Token Compression for Unified Vision-Language Understanding and Generation
Ziyao Wang, Chen Chen, Jingtao Li et al. (+4 more)
Unified models aim to support both understanding and generation by encoding images into discrete
tokens and processing them alongside text within a single autoregressive framework. This unified design offers architectural simplicity and cross-modal synergy, which facilitates shared parameterization, consistent training objectives, and seamless transfer between modalities. However, the large number of visual
tokens required by such models introduces substantial computation and
memory overhead, and this inefficiency directly hinders deployment in resource constrained scenarios such as
embodied AI systems. In this work, we propose a unified
token compression
algorithm UniCompress that significantly reduces visual
token count while preserving performance on both image understanding and generation tasks. Our method introduces a plug-in compression and decompression mechanism guided with learnable global meta
tokens. The framework is lightweight and modular, enabling efficient integration into existing models without full retraining. Experimental results show that our approach reduces image
tokens by up to 4 times, achieves substantial gains in inference latency and training cost, and incurs only minimal performance degradation, which demonstrates the promise of
token-efficient unified modeling for real world
multimodal applications.
Read the full abstract ▾
All authors Ziyao Wang, Chen Chen, Jingtao Li, Weiming Zhuang, Jiabo Huang, Ang Li, Lingjuan Lyu
Computer Vision · Cross-listed from physics.med-ph
MRI2Qmap: multi-parametric quantitative mapping with MRI-driven denoising priors
Mohammad Golbabaee, Matteo Cencini, Carolin Pirkl et al. (+3 more)
Magnetic Resonance Fingerprinting (MRF) and other highly accelerated transient-state parameter mapping techniques enable simultaneous quantification of multiple tissue properties, but often suffer from aliasing artifacts due to compressed sampling. Incorporating spatial image priors can mitigate these artifacts, and deep learning has shown strong potential when large training
datasets are available. However, extending this paradigm to MRF-type sequences remains challenging due to the scarcity of quantitative imaging data for training. Can this limitation be overcome by leveraging sources of training data from clinically-routine weighted MRI images? To this end, we introduce MRI2Qmap, a plug-and-play quantitative reconstruction framework that integrates the physical acquisition model with priors learned from deep denoising autoencoders pretrained on large
multimodal weighted-MRI
datasets. MRI2Qmap demonstrates that spatial-domain structural priors learned from independently acquired
datasets of routine weighted-MRI images can be effectively used for quantitative MRI reconstruction. The proposed method is validated on highly accelerated 3D whole-
brain MRF data from both in-vivo and simulated acquisitions, achieving competitive or superior performance relative to existing baselines without requiring ground-truth quantitative imaging data for training. By decoupling quantitative reconstruction from the need for ground-truth MRF training data, this framework points toward a scalable paradigm for quantitative MRI that can capitalize on the large and growing repositories of routine clinical MRI.
Read the full abstract ▾
All authors Mohammad Golbabaee, Matteo Cencini, Carolin Pirkl, Marion Menzel, Michela Tosetti, Bjoern Menze
Computer Vision
Hierarchical Granularity Alignment and State Space Modeling for Robust Multimodal AU Detection in the Wild
Jun Yu, Yunxiang Zhang, Naixiang Zheng et al. (+2 more)
Facial Action Unit (AU) detection in in-the-wild environments remains a formidable challenge due to severe spatial-temporal heterogeneity, unconstrained poses, and complex audio-visual dependencies. While recent
multimodal approaches have made progress, they often rely on capacity-limited encoders and shallow fusion mechanisms that fail to capture fine-grained semantic shifts and ultra-long temporal contexts. To bridge this gap, we propose a novel
multimodal framework driven by Hierarchical Granularity
Alignment and State Space Models.Specifically, we leverage powerful foundation models, namely DINOv2 and WavLM, to extract robust and high-fidelity visual and audio representations, effectively replacing traditional feature extractors. To handle extreme facial variations, our Hierarchical Granularity
Alignment module dynamically aligns global facial semantics with fine-grained local active patches. Furthermore, we overcome the receptive field limitations of conventional temporal convolutional networks by introducing a
Vision-Mamba architecture. This approach enables temporal modeling with O(N) linear complexity, effectively capturing ultra-long-range dynamics without performance degradation. A novel asymmetric cross-
attention mechanism is also introduced to deeply synchronize paralinguistic audio cues with subtle visual movements.Extensive experiments on the challenging Aff-Wild2
dataset demonstrate that our approach significantly outperforms existing baselines, achieving state-of-the-art performance. Notably, this framework secured top rankings in the AU Detection track of the 10th Affective Behavior Analysis in-the-wild Competition.
Read the full abstract ▾
All authors Jun Yu, Yunxiang Zhang, Naixiang Zheng, Lingsi Zhu, Guoyuan Wang
Computer Vision
InstantHDR: Single-forward Gaussian Splatting Initialization for HDR 3D Reconstruction
Dingqiang Ye, Jiacong Xu, Jianglu Ping et al. (+3 more)
High dynamic range (HDR) novel view synthesis (NVS) aims to reconstruct HDR scenes from multi-exposure low dynamic range (LDR) images. Existing HDR pipelines heavily rely on known camera poses, well-initialized dense point clouds, and time-consuming per-scene optimization. Current feed-forward alternatives overlook the HDR problem by assuming exposure-invariant appearance. To bridge this gap, we propose InstantHDR, a feed-forward network that initializes 3D HDR scenes from uncalibrated multi-exposure LDR collections in a fast single forward pass. Specifically, we design a geometry-guided appearance modeling for multi-exposure fusion, and a meta-network for generalizable scene-specific tone mapping. Due to the lack of HDR scene data, we build a pre-training
dataset, called HDR-Pretrain, for generalizable feed-forward HDR models, featuring 168 Blender-rendered scenes, diverse lighting types, and multiple camera response functions. Comprehensive experiments show that our InstantHDR delivers a single-forward HDR initialization at approximately 700 times the speed of SoTA optimization-based methods, and reaches comparable quality in real settings after lightweight post-optimization while remaining approximately 20 times faster. All code, models, and
datasets: https://github.com/Bugjudger/InstantHDR.
Read the full abstract ▾
All authors Dingqiang Ye, Jiacong Xu, Jianglu Ping, Yuxiang Guo, Chao Fan, Vishal M. Patel
Computer Vision
Towards Automated Initial Probe Placement in Transthoracic Teleultrasound Using Human Mesh and Skeleton Recovery
Yu Chung Lee, David G. Black, Ryan S. Yeung et al. (+1 more)
Cardiac and lung ultrasound are technically demanding because operators must identify patient-specific intercostal acoustic windows and then navigate between standard views by adjusting probe position, rotation, and force across different imaging planes. These challenges are amplified in teleultrasound, where the examination proceeds without in-person expert assistance: once the probe is approximately positioned, the expert can navigate in ultrasound image space, but guiding the initial placement remotely remains difficult given the limited 3D perception of the patient. We present a framework for automating patient registration and anatomy-informed initial probe placement guidance (PIPG) using RGB images obtained from a calibrated camera and a point cloud accumulated from depth images. The novice first captures the patient using the camera on a mixed reality (MR) head-mounted display (HMD), and an edge server then infers a patient-specific body-surface and skeleton model. By leveraging the patient's spatial and temporal consistency across multiview and point cloud data, we achieved robust, training-free human registration, verified in a healthcare setting. Using bony landmarks from the predicted skeleton, we estimate the intercostal region and project the guidance back onto the reconstructed body surface. To validate the framework, we rendered the reconstructed body mesh and the virtual probe pose guidance in the MR headset across multiple transthoracic echocardiography scan planes in situ and measured the quantitative placement error. Pilot experiments with five healthy volunteers suggest that the proposed probe placement prediction and MR guidance yield consistent initial placement, with a mean surface error of 15 mm, positional error to palpated anatomical landmarks of 41.0 mm, and torso orientation errors within 9 deg, acceptable for the teleultrasound setup.
Read the full abstract ▾
All authors Yu Chung Lee, David G. Black, Ryan S. Yeung, Septimiu E. Salcudean
Computer Vision
Radiometric fingerprinting of object surfaces using mobile laser scanning and semantic 3D road space models
Benedikt Schwab and Thomas H. Kolbe
Although semantic 3D city models are internationally available and becoming increasingly detailed, the incorporation of material information remains largely untapped. However, a structured representation of materials and their physical properties could substantially broaden the application spectrum and analytical capabilities for urban digital twins. At the same time, the growing number of repeated mobile
laser scans of cities and their street spaces yields a wealth of observations influenced by the material characteristics of the corresponding surfaces. To leverage this information, we propose radiometric fingerprints of object surfaces by grouping
LiDAR observations reflected from the same semantic object under varying distances, incident angles, environmental conditions, sensors, and scanning campaigns. Our study demonstrates how 312.4 million individual beams acquired across four campaigns using five
LiDAR sensors on the Audi Autonomous Driving
Dataset (A2D2) vehicle can be automatically associated with 6368 individual objects of the semantic 3D city model. The model comprises a comprehensive and semantic representation of four inner-city streets at Level of Detail (LOD) 3 with centimeter-level accuracy. It is based on the CityGML 3.0 standard and enables fine-grained sub-differentiation of objects. The extracted radiometric fingerprints for object surfaces reveal recurring intra-class patterns that indicate class-dominant materials. The semantic model, the method implementations, and the developed geodatabase solution 3DSensorDB are released under: https://github.com/tum-gis/sensordb
Read the full abstract ▾
Computer Vision
When Slots Compete: Slot Merging in Object-Centric Learning
Christos Chatzisavvas, Panagiotis Rigas, George Ioannakis et al. (+2 more)
Slot-based object-centric learning represents an image as a set of latent slots with a decoder that combines them into an image or features. The decoder specifies how slots are combined into an output, but the slot set is typically fixed: the number of slots is chosen upfront and slots are only refined. This can lead to multiple slots competing for overlapping regions of the same entity rather than focusing on distinct regions. We introduce slot merging: a drop-in, lightweight operation on the slot set that merges overlapping slots during training. We quantify overlap with a Soft-IoU score between slot-
attention maps and combine selected pairs via a barycentric update that preserves gradient flow. Merging follows a fixed policy, with the decision threshold inferred from overlap statistics, requiring no additional learnable modules. Integrated into the established feature-reconstruction pipeline of DINOSAUR, the proposed method improves object factorization and mask quality, surpassing other adaptive methods in object discovery and
segmentation benchmarks.
Read the full abstract ▾
All authors Christos Chatzisavvas, Panagiotis Rigas, George Ioannakis, Vassilis Katsouros, Nikolaos Mitianoudis
Computer Vision
Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Models
Qingtao Pan, Zhihao Dou and Shuo Li
Large
Multimodal Models (LMMs) struggle to adapt varying computational budgets due to numerous visual
tokens. Previous methods attempted to reduce the number of visual
tokens before or within LLMs. However, these strategies inevitably result in the loss of visual semantic. To address these issues, we introduce FMVR, a plug-and-play and extremely simple Frequency-Modulated Visual Restoration strategy to boost the
reasoning ability of LMMs under visual
token reduction. Specifically, FMVR disentangles the visual representation of fewer visual
tokens into low- and high-frequency components through AvgPool and MaxPool. The derived frequencies are subsequently modulated using lightweight learnable parameters. The high-frequency from AvgPool acts as a saliency filter to enhance saliency visual semantics, while the low-frequency from MaxPool acts as an anti-saliency filter to strengthen weak visual semantics. It enables the preservation of visual semantics dominated by few visual
tokens and the restoration of diluted visual semantics. Additionally, we inject FMVR into Matryoshka Representation Learning to learn coarse-to-fine visual
token sets, thus enabling to elastically adjust the number of visual
tokens during inference while maintaining comparable performance. Experiments across 10 image-based and 4
video-based bench marks demonstrate that FMVR-LLaVA reduce the FLOPs of LLaVA-1.5-7B by 89%, while maintaining almost 100% of the original accuracy. The code will be open.
Read the full abstract ▾
Computer Vision
Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planning
Yuehao Song, Shaoyu Chen, Hao Gao et al. (+8 more)
Vision-
language models (VLMs) enhance the planning capability of end-to-end (E2E) driving policy by leveraging high-level semantic
reasoning. However, existing approaches often overlook the dual-system consistency between VLM's high-level decision and E2E's low-level planning. As a result, the generated trajectories may misalign with the intended driving decisions, leading to weakened top-down guidance and decision-following ability of the system. To address this issue, we propose Senna-2, an advanced VLM-E2E driving policy that explicitly aligns the two systems for consistent decision-making and planning. Our method follows a consistency-oriented three-stage training paradigm. In the first stage, we conduct driving pre-training to achieve preliminary decision-making and planning, with a decision adapter transmitting VLM decisions to E2E policy in the form of implicit
embeddings. In the second stage, we align the VLM and the E2E policy in an open-loop setting. In the third stage, we perform closed-loop
alignment via bottom-up Hierarchical Reinforcement Learning in 3DGS environments to reinforce the safety and efficiency. Extensive experiments demonstrate that Senna-2 achieves superior dual-system consistency (19.3% F1 score improvement) and significantly enhances driving safety in both open-loop (5.7% FDE reduction) and closed-loop settings (30.6% AF-CR reduction).
Read the full abstract ▾
All authors Yuehao Song, Shaoyu Chen, Hao Gao, Yifan Zhu, Weixiang Yue, Jialv Zou, Bo Jiang, Zihao Lu, Yu Wang, Qian Zhang, Xinggang Wang
Computer Vision
A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters
Haihua Luo, Xuming Ran, Jiangrong Shen et al. (+4 more)
Incremental Learning (IL) aims to learn new tasks while preserving previously acquired knowledge. Integrating the zero-shot learning capabilities of pre-trained
vision-
language models into IL methods has marked a significant advancement. However, these methods face three primary challenges: (1) the need for improved training efficiency; (2) reliance on a
memory bank to store previous data; and (3) the necessity of a strong backbone to augment the model's capabilities. In this paper, we propose SimE, a Simple and Efficient framework that employs a
vision-
language model with adapters designed specifically for the IL task. We report a remarkable phenomenon: there is a nonlinear correlation between the number of adaptive adapter connections and the model's IL capabilities. While increasing adapter connections between
transformer blocks improves model performance, adding more adaptive connections within
transformer blocks during smaller incremental steps does not enhance, and may even degrade the model's IL ability. Extensive experimental results show that SimE surpasses traditional methods by 9.6% on TinyImageNet and outperforms other CLIP-based methods by 5.3% on CIFAR-100. Furthermore, we conduct a systematic study to enhance the utilization of the zero-shot capabilities of CLIP. We suggest replacing SimE's encoder with a CLIP model trained on larger
datasets (e.g., LAION2B) and stronger architectures (e.g., ViT-L/14).
Read the full abstract ▾
All authors Haihua Luo, Xuming Ran, Jiangrong Shen, Timo Hämäläinen, Zhonghua Chen, Qi Xu, Fengyu Cong
Computer Vision
Evidential learning driven Breast Tumor Segmentation with Stage-divided Vision-Language Interaction
Jingxing Zhong, Qingtao Pan, Xuchang Zhou et al. (+2 more)
Breast cancer is one of the most common causes of death among women worldwide, with millions of fatalities annually.
Magnetic Resonance Imaging (MRI) can provide various sequences for characterizing tumor morphology and internal patterns, and becomes an effective tool for detection and diagnosis of breast tumors. However, previous deep-learning based tumor
segmentation methods have limitations in accurately locating tumor contours due to the challenge of low contrast between cancer and normal areas and blurred boundaries. Leveraging text
prompt information holds promise in ameliorating tumor
segmentation effect by delineating
segmentation regions. Inspired by this, we propose text-guided Breast Tumor
Segmentation model (TextBCS) with stage-divided
vision-
language interaction and evidential learning. Specifically, the proposed stage-divided
vision-
language interaction facilitates information mutual between visual and text features at each stage of down-sampling, further exerting the advantages of text
prompts to assist in locating lesion areas in low contrast scenarios. Moreover, the evidential learning is adopted to quantify the
segmentation uncertainty of the model for blurred boundary. It utilizes the variational Dirichlet to characterize the distribution of the
segmentation probabilities, addressing the
segmentation uncertainties of the boundaries. Extensive experiments validate the superiority of our TextBCS over other
segmentation networks, showcasing the best breast tumor
segmentation performance on publicly available
datasets.
Read the full abstract ▾
All authors Jingxing Zhong, Qingtao Pan, Xuchang Zhou, Jiazhen Lin, Xinguo Zhuang
Computer Vision
GGPT: Geometry Grounded Point Transformer
Yutong Chen, Yiming Wang, Xucong Zhang et al. (+2 more)
Recent feed-forward networks have achieved remarkable progress in sparse-view 3D reconstruction by predicting dense point maps directly from RGB images. However, they often suffer from
geometric inconsistencies and limited fine-grained accuracy due to the absence of explicit multi-view constraints. We introduce the Geometry-Grounded Point
Transformer (GGPT), a framework that augments feed-forward reconstruction with reliable sparse
geometric guidance. We first propose an improved Structure-from-Motion pipeline based on dense feature matching and lightweight
geometric optimisation to efficiently estimate accurate camera poses and partial 3D point clouds from sparse input views. Building on this foundation, we propose a geometry-guided 3D point
transformer that refines dense point maps under explicit partial-geometry supervision using an optimised guidance encoding. Extensive experiments demonstrate that our method provides a principled mechanism for integrating
geometric priors with dense feed-forward predictions, producing reconstructions that are both geometrically consistent and spatially complete, recovering fine structures and filling gaps in textureless areas. Trained solely on ScanNet++ with VGGT predictions, GGPT generalises across architectures and
datasets, substantially outperforming state-of-the-art feed-forward 3D reconstruction models in both in-domain and out-of-domain settings.
Read the full abstract ▾
All authors Yutong Chen, Yiming Wang, Xucong Zhang, Sergey Prokudin, Siyu Tang
Computer Vision · Cross-listed from cs.MM
Catalogue-Grounded Multimodal Attribution for Museum Video under Resource and Regulatory Constraints
Minsak Nanang, Adrian Hilton and Armin Mustafa
Museum audiovisual archives are important records of visual culture, yet in-gallery
video remains difficult to search because it lacks catalogue-compatible metadata. We propose CaGMulA, catalogue-grounded
multimodal attribution for museum
video: a locally deployable pipeline that combines
video-
language models, structured catalogue
retrieval, and abstention-aware decision policies. The system summarises visible artworks, generates catalogue-style descriptions, and attributes title and artist only when evidence is sufficient. We evaluate VideoLLaMA2 and Qwen2-VL across four model configurations and four attribution modes on 18 in-gallery
videos. Open-ended generation fails under conservative conditions, producing 0% coverage across models. By contrast, closed-set catalogue
retrieval reaches 100% coverage with 62% accepted title–artist correctness, while a verifier-gated hybrid mode reduces coverage to 50% but improves accepted correctness to 78%. The results show that reliable museum-
video attribution is driven less by unrestricted generation than by catalogue grounding, visual
retrieval, and selective abstention for curatorial review.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Attention Gathers, MLPs Compose: A Causal Analysis of an Action-Outcome Circuit in VideoViT
Sai V R Chereddy
The paper explores how
video models trained for classification tasks represent nuanced, hidden semantic information that may not affect the final outcome, a key challenge for Trustworthy AI models. Through Explainable and Interpretable AI methods, specifically mechanistic interpretability techniques, the internal
circuit responsible for representing the action's outcome is reverse-engineered in a pre-trained
video vision transformer, revealing that the "Success vs Failure" signal is computed through a distinct amplification cascade. While there are low-level differences observed from layer 0, the abstract and semantic representation of the outcome is progressively amplified from layers 5 through 11. Causal analysis, primarily using activation patching supported by ablation results, reveals a clear division of labor:
Attention Heads act as "evidence gatherers", providing necessary low-level information for partial signal recovery, while MLP Blocks function as robust "concept composers", each of which is the primary driver to generate the "success" signal. This distributed and redundant
circuit in the model's internals explains its resilience to simple ablations, demonstrating a core computational pattern for processing human-action outcomes. Crucially, the existence of this sophisticated
circuit for representing complex outcomes, even within a model trained only for simple classification, highlights the potential for models to develop forms of 'hidden knowledge' beyond their explicit task, underscoring the need for mechanistic oversight for building genuinely Explainable and Trustworthy AI systems intended for deployment.
Read the full abstract ▾
Computer Vision
RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation
Shijie Zhou, Bin Zhu, Jiarui Yang et al. (+3 more)
Recent advances in
Vision-
Language-Action (VLA) models have enabled
robots to execute increasingly complex tasks. However, VLA models trained through imitation learning struggle to operate reliably in dynamic environments and often fail under Out-of-Distribution (OOD) conditions. To address this issue, we propose
Robot-Conditioned Normalizing Flow (RC-NF), a real-time monitoring model for robotic anomaly detection and intervention that ensures the
robot's state and the object's motion trajectory align with the task. RC-NF decouples the processing of task-aware
robot and object states within the normalizing flow. It requires only positive samples for unsupervised training and calculates accurate robotic anomaly scores during inference through the probability density function. We further present LIBERO-Anomaly-10, a
benchmark comprising three categories of robotic anomalies for simulation evaluation. RC-NF achieves state-of-the-art performance across all anomaly types compared to previous methods in monitoring robotic tasks. Real-world experiments demonstrate that RC-NF operates as a plug-and-play module for VLA models (e.g., pi0), providing a real-time OOD signal that enables state-level rollback or task-level replanning when necessary, with a response latency under 100 ms. These results demonstrate that RC-NF noticeably enhances the robustness and adaptability of VLA-based robotic systems in dynamic environments.
Read the full abstract ▾
All authors Shijie Zhou, Bin Zhu, Jiarui Yang, Xiangyu Zhao, Jingjing Chen, Yu-Gang Jiang