Computer Vision · Cross-listed from eess.IV
Mammography Foundation Models for Opportunistic Prediction of Major Adverse Cardiovascular Events
Paula Feldman, Nusrat Binta Nizam, Sunwoo Kwak et al. (+3 more)
Cardiovascular disease (CVD) remains the leading cause of death among women, yet cardiovascular risk assessment often relies on clinical variables that may be missing, outdated, or unavailable in routine care. Screening mammography offers an opportunity for opportunistic cardiovascular risk stratification because it is routinely acquired and contains vascular features, including breast arterial calcifications (BAC), that are associated with cardiovascular risk and events. We evaluate whether mammography specific foundation models, originally pretrained for breast cancer-related tasks, can transfer to cardiovascular risk prediction without cardiovascular specific supervision or explicit BAC annotation. We constructed a 5-year major adverse cardiovascular event (MACE) cohort of 22,497 women linked to electronic health record outcomes, including 500 events (2.22% prevalence). The foundation models achieved AUROCs of 0.823 and 0.822 substantially exceeding an age-only model (AUROC 0.765), despite using only the screening mammogram as input, with no clinical variables. Both foundation models evaluated assigned substantially higher predicted risk to patients with radiologist-documented BAC, despite BAC never being used as a training label, and showed activation patterns consistent with vascular findings. Together, these findings suggest that mammography foundation models can recover clinically relevant cardiovascular risk information directly from mammographic pixels and suggest that screening mammography may provide an opportunistic source of cardiovascular risk information to complement conventional clinical assessment without additional imaging. Code is available in https://github.com/PauFeld/MammoCVD
Read the full abstract ▾
All authors Paula Feldman, Nusrat Binta Nizam, Sunwoo Kwak, Batuhan Karaman, Katerina Dodelzon, Mert Sabuncu
Computer Vision
Riemannian–Lorentz Fusion of Vision Transformers and State-Space Models
Badri N. Patro and Vijay S. Agneeswaran
Scaling deep learning faces critical bottlenecks: data exhaustion, exponential training costs, and resource concentration. Model merging combines pre-trained checkpoints without gradient descent, offering orders-of-magnitude savings versus retraining. Combining independently trained
vision models is difficult when their architectures and parameter shapes differ. Existing weight-space merging methods generally assume aligned, shape-compatible checkpoints, whereas a
Vision Transformer (ViT) and a state-space model (SSM) implement
token mixing with different operators. We study a hybrid Heterogeneous merging setting that retains both architectures while aligning parameter groups by semantic role. Our proposed Riemannian–Lorentz Parameter Fusion (RLPF) method projects aligned groups to common coordinates, lifts selected coordinates to the Lorentz hyperboloid model of
hyperbolic space, computes a regularized geodesic barycenter, and decodes the result into the two branches. A learned gate then combines branch logits for each input. Component groups use fixed
curvature values, with normalization parameters treated as Euclidean. In the results available in this manuscript, the fine-tuned system obtains 82.37% on CIFAR-10, 75.04% on Oxford-IIIT Pet, and 78.58% top-1 accuracy on ImageNet-1K; the corresponding best-parent accuracies are 76.54%, 71.42%, and 76.42%. On ImageNet-1K, the reported pre-fine-tuning initialization reaches 77.80%. These results support further study of geometry-aware heterogeneous fusion, but not a training-free single-checkpoint merge: RLPF is a two-branch hybrid whose gate and reported final models are trained.
Read the full abstract ▾
Computer Vision
LinePilot Digitizer: Line-Plot Recovery with Manual and Automatic Calibration
Fengbo Ma, Rayan Akhtar, Aakash H. Joshi et al. (+5 more)
Recovering numerical series from line plots requires accurate axis calibration and reliable curve extraction. We present LinePilot Digitizer (LinePilot), which combines continuous color-based curve recovery with three calibration modes: LinePilot (standard), LinePilot (enhanced), and LinePilot (OCR). We also introduce DigitizerBench, the first dedicated
benchmark for systematically evaluating digitizer performance, using an orthogonal design spanning signal, rendering, and plot-structure factors with complementary automatic and human-guided evaluations. We evaluate performance using failure-penalized capped normalized root-mean-square error (FPC-NRMSE), which assigns unit loss to missing, unusable, or catastrophically inaccurate outputs. On DigitizerBench-Full, LinePilot (OCR) achieves the lowest mean FPC-NRMSE (0.672) and highest trusted usability (38.2%) among the tested automatic pipelines. On DigitizerBench-Lite, LinePilot (enhanced) achieves the lowest mean FPC-NRMSE (0.081), 100% output success, and highest trusted usability (93.3%). The orthogonal
benchmark design further enables factor analysis to identify the factors that most significantly affect digitizer performance. Together, the three calibration modes provide a practical trade-off between automation, user control, and accuracy within a shared curve-recovery workflow.
Read the full abstract ▾
All authors Fengbo Ma, Rayan Akhtar, Aakash H. Joshi, Xiaoting Li, Haijian Sun, Zhen Xiang, Xianyan Chen, Yiping Zhao
Computer Vision
Open-vocabulary 3D object detection with promptable segmentation
Ömer Faruk Deniz and Mustafa Taha Koçyiğit
Three-dimensional object detection for autonomous driving is dominated by detectors trained on large corpora of human-annotated 3D boxes. Such a detector learns a fixed category list, and everything outside it is invisible. This paper asks whether the task can be solved training-free and open-vocabulary. A promptable
segmentation model (SAM3), queried with class names as text
prompts, supplies instance masks in the vehicle's six surround-view cameras, and the masks are turned into metric 3D boxes using the geometry of the scene. The core is a controlled three-stage comparison on nuScenes in which 2D detection is held fixed and only the source of 3D geometry changes. Geometry predicted from images alone reaches 0.183 mean average precision (mAP) under the official protocol; fitting boxes from raw
LiDAR points inside the same masks with training-free rules reaches 0.298 mAP / 0.348 nuScenes detection score (NDS) at zero labeling cost; borrowing supervised box geometry at inference time lifts the same detections to 0.413 mAP / 0.555 NDS, which locates the pipeline's largest deficit in measurement precision rather than 2D detection, while class confusion and confidence calibration survive that substitution. Reversing the direction, a three-state camera-witness rule built from the same masks improves a supervised
LiDAR-only detector from 0.596 to 0.630 mAP, roughly half the gain of fully supervised camera fusion, with no training. A coverage analysis shows that SAM3 finds 84% of in-range objects with a correctly named mask; the classes that fail in the official metric are misnamed or geometrically unforgiving, not unseen.
Read the full abstract ▾
Computer Vision
Can Vision-Language Models Judge Olympic Diving? From Reasoning to Scores in Zero-Shot Action Quality Assessment
Henry O. Velesaca, David Freire-Obregon, Luigi Miranda et al. (+1 more)
Automated action quality assessment (AQA) in Olympic sports remains a challenging task due to the complexity of human motion and the subjectivity inherent in expert judging. This work evaluates the capability of open-source
Vision-
Language Models (VLMs) to perform zero-shot action quality assessment on Olympic diving
videos using the AQA-7
benchmark dataset. In this regard, a regression-based framework is pro-posed to leverage both the semantic
reasoning and phase-level sub-scores generated by the VLMs, combining TF-IDF vectorization, dimensionality reduction, and ensemble learning to predict final competition scores. Experimental results show that standalone VLMs achieve moderate Spearman correlations below 0.32, while the proposed ensemble regression framework substantially improves performance in the reported evaluation, reaching a Spearman correlation of 0.67 with a four-model configuration. Textual
reasoning features con-sistently outperformed raw numerical sub-scores, highlighting the richness of VLM-generated explanations for action quality analysis. These findings suggest that VLMs hold strong potential as assistive tools for explainable and semi-automated sports performance evaluation. The code is publicly available on GitHub https://github.com/hvelesaca/olympic diving judge vlm
Read the full abstract ▾
All authors Henry O. Velesaca, David Freire-Obregon, Luigi Miranda, Abel Reyes-Angulo
Computer Vision
RAUL: Reference-Assisted Ureteroscopy Localization for Skill Assessment
Fangjie Li, Mai Bui, Charan Mohan et al. (+4 more)
Objective: Incomplete
navigation of anatomy during ureteroscopic kidney stone surgeries can contribute to repeat interventions. While skilled surgeons have lower reintervention rates, there are no objective metrics to quantify scope-
navigation performance to evaluate when a trainee becomes skilled. This work aims to recover ureteroscope trajectories from endoscopic
video and derive
navigation metrics to quantify differences in skill. Methods: We propose RAUL, a reference-assisted reconstruction framework for recovering ureteroscope trajectories from ureteroscope
videos only in phantoms. For each phantom, we use a slow, high-quality reference exploration
video to generate a reference reconstruction. We localize subsequent exploration
videos against this reference. We evaluate localization accuracy against electromagnetically tracked scope pose. We compute
navigation metrics from phantom exploration trajectories to compare
surgical residents across experience levels. Results: The proposed reference-assisted framework achieves a mean translation root mean square error of $0.5 \pm 0.1$ mm across 9 phantoms. Compared to standard Structure-from-Motion (SfM), the proposed pipeline increases frame-wise localization coverage from $50.5 \pm 14.9\%$ to $86.1 \pm 7.2\%$ of all
video frames. The reconstructed trajectories revealed significant differences between high- and low-experience trainees in established
navigation metrics. Conclusion: RAUL enables substantially more complete recovery of ureteroscope trajectories from
videos compared to standard SfM pipelines, enabling trajectory-based skill assessment without additional tracking equipment. Significance: To the best of our knowledge, this is the first use of
video-only recovery of ureteroscope trajectories without external tracking sensors for skill assessment, supporting scalable automated assessment of ureteroscopy
navigation skill.
Read the full abstract ▾
All authors Fangjie Li, Mai Bui, Charan Mohan, Michael Miga, Matthieu Chabanas, Nicholas Kavoussi, Jie Ying Wu
Computer Vision
Open ultrasound foundation model for robust segmentation and clinical measurement across heterogeneous settings
Chao Qin, Fahad Shahbaz Khan, Salman Khan et al. (+4 more)
Ultrasound is the most widely deployed imaging modality worldwide, yet clinical AI remains fragmented into narrow single-task models that fail when device, operator, or anatomy changes. Here we present SonoCorpus, an open resource unifying 456,963 images and 1,626,085 expert masks from 53 public
datasets spanning 24 clinical applications and 17 countries, and SonoBase, an interactive
segmentation foundation model pretrained on it. Across fifteen evaluation
datasets introducing new organs, devices, operators, and geographies, SonoBase outperforms SAM2, MedSAM2, and the concept-promptable MedSAM3 on every
dataset and matches per-
dataset specialist models trained on the same data; on fully external data it exceeds the accuracy these baselines achieve on their own in-distribution
benchmarks. Ejection fraction derived from its
segmentations falls within inter-observer variability (6.63% error), with fewer misclassifications at the defibrillator-candidacy threshold than either promptable baseline (13% versus 18–42%); fetal head-circumference (1.81~mm) and gestational-age (1.2 days) errors fall below inter-observer variability. Where a baseline fails outright, one in four test cases, SonoBase recovers a usable
segmentation in 81% of them, including on handheld probes operated by minimally trained users in two low- and middle-income countries (Sierra Leone and Tanzania). Five labeled examples can help the model adapt to a new setting, and the identical training protocol transfers well to newer models such as SAM3, locating the advantage in ultrasound-specific pretraining rather than any single architecture. To ensure reproducibility and enable the community to build on SonoBase as a platform, we release all checkpoints, optimizer states, data-split indices, deduplication hashes, and starter code at https://github.com/AlfredQin/sonobase.
Read the full abstract ▾
All authors Chao Qin, Fahad Shahbaz Khan, Salman Khan, Sarim Ather, Siddiq Anwar, Rao Muhammad Anwer, Shadab Khan
Computer Vision · Cross-listed from cs.RO
4D Radar Perception Algorithms for Autonomous Driving: A Review
Xumin Wu, Jun Zhou, Jilin Mei et al. (+2 more)
Research on 4D millimeter-wave radar perception
algorithms has flourished in recent years, extending from signal processing and object detection to semantic
segmentation, motion estimation, occupancy prediction, and dynamic scene reconstruction. This review organizes the field according to the
evolution of perception tasks and
algorithms. It first introduces radar fundamentals, data representations, and quality-enhancement methods, and then reviews object-level perception, motion and localization, local and dense spatial perception, and dynamic scene understanding. Across these directions, we compare radar-only learning,
multimodal fusion, and cross-modal supervision and knowledge
distillation. Particular
attention is paid to how elevation, Doppler measurements, and radar physical priors are exploited across tasks. We further summarize the task coverage, input data, annotations, and evaluation protocols of existing
datasets, clarifying the empirical support for different research directions. Finally, we discuss the common challenges and future directions of 4D radar perception for autonomous driving. This review provides a task-oriented perspective on the transition from sparse object perception to dynamic spatial understanding.
Read the full abstract ▾
All authors Xumin Wu, Jun Zhou, Jilin Mei, Chen Min, Yu Hu
Computer Vision · Cross-listed from eess.IV
Perceptual Refinement of an End-to-End Video Streaming Pipeline via Generative AI Layers
Emanuele Artioli, Farzad Tashtarian and Christian Timmerer
Traditional codecs treat every region of a frame alike; a generative layer can instead degrade the regions a viewer attends to least and reconstruct them at the client. We present PRESLEY, which extends the prior conference work ELVIS by replacing destructive block removal with adaptive in-place degradation under a removability mask, signaling per-block strength in a bit-packed side channel, and restoring via generative backbones conditioned on transmitted visual priors rather than unconditioned in-painting. We separate the problem into three goals: choosing which blocks to degrade, degrading them so the encoder spends fewer bits, and restoring them. Against its predecessor at matched rate, PRESLEY achieves a decisive mean -56.4% BD-rate reduction on delivered background quality across 13 rate ladders spanning multiple codecs and
dataset families. Against pristine baselines, PRESLEY defines the operating regime of generative transport: delivering substantial bitrate savings (up to -29.4% BD-rate) and superior background quality (17/23 sequences) in the target bit-starved regime, while maintaining foreground fidelity bit-exact. We further map where the theoretical headroom in this class of architecture lies. Using an exact leave-one-superblock-out combinatorial oracle as an additive empirical bound, we show that existing complexity heuristics already capture 83.3% of bit-cost savings, bounding remaining cost-axis headroom at about 5% of total bitrate. We then identify and model the primary unaddressed axis – post-restoration damage – which disperses widely (4.9-8.4 dB). We prove that this damage is predictable before transmission (held-out rho = +0.400), establishing the feasibility of transmit-time restorability modeling and defining the roadmap for joint rate-distortion-restoration selection rules.
Read the full abstract ▾
Computer Vision
PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection
Sara Pieri, Evangelos Kazakos, Shizhe Chen et al. (+2 more)
Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current
vision-
language models (VLMs) can generate fluent and detailed image captions, but reliably associating them with image pixels remains challenging. Existing methods that combine dense captioning with pixel-level grounding often produce either incomplete descriptions or inaccurate
segmentation masks. We study this problem through panoptic grounded captioning, a task that requires a VLM to describe both foreground objects and background regions while grounding each referring phrase with pixel-level masks. We make three contributions. First, we introduce PanoCaps, a human-annotated
benchmark constructed from panoptic
segmentation datasets. It provides dense captions with near-complete pixel coverage and image-text
alignments at the entity level, supporting both training and evaluation. We further propose a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric that jointly evaluates textual and mask agreement. Second, we formulate phrase grounding as selection from a phrase-conditioned pool of mask proposals and introduce PANORAMA, a VLM that conditions a pretrained segmenter on contextualized phrase representations to obtain candidate masks and learns to select those corresponding to each phrase. Training this interface jointly with caption generation enables PANORAMA to produce high-quality masks while allowing each phrase to refer to a single region or multiple instances. Third, PANORAMA achieves the best overall grounding on PanoCaps and matches or exceeds specialized models across several pixel-level grounding tasks. Experiments show that our method produces precise entity-level
segmentations while maintaining detailed, mask-consistent captions. Code, data and models are available at https://www.di.ens.fr/willow/research/panorama/.
Read the full abstract ▾
All authors Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic, Cordelia Schmid
Computer Vision
PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics
Bardienus P. Duisterhof, Kaifeng Zhang, Adam Hung et al. (+5 more)
World models endow perceptual systems with the ability to predict how scenes evolve under interaction. They are most beneficial when trained on diverse volumes of data, to instill a rich prior into downstream applications. Existing methods typically require
robot action labels to learn action-conditioned 3D dynamics, which excludes web
video data from the training pool. We study 3D point track completion as a pre-training objective for learning transferable 3D dynamics without
robot data. Given a single RGB-D observation and sparse partial 3D trajectories (tracks), we predict future 3D tracks of all observed points. We show this objective produces a rich 3D dynamics prior, without requiring
robot action labels. We contribute a diverse
dataset of 2.9 million synthetic frames spanning deformable, articulated, and rigid objects, and use it to train PointZero. We show that a flexible and expressive
transformer, PointZero, outperforms prior methods on the same data. We demonstrate the utility of our pre-training objective by post-training PointZero for two downstream applications: (1) action-conditioned 3D dynamics prediction and (2) imitation learning. When fine-tuned to condition on end-effector pose, PointZero outperforms the baselines on the recent PGND 3D dynamics
benchmark. When fine-tuned to predict
robot actions and 3D tracks, PointZero outperforms or matches the baselines on 6/7 simulated and real-world
robot manipulation tasks. We furthermore evaluate training PointZero from scratch to isolate the benefits of our proposed architecture from those of our proposed pre-training objective and
dataset. We release the
dataset, checkpoints, and full training recipe.
Read the full abstract ▾
All authors Bardienus P. Duisterhof, Kaifeng Zhang, Adam Hung, Bowen Wen, Stan Birchfield, Yunzhu Li, Deva Ramanan, Jeffrey Ichnowski
Computer Vision
In-Context Robot Learning with VLM Agents
Dongzhou Cheng, Taoran Yi, Ye Fang et al. (+12 more)
Enabling
robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of
embodied AI. No finite collection of demonstrations can cover every task and situation a
robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial
vision-
language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable
robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters? We introduce GPT-Policy, a general-
agent framework for in-context
robot learning. GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes
robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. In real-
robot trials, human
video demonstrations improve task completion even without
robot action labels, while aligned action references yield further gains on contact-sensitive tasks. These findings position GPT-Policy as a step toward
robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.
Read the full abstract ▾
All authors Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, Tong Wu
Computer Vision
Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation
Meng'en Qin, Yinchen Liu, Mingxuan Cui et al. (+1 more)
Visual signals require compact yet sufficient representations for robust downstream prediction. Convolutional sparse coding (CSC) provides an explicit mechanism for suppressing redundant components while preserving signal content, but its sparsity coefficient is often manually selected during training. We propose a training-adaptive convolutional sparse coding framework for robust visual signal representation. Specifically, we unfold the CSC optimization with the Fast Iterative Shrinkage-Thresholding
Algorithm (FISTA) and treat the sparsity coefficient as a differentiable variable jointly learned with the network parameters. From the information bottleneck perspective, this coefficient controls the trade-off between information retention and compression: the sparsity term promotes compact representations, while the reconstruction term together with task loss preserves task-relevant signal content. We further introduce a label-free post-training strategy that adjusts the compression strength for corrupted inputs with the main network parameters fixed. Experiments on CIFAR and ImageNet demonstrate competitive clean-data recognition and greatly improved robustness under different input perturbations.
Read the full abstract ▾
All authors Meng'en Qin, Yinchen Liu, Mingxuan Cui, Youlu Xing
Computer Vision
Track, Articulate, Act: Generating Articulation from Casual Human Videos
Jiaming Zhang and Homanga Bharadhwaj
Human
videos contain rich causal evidence for
robot manipulation: they reveal how hand motion induces object motion and produces task-relevant changes in object state. In this work, we study articulated objects such as doors, drawers, cabinets, laptops, ovens, and hinged containers that are ubiquitous in daily life and present unique challenges for
embodied interaction. These objects cannot be represented by a single pose; their motion depends on the underlying parts and joints. We introduce a real-to-sim framework that reconstructs a simulation-ready articulated object and hand-object interaction from a casual
monocular RGB
video, without RGB-D or multi-view input, prior scans, manually specified joints, or
robot demonstrations. Our key insight is that dense 3D point tracks provide an embodiment-agnostic articulation cue: points on the fixed link remain approximately stationary, while points on the moving link follow coherent revolute or prismatic motion. Our method segments the links, estimates the joint and its state trajectory, reconstructs an articulated asset, and aligns the recovered 3D hand motion with the object. Central to our approach is a modular recipe that repurposes powerful pretrained models for single-image 3D reconstruction, mesh
segmentation, and 3D scene flow, connecting their predictions through explicit
geometric reasoning to infer articulation. We use the reconstructed articulated object and the human hand trajectory to replay interactions through contact in MuJoCo. The framework shows how pretrained
vision models and explicit motion
reasoning can turn casual human
videos into articulated object models suitable for downstream
embodied interactions. https://track-articulate-act.github.io/
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
Luyao Zhu, Xun Wei Yee, Wei Li et al. (+2 more)
Large
vision-
language models have achieved remarkable progress in multi-modal understanding, yet their capabilities in educational settings remain insufficiently evaluated. In AI-assisted
language learning, models must interpret artistic imagery, understand its semantic, affective, and cultural content, and reason about visual context to support meaningful interaction. However, existing
benchmarks primarily focus on real-world images or domain-specific educational
reasoning, providing limited coverage of artistic educational content. To address this gap, we introduce MUSE, a
benchmark for evaluating large
vision-
language models on artistic image understanding in situated educational applications. MUSE decouples image annotation from question generation, enabling diverse tasks with controllable difficulty while reducing annotation effort. It comprises twelve tasks spanning visual perception, semantic and affective interpretation, culture understanding, and compositional
reasoning, together with diverse artistic images deliberately curated to center Singaporean and Southeast Asian multicultural contexts alongside Western art traditions, covering multiple themes and difficulty levels. Evaluation of open-source and proprietary models reveals substantial disparities across capability dimensions, particularly in affective interpretation and compositional
reasoning. Our analysis further identifies common failure modes and key challenges for developing trustworthy multi-modal models for education. We hope MUSE will serve as a standardized
benchmark for advancing multi-modal understanding in situated educational applications.
Read the full abstract ▾
All authors Luyao Zhu, Xun Wei Yee, Wei Li, Mun Thye Mak, Wee Siong Ng
Computer Vision
PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image
Sneha Paul, Guile Wu, Bingbing Liu et al. (+1 more)
Physical properties, such as friction, hardness, stiffness, and density, govern how
robots should grasp, manipulate and interact with objects, yet estimating these properties from RGB images remains challenging. Existing methods typically employ per-object reconstruction augmented with physical properties or directly query
vision-
language models at test time, which results in substantial computational overhead that limits their applicability. In this work, we present PhysVGGT, a feed-forward model that predicts dense maps of friction coefficient, Shore hardness, Young's modulus, and density, together with object-level mass, from a single RGB image in one forward pass. The key idea of PhysVGGT is to formulate physical property estimation as a dense per-pixel prediction problem and employ a visual geometry
transformer to extract geometry-aware
tokens from the input image followed by a dense prediction branch for estimating local physical properties and a global prediction branch for estimating object-level mass. In addition, we introduce a scalable pseudo-label generation pipeline that enables large-scale weakly supervised training for dense physical property prediction, substantially reducing the need for expensive direct physical measurements. Extensive experiments show that PhysVGGT achieves state-of-the-art performance on the ABO-500
dataset and generalizes effectively to the out-of-distribution NeRF2Physics
dataset. Moreover, PhysVGGT eliminates the need for per-object reconstruction and test-time optimization, achieving an inference latency of only 0.13s per image, making it $27\times$ faster than the previous state of the art.
Read the full abstract ▾
All authors Sneha Paul, Guile Wu, Bingbing Liu, Dongfeng Bai
Computer Vision
NormLift: From Lifted Features To Semantic Reliability In 3D Gaussian Splatting
Yihan Zang, Da Li, Dominik Engel et al. (+2 more)
Training-free weighted aggregation is widely used to lift 2D semantic features onto 3D Gaussians for open-vocabulary scene understanding, yet its theoretical role remains insufficiently understood. Existing analyses typically justify this operation from the rendering side, treating Gaussian features as linearly composable Euclidean variables for reconstructing 2D feature maps. However, this view does not match downstream 3D usage, where each Gaussian is often queried independently in a cosine-based
embedding space. We revisit feature lifting from the 3D side and formulate per-Gaussian assignment as a cosine
alignment problem on the CLIP unit sphere. Under this objective, the L2-normalized semantic back-projected feature emerges as the closed-form solution, providing a complementary interpretation of the standard lifting rule from the perspective of per-Gaussian semantic assignment. The same formulation further yields a norm decomposition into intra-view and inter-view consistency, suggesting that feature magnitude itself can serve as a semantic reliability signal. Calibrated by effective multi-view support, this reliability score guides a mode-voting refinement that preserves CLIP feature validity by avoiding linear averaging. Experiments on open-vocabulary 3D semantic
segmentation show that NormLift is an efficient, training-free framework that achieves strong performance across evaluation protocols.
Read the full abstract ▾
All authors Yihan Zang, Da Li, Dominik Engel, Shinkyu Park, Ivan Viola
Computer Vision
Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection
Girish A. Koushik, Diptesh Kanojia and Helen Treharne
When large
vision-
language models misclassify harmful memes, the failure may reflect missing internal evidence or an inability to route represented evidence to their outputs. We distinguish these cases in Gemma-3 and Qwen3.5 using sparse autoencoders, role-conditioned probes, causal interventions, and recovery experiments across six harmful content
benchmarks, with additional Spanish and Hindi-English code-mixed evaluations. Sparse readouts outperform native prediction on all six primary binary tasks: Qwen averages $0.740$ versus $0.432$ for native macro-F1, residual reconstruction reaches $0.486$, and Gemma improves from $0.532$ to $0.714$. These gains measure how accessible the label is to a supervised readout; they do not show that the model's native generation already applies such a decision rule. Under the evaluated scales, Qwen silent-feature ablation is $24-63$ times more probe-sensitive, whereas routed-feature patching on literal yes/no tasks is $16-140$ times more output-sensitive. Native-only threshold calibration explains much, but not all of the gap: on five tasks with matched probe scores, it recovers $69.8$% of the raw native-to-probe difference, while direct routing adds $0.094$ mean macro-F1 beyond calibrated native scoring. Joint gold-label, probe-KL, and pairwise LoRA supervision improves dedicated FHM prediction, but a gold-only adapter performs better on the shared seven-task mean. A case study of Gemma-3-12B on the Facebook Hateful Memes
dataset finds a distributed rank-32 image-
prompt interaction, reaching $0.756$ versus $0.685$ native macro-F1. Robustness controls show that the signal is not explained solely by accompanying OCR and depends on paired visual evidence, and that it extends beyond English. In many of the errors we study, the evidence is represented but does not reach the answer; therefore, routing is a common bottleneck in harmful meme classification.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts
Liyang Fan, Chi Wei, Yitai Li et al. (+8 more)
Multimodal coding
agents are expected to turn visual inputs into usable artifacts, and they act through a harness, the layer of tools, context management, and execution environment around the model. Existing evaluations often isolate short tool calls, API traces, or screenshot resemblance, and a low score under these proxies cannot say whether the model saw poorly, planned poorly, or was failed by its harness. We study scientific overview figure reconstruction, an
agent task in which a source image must become an editable PowerPoint slide that preserves text,
topology, layout, and native document structure. We introduce ReFigBench, a
benchmark and evaluation framework built on 1,000 real overview figures retrieved from arXiv papers with full provenance. Coding
agents from four model families reconstruct every figure under two workflows, direct code generation and a specialized PPTX workflow, and the strongest model runs inside two commercial harnesses, yielding ten configurations. Evaluation combines deterministic artifact checks, repeated automated scoring by judges from two model families, and blinded human comparisons. Perception remains a bottleneck that iterative rendering only partly repays. Whether workflow effort converts into quality depends on the model together with its harness, since the same model gains from the specialized workflow inside one harness and loses inside the other, and the harness shifts scores even under an identical direct
prompt. The specialized workflow erases native connectors in every configuration, human judges still prefer its renderings in most matchups, and even the strongest
agent falls short of the rubric ceiling. These results expose the tension between fidelity and editability as the central challenge for practical
multimodal document
agents.
Read the full abstract ▾
All authors Liyang Fan, Chi Wei, Yitai Li, Xinping Bi, Guhong Chen, Chenghao Sun, Haoxiang Yang, Qingwen Li, Kai Yan, Hong Li, Bo Li
Computer Vision
Copy What Is Seen, Generate What Is Not: Training-Free Anomaly-Aware Video Restoration
Zhida Qu and Shengchao Chen
A surveillance system that detects an anomaly often has to repair the footage as well, yet the two tasks are studied in isolation: training-free anomaly detectors stop at a score or a label, while training-free
video editing answers to a user
prompt rather than to a detector. This paper proposes AVR (Anomaly-aware
Video Restoration), which closes that gap with frozen pretrained models alone and generates content only where the clip offers no evidence to copy. Motion evidence first gates open-vocabulary proposals into spatio-temporal masks. A background prior computed from the clip then fills every pixel the anomaly ever uncovers, leaving
diffusion to synthesize only what no frame showed, and a frozen verifier decides per clip whether to trust a classical, a prior-anchored, or a background-conditioned restorer. Extensive experiments on three surveillance
datasets, under both full-reference anomaly injection and real anomalies, show that AVR leads full-frame fidelity under oracle masks, matches three trained
video inpainters inside the edited region, and outperforms a detect-then-generate pipeline on the masks it produces itself, while suppressing both the residual anomaly and the flicker of free
diffusion.
Read the full abstract ▾
Computer Vision
Using OCR Heads to Verbalize Image Semantics
Sheridan Feucht, Benno Krojer, Sarah Wang et al. (+3 more)
How do VLMs map from pixels to semantics? To understand this general question, we focus on a narrow one: studying how VLMs perform optical character recognition (OCR). Across four models, we identify
attention heads causally necessary for OCR, and discover that these are in fact general-purpose heads that output interpretable semantic features across all image
tokens. For example, pointing these heads at an image
token containing the word "bike" causes Qwen3-VL-8B to output "bike," but pointing them at a bird wing causes the model to output the
token "feathers." We collapse these heads'
attention weights into a single verbalization lens transformation that reveals interpretable semantic features in hidden states across all layers. When combined with projection to vocabulary space, we can obtain interpretable labels starting from layer 0, showing that image representations are in fact aligned with
language in early layers. We find that we can also use the inverse of this transformation to edit non-word concepts, e.g., replacing a tractor with a revolver in a naturalistic image, providing causal evidence that this subspace is useful for more than just OCR. Our results are an example of how the study of specific mechanisms can shed light on broader interpretability problems.
Read the full abstract ▾
All authors Sheridan Feucht, Benno Krojer, Sarah Wang, Henry Abrahamsen, Byron C. Wallace, David Bau
Computer Vision
DISTA-Net++: Rethinking Infrared Small Target Unmixing Beyond Sub-Pixel Separation
Mengze Xu, Zhu Liu, Weidong Sheng et al. (+4 more)
Long-range infrared imaging frequently confronts dense target clusters whose diffraction-limited signatures merge into a single indistinguishable blob, concealing the number, sub-pixel positions, and radiant intensities of the underlying sources. While deep learning has advanced general object detection, resolving such Closely-Spaced Infrared Small Targets (CSIST) remains largely unexplored, owing to a systemic infrastructure void and a fundamental paradigm mismatch. The dominant formulation, which reduces unmixing to a blind, discrete sub-pixel separation, is inherently insufficient: without semantic guidance, the ill-posed inverse problem admits ambiguous solutions plagued by false and missed detections, while grid-based discretization locks predictions onto fixed
lattice centers, chaining precision to prohibitively expensive grid refinement. We argue that CSIST unmixing should instead be informed and continuous. To ground this paradigm shift, we establish the first comprehensive open-source ecosystem for the field, comprising the large-scale CSIST-100K
benchmark, a tailored metric suite, and the GrokCSO toolkit. Upon this foundation, we propose DISTA-Net++, which anchors a dynamic deep unfolding backbone with two synergistic mechanisms: a Count-Guided Prior that injects the global target count as an explicit semantic constraint to regularize the solution space, and a Continuous Coordinate Rectification that regresses off-grid offsets to decouple localization accuracy from grid resolution. Extensive experiments validate our paradigm: even under the most economical 3x division, DISTA-Net++ surpasses 7x-division state-of-the-art methods by 16.15% in CSO-mAP and 62.96% in count accuracy at merely one-sixth of their computation, demonstrating that unmixing precision need not be purchased with finer discretization. The complete ecosystem is available at https://github.com/GrokCV/GrokDet.
Read the full abstract ▾
All authors Mengze Xu, Zhu Liu, Weidong Sheng, Boyang Li, Yimian Dai, Ming-Ming Cheng, Jian Yang
Computer Vision · Cross-listed from cs.CL
Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes
Marcel Granero-Moya, Carolina del Corral Farrarós, Gloria Haro et al. (+2 more)
Sign
language processing advances rapidly for high-resource
languages such as American Sign
Language (ASL), yet most of the world's sign
languages lack the phonological annotations new methods require. We present the first zero-shot cross-lingual framework for handshape recognition, transferring from ASL to Catalan Sign
Language (LSC). Our approach leverages the decomposition of handshapes into five phonological features – selected fingers, flexion, spread, thumb position, and thumb contact – shared across both
languages, to decode LSC handshapes from predicted features via a composite phonological distance metric. We evaluate three architectures (MLP, SL-GCN, SHuBERT) trained on two ASL corpora (PopSign, Sem-Lex) against a 37-handshape, single-signer LSC
benchmark. Zero-shot transfer proves viable once recording-format disparities are harmonized, reaching 80.0% phonological feature accuracy and 54.5% expected handshape accuracy. Phonological decomposition thus offers a bridge for extending sign
language technologies to low-resource
languages without any target-
language video training labels.
Read the full abstract ▾
All authors Marcel Granero-Moya, Carolina del Corral Farrarós, Gloria Haro, Coloma Ballester, Ricardo Marques
Computer Vision
Toward Markerless Video-based Tremor Analysis: Objective Quantification of Pathological Tremor in Mouse Preclinical Models
Yota Koshimoto, Akihiro Tsukahara, Yasuhiro Moriwaki et al. (+1 more)
Tremor is a movement disorder characterized by involuntary, rhythmic oscillations of body parts and is a hallmark of several neurological conditions, including Parkinson's disease and essential tremor. Elucidating its underlying mechanisms relies heavily on mouse models, which offer genetic manipulability and translational relevance to human neural circuitry. Accordingly, these models are indispensable for studying tremor pathophysiology. So far, electromyography and accelerometers have been used as methods to quantitatively observe tremors in mice. However, these methods have several drawbacks, such as high costs and complex setups. In particular, the invasive
surgical implantation of devices causes significant stress to the animals. Although RGB-based methods offer non-invasive and cost-effective alternatives, they often lack the sensitivity required to detect subtle tremors. Therefore, this paper addresses these challenges by achieving mouse tremor severity estimation using conventional RGB cameras only. To address the challenging task of isolating tremor-related vibrations while the mouse itself is also in motion, our pipeline incorporates
segmentation-based pre-processing to extract the mouse region and a Tremor Score Estimation Module that captures subtle tremors with high sensitivity. In the experiments, we assessed tremors in unrestrained mice using a non-invasive method with two standard cameras. The results demonstrated a strong correlation with accelerometer measurements and confirmed that the method accurately captured the intensity-dependent characteristics of tremors. The project page is available at https://isogawalab.github.io/
Video-based-Tremor-Analysis-Project/.
Read the full abstract ▾
All authors Yota Koshimoto, Akihiro Tsukahara, Yasuhiro Moriwaki, Mariko Isogawa
Computer Vision
Geometry beneath the Waves: Dense Priors for Sparse-View Underwater 3D Gaussian Splatting
Harvey Caldeira, Haoran Wang, Guoxi Huang et al. (+3 more)
Underwater 3D reconstruction remains challenging under sparse views, where scattering, absorption, and suspended particles degrade feature correspondences and
geometric estimation. Although feed-forward geometry foundation models offer an alternative to conventional Structure-from-Motion, their direct application
underwater produces noisy and fragmented geometry that limits subsequent 3D Gaussian Splatting (3DGS). We propose a sparse-view
underwater reconstruction framework that adapts feed-forward geometry to
underwater degradation and exploits its dense
geometric priors for view synthesis. First, we adapt VGGT using LoRA and teacher–student
distillation, training on synthetically degraded
underwater images while preserving clean
geometric supervision. This improves robustness to
underwater appearance distortions without modifying the pretrained prediction heads. Second, the predicted dense geometry initialises an intermediate 3DGS representation that generates geometry-guided pseudo-views, increasing view overlap and strengthening feature tracks for subsequent RUSplatting optimisation. Experiments on SeaThru-NeRF and Submerged3D demonstrate improved reconstruction quality under sparse-view conditions. On SeaThru-NeRF, our method improves RUSplatting from 24.37 to 27.11 dB PSNR and increases SSIM from 0.7611 to 0.8634, while achieving the best average PSNR and LPIPS on Submerged3D. These results demonstrate the potential of domain-adapted
geometric priors for robust sparse-view
underwater 3D reconstruction.
Read the full abstract ▾
All authors Harvey Caldeira, Haoran Wang, Guoxi Huang, Shaoyu Cai, Rachel Fu, Nantheera Anantrasirichai
Computer Vision
Mask IPL: Noise-Free Intrinsic Position Learning via Computation Graph Clipping for Event-Based Spike-Driven Tracking
Yimeng Shan and Malu Zhang
Spiking Neural Networks (SNNs) match the event-driven nature of event cameras and naturally extract spatiotemporal features. These properties have motivated a series of recent studies on event-based tracking with SNNs. Intrinsic Position Learning (IPL) acquires strong position information without introducing additional parameters, making it a mainstream approach for position encoding in event-based spike-driven tracking. However, the mechanism behind its effectiveness lacks systematic theoretical analysis. Moreover, our analysis reveals that IPL introduces noise in both forward and backward propagation. The former increases inference error, while the latter prevents parameters from converging to better solutions. This paper presents a systematic analysis of IPL and demonstrates that its effectiveness stems from the synergy between IPL and multi-stage convolution. The zero blocks in the joint tensor act as zero padding for convolution, and the resulting boundary effect propagates layer by layer through multi-stage convolution. Every parameter update is therefore driven by a gradient that perceives the relative displacement between template and search frames. Positional encoding added after the convolutional stage cannot provide this information. We further propose a simple Computation Graph Clipping method that applies a validity mask determined by the layout to the operations of every layer, making invalid regions equivalent to zero padding in both forward and backward propagation. This eliminates the noise without introducing additional parameters and makes the actual gradient coincide with the ideal gradient. We name the improved method Mask IPL. Without increasing parameters or computational cost, Mask IPL improves the AUC of the Tiny-scale tracker on FE108, FELT, and VisEvent, and consistently improves the Base-scale tracker as well.
Read the full abstract ▾
Computer Vision
RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
Liyang Fan, Xinping Bi, Yitai Li et al. (+3 more)
Graphical User Interface (GUI) grounding is a fundamental perception task for
multimodal agents, enabling them to interpret natural
language instructions and interact with digital interfaces. Existing methods face a fundamental trade-off between accuracy and efficiency: direct full-image inference often fails to capture small or visually similar UI elements, while multi-crop strategies improve localization at the cost of multiple expensive
Vision-
Language Model (VLM) calls per query. To address this challenge, we propose RankGround, a two-stage framework that achieves accurate GUI grounding with a single VLM call per query. Central to our approach is GroundRanker, a lightweight
multimodal reranker that identifies the most promising crop from a dense candidate set. Because no off-the-shelf ranking
dataset is available, we construct ranking supervision data from existing grounding
datasets. A strict containment criterion and boundary-aware positive augmentation improve
alignment and spatial coverage in cluttered layouts. GroundRanker is then trained with a two-stage curriculum: a pointwise objective first learns coarse containment, and a listwise objective refines subtle semantic and spatial distinctions among visually similar crops. Experimental results show that RankGround consistently outperforms strong baselines while reducing computational cost. It achieves 1.4 times faster inference and improves localization accuracy by 5.5% on average over the second-best method across all backbones and screen scales, establishing a new state of the art in both efficiency and precision for GUI grounding.
Read the full abstract ▾
All authors Liyang Fan, Xinping Bi, Yitai Li, Shuaimin Li, Hui Li, Min Yang
Computer Vision
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging
Johannes Kaiser, Florian Braunmiller, Daniel Rückert et al. (+1 more)
AI models for
multimodal medical imaging must balance modality-specific specialization with cross-modal shared representations, a trade-off that pure Mixture-of-Experts (MoE) architectures currently fail to satisfy. Expert-based routing improves in-domain learning but may sacrifice cross-modal signals, which appear particularly important for rare (low-prevalence) pathologies in our experiments. To resolve this, we introduce Generalist-Specialist-MoE (GS-MoE), a two-branch (MoE) architecture that couples a cross-modal generalist model with distinct modality-specific specialists (experts) via domain-constrained feature fusion. On RadImageNet (1.35M images, 165 pathologies, three modalities), GS-MoE recovers detection of six low-prevalence pathologies on which every baseline scores F1 $=$ 0, with per-class gains up to +0.60 F1. It attains this while even slightly exceeding dense and specialist-only MoE aggregate baselines (MCC 0.770), while using ${\sim}53\%$ fewer active parameters at inference than the strongest investigated dense model.
Read the full abstract ▾
All authors Johannes Kaiser, Florian Braunmiller, Daniel Rückert, Georgios Kaissis
Computer Vision
Video-Based Markerless Motion Capture for Clinical and Rehabilitation Biomechanics: A PRISMA-ScR Scoping Review of Validated Architectures, Clinical Readiness, and Emerging Methods
Florian Delaplace, Elodie Piche, Frédéric Chorin et al. (+1 more)
Background..
Video-based markerless motion capture promises movement analysis without the cost, space and skin-marker constraints of optoelectronic systems, with particular potential for clinical and rehabilitation settings. Whether validated pipelines yet deliver clinically acceptable biomechanics, and how they relate to the underlying computer-
vision research, remains unclear. Methods. We conducted a scoping review following the PRISMA extension for Scoping Reviews, with a registered protocol and searches of PubMed, Scopus and IEEE Xplore (January 2015 to February 2026; the computer-
vision scan was updated to July 2026). A dual-tier design paired a primary corpus of validated biomechanical studies with a complementary, curated and deliberately non-exhaustive corpus of emerging computer-
vision work, used qualitatively. We charted study characteristics, pipeline architecture, validation methods and joint-angle accuracy. Results. We included 117 studies, most published from 2024 onward and conducted on healthy adults walking in a laboratory. Pipelines formed five architectural families across
monocular and multi-camera modalities; most reported raw joint angles without biomechanical refinement. Sagittal lower-limb agreement clustered around 5 to 6{\textdegree}, generally short of clinical acceptability, while out-of-plane kinematics, kinetics, and pathological or older populations were rarely validated. Emerging computer-
vision building blocks (foundation-model mesh recovery, differentiable inverse kinematics,
video-based kinetics) were almost absent from validated studies. Conclusions.
Video-based markerless capture is not yet interchangeable with marker-based systems for clinical joint kinematics, and it remains barely validated where rehabilitation needs it most: older and pathological populations, out-of-plane kinematics, and kinetics. Mapping this evidence gap onto emerging computer-
vision advances, we propose hypothesis-generating design guidelines, not a validated method, to steer the next generation of pipelines toward accessible, clinically meaningful movement analysis.
Read the full abstract ▾
All authors Florian Delaplace, Elodie Piche, Frédéric Chorin, Raphael Zory
Computer Vision
GenStream: Semantic Streaming Framework for Generative Reconstruction of Human-centric Media
Emanuele Artioli, Daniele Lorenzi, Shivi Vats et al. (+2 more)
Video streaming dominates global internet traffic, yet conventional pipelines remain inefficient for structured, human-centric content such as sports, performance, or interactive media. Standard codecs re-encode entire frames, foreground and background alike, treating all pixels uniformly and ignoring the semantic structure of the scene. This leads to significant bandwidth waste, particularly in scenarios where backgrounds are static and motion is constrained to a few salient actors. We introduce GenStream, a semantic streaming framework that replaces dense
video frames with compact, structured metadata. Instead of transmitting pixels, GenStream encodes each scene as a combination of skeletal keypoints, camera viewpoint parameters, and a static 3D background model. These elements are transmitted to the client, where a generative model reconstructs photorealistic human figures and composites them into the 3D scene from the original viewpoint. This paradigm enables extreme compression, achieving over 99.9% bandwidth reduction compared to HEVC for the continuous data stream. We partially validate GenStream on Olympic figure skating footage and demonstrate potential for high perceptual fidelity under minimal data. While acknowledging the significant computational costs shifted to the client and challenges in generalization, GenStream opens new directions in volumetric avatar synthesis, canonical 3D actor fusion across views, and personalized viewing experiences, laying the groundwork for scalable, intelligent streaming in the post-codec era.
Read the full abstract ▾
All authors Emanuele Artioli, Daniele Lorenzi, Shivi Vats, Farzad Tashtarian, Christian Timmerer