Computer Vision · Cross-listed from eess.IV
Few-Shot Continual Learning for 3D Brain MRI with Frozen Foundation Models
Chi-Sheng Chen, Xinyu Zhang, Guan-Ying Chen et al. (+3 more)
Foundation models pretrained on large-scale 3D medical imaging data face challenges when adapted to multiple downstream tasks under continual learning with limited labeled data. We address few-shot continual learning for 3D
brain MRI by combining a frozen pretrained backbone with task-specific Low-Rank Adaptation (LoRA) modules. Tasks arrive sequentially – tumor
segmentation (BraTS) and
brain age estimation (IXI) – with no replay of previous task data. Each task receives a dedicated LoRA adapter; only the adapter and task-specific head are trained while the backbone remains frozen, thereby eliminating catastrophic forgetting by design (BWT=0). In continual learning, sequential full fine-tuning suffers severe forgetting (T1 Dice drops from 0.80 to 0.16 after T2), while sequential linear probing achieves strong T1 (Dice 0.79) but fails on T2 (MAE 1.45). Our LoRA approach achieves the best balanced performance across both tasks: T1 Dice 0.62$\pm$0.07, T2 MAE 0.16$\pm$0.05, with zero forgetting and $<$0.1% trainable parameters per task, though with noted systematic age underestimation in T2 (Wilcoxon $p<0.001$). Frozen foundation models with task-specific LoRA adapters thus offer a practical solution when both tasks must be maintained under few-shot continual learning.
Read the full abstract ▾
All authors Chi-Sheng Chen, Xinyu Zhang, Guan-Ying Chen, Qiuzhe Xie, Fan Zhang, En-Jui Kuo
Computer Vision · Cross-listed from cs.RO
V-MORALS: Visual Morse Graph-Aided Estimation of Regions of Attraction in a Learned Latent Space
Faiz Aladin, Ashwin Balasubramanian, Lars Lindemann et al. (+1 more)
Reachability analysis has become increasingly important in robotics to distinguish safe from unsafe states. Unfortunately, existing reachability and safety analysis methods often fall short, as they typically require known system dynamics or large
datasets to estimate accurate system models, are computationally expensive, and assume full state information. A recent method, called MORALS, aims to address these shortcomings by using topological tools to estimate Regions of Attraction (ROA) in a low-dimensional latent space. However, MORALS still relies on full state knowledge and has not been studied when only sensor measurements are available. This paper presents Visual Morse Graph-Aided Estimation of Regions of Attraction in a Learned Latent Space (V-MORALS). V-MORALS takes in a
dataset of image-based trajectories of a system under a given controller, and learns a latent space for reachability analysis. Using this learned latent space, our method is able to generate well-defined Morse Graphs, from which we can compute ROAs for various systems and controllers. V-MORALS provides capabilities similar to the original MORALS architecture without relying on state knowledge, and using only high-level sensor data. Our project website is at: https://v-morals.onrender.com.
Read the full abstract ▾
All authors Faiz Aladin, Ashwin Balasubramanian, Lars Lindemann, Daniel Seita
Computer Vision
All in One: Unifying Deepfake Detection, Tampering Localization, and Source Tracing with a Robust Landmark-Identity Watermark
Junjiang Wu, Liejun Wang and Zhiqing Guo
With the rapid advancement of deepfake technology, malicious face manipulations pose a significant threat to personal
privacy and social security. However, existing proactive forensics methods typically treat deepfake detection, tampering localization, and source tracing as independent tasks, lacking a unified framework to address them jointly. To bridge this gap, we propose a unified proactive forensics framework that jointly addresses these three core tasks. Our core framework adopts an innovative 152-dimensional landmark-identity watermark termed LIDMark, which structurally interweaves facial landmarks with a unique source identifier. To robustly extract the LIDMark, we design a novel Factorized-Head Decoder (FHD). Its architecture factorizes the shared backbone features into two specialized heads (i.e., regression and classification), robustly reconstructing the embedded landmarks and identifier, respectively, even when subjected to severe distortion or tampering. This design realizes an "all-in-one" trifunctional forensic solution: the regression head underlies an "intrinsic-extrinsic" consistency check for detection and localization, while the classification head robustly decodes the source identifier for tracing. Extensive experiments show that the proposed LIDMark framework provides a unified, robust, and imperceptible solution for the detection, localization, and tracing of deepfake content. The code is available at https://github.com/vpsg-research/LIDMark.
Read the full abstract ▾
Computer Vision
Modelling and Simulation of Neuromorphic Datasets for Anomaly Detection in Computer Vision
Mike Middleton, Teymoor Ali, Hakan Kayan et al. (+6 more)
Limitations on the availability of Dynamic
Vision Sensors (DVS) present a fundamental challenge to researchers of neuromorphic computer
vision applications. In response,
datasets have been created by the research community, but often contain a limited number of samples or scenarios. To address the lack of a comprehensive simulator of neuromorphic
vision datasets, we introduce the Anomalous Neuromorphic Tool for Shapes (ANTShapes), a novel
dataset simulation framework. Built in the Unity engine, ANTShapes simulates abstract, configurable 3D scenes populated by objects displaying randomly-generated behaviours describing attributes such as motion and rotation. The sampling of object behaviours, and the labelling of anomalously-acting objects, is a statistical process following central limit
theorem principles.
Datasets containing an arbitrary number of samples can be created and exported from ANTShapes, along with accompanying label and frame data, through the adjustment of a limited number of parameters within the software. ANTShapes addresses the limitations of data availability to researchers of event-based computer
vision by allowing for the simulation of bespoke
datasets to suit purposes including object recognition and localisation alongside anomaly detection.
Read the full abstract ▾
All authors Mike Middleton, Teymoor Ali, Hakan Kayan, Basabdatta Sen Bhattacharya, Charith Perera, Oliver Rhodes, Elena Gheorghiu, Mark Vousden, Martin A. Trefzer
Computer Vision · Cross-listed from eess.IV
SegReg: Latent Space Regularization for Improved Medical Image Segmentation
Puru Vaish, Amin Ranem, Felix Meister et al. (+3 more)
Medical image
segmentation models are typically optimised with voxel-wise losses that constrain predictions only in the output space. This leaves latent feature representations largely unconstrained, potentially limiting generalisation. We propose {SegReg}, a latent-space regularisation framework that operates on feature maps of U-Net models to encourage structured
embeddings while remaining fully compatible with standard
segmentation losses. Integrated with the nnU-Net framework, we evaluate SegReg on prostate, cardiac, and hippocampus
segmentation and demonstrate consistent improvements in domain generalisation. Furthermore, we show that explicit latent regularisation improves continual learning by reducing task drift and enhancing forward transfer across sequential tasks without adding
memory or any extra parameters. These results highlight latent-space regularisation as a practical approach for building more generalisable and continual-learning-ready models.
Read the full abstract ▾
All authors Puru Vaish, Amin Ranem, Felix Meister, Tobias Heimann, Christoph Brune, Jelmer M. Wolterink
Computer Vision · Cross-listed from eess.IV
SGDC: Structurally-Guided Dynamic Convolution for Medical Image Segmentation
Bo Shi, Wei-ping Zhu and M.N.S. Swamy
Spatially variant dynamic convolution provides a principled approach of integrating spatial adaptivity into deep neural networks. However, mainstream designs in medical
segmentation commonly generate dynamic kernels through average pooling, which implicitly collapses high-frequency spatial details into a coarse, spatially-compressed representation, leading to over-smoothed predictions that degrade the fidelity of fine-grained clinical structures. To address this limitation, we propose a novel Structure-Guided Dynamic Convolution (SGDC) mechanism, which leverages an explicitly supervised structure-extraction branch to guide the generation of dynamic kernels and gating signals for structure-aware feature modulation. Specifically, the high-fidelity boundary information from this auxiliary branch is fused with semantic features to enable spatially-precise feature modulation. By replacing context aggregation with pixel-wise structural guidance, the proposed design effectively prevents the information loss introduced by average pooling. Experimental results show that SGDC achieves state-of-the-art performance on ISIC 2016, PH2, ISIC 2018, and CoNIC
datasets, delivering superior boundary fidelity by reducing the Hausdorff Distance (HD95) by 2.05, and providing consistent IoU gains of 0.99%-1.49% over pooling-based baselines. Moreover, the mechanism exhibits strong potential for extension to other fine-grained, structure-sensitive
vision tasks, such as small-object detection, offering a principled solution for preserving structural integrity in medical image analysis. To facilitate reproducibility and encourage further research, the implementation code for both our SGE and SGDC modules has been is publicly released at https://github.com/solstice0621/SGDC.
Read the full abstract ▾
Computer Vision · Cross-listed from math.AC
Multiprojective Geometry of Compatible Triples of Fundamental and Essential Matrices
Timothy Duff, Viktor Korotynskiy, Anton Leykin et al. (+1 more)
We characterize the variety of compatible fundamental
matrix triples by computing its multidegree and multihomogeneous vanishing ideal. This answers the first interesting case of a question recently posed by Bråtelund and Rydell. Our result improves upon previously discovered sets of algebraic constraints in the
geometric computer
vision literature, which are all incomplete (as they do not generate the vanishing ideal) and sometimes make restrictive assumptions about how a
matrix triple should be scaled. Our discussion touches more broadly on generalized compatibility varieties, whose multihomogeneous vanishing ideals are much less well understood. One of our key new discoveries is a simple set of quartic constraints vanishing on compatible fundamental
matrix triples. These quartics are also significant in the setting of essential
matrices: together with some previously known constraints, we show that they locally cut out the variety of compatible essential
matrix triples.
Read the full abstract ▾
All authors Timothy Duff, Viktor Korotynskiy, Anton Leykin, Tomas Pajdla
Computer Vision · Cross-listed from eess.IV
SALIENT: Frequency-Aware Paired Diffusion for Controllable Long-Tail CT Detection
Yifan Li, Mehrdad Salimitari, Taiyu Zhang et al. (+2 more)
Detection of rare lesions in whole-body CT is fundamentally limited by extreme class imbalance and low target-to-volume ratios, producing precision collapse despite high AUROC. Synthetic augmentation with
diffusion models offers promise, yet pixel-space
diffusion is computationally expensive, and existing mask-conditioned approaches lack controllable attribute-level regulation and paired supervision for accountable training. We introduce SALIENT, a mask-conditioned wavelet-domain
diffusion framework that synthesizes paired lesion-masking volumes for controllable CT augmentation under long-tail regimes. Instead of denoising in pixel space, SALIENT performs structured
diffusion over discrete wavelet coefficients, explicitly separating low-frequency brightness from high-frequency structural detail. Learnable frequency-aware objectives disentangle target and background attributes (structure, contrast, edge fidelity), enabling interpretable and stable optimization. A 3D VAE generates diverse volumetric lesion masks, and a semi-supervised teacher produces paired slice-level pseudo-labels for downstream mask-guided detection. SALIENT improves generative realism, as reflected by higher MS-SSIM (0.63 to 0.83) and lower FID (118.4 to 46.5). In a separate downstream evaluation, SALIENT-augmented training improves long-tail detection performance, yielding disproportionate AUPRC gains across low prevalences and target-to-volume ratios. Optimal synthetic ratios shift from 2x to 4x as labeled seed size decreases, indicating a seed-dependent augmentation regime under low-label conditions. SALIENT demonstrates that frequency-aware
diffusion enables controllable, computationally efficient precision rescue in long-tail CT detection.
Read the full abstract ▾
All authors Yifan Li, Mehrdad Salimitari, Taiyu Zhang, Guang Li, David Dreizin
Computer Vision
DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation
Varun Gopal, Rishabh Jain, Aradhya Mathur et al. (+6 more)
Graphic layouts serve as an important and engaging medium for visual communication across different channels. While recent layout generation models have demonstrated impressive capabilities, they frequently fail to align with nuanced human aesthetic judgment. Existing preference
datasets and reward models trained on text-to-image generation do not generalize to layout evaluation, where the spatial arrangement of identical elements determines quality. To address this critical gap, we introduce DesignSense-10k, a large-scale
dataset of 10,235 human-annotated preference pairs for graphic layout evaluation. We propose a five-stage curation pipeline that generates visually coherent layout transformations across diverse aspect ratios, using semantic grouping, layout prediction, filtering, clustering, and VLM-based refinement to produce high-quality comparison pairs. Human preferences are annotated using a 4-class scheme (left, right, both good, both bad) to capture subjective ambiguity. Leveraging this
dataset, we train DesignSense, a
vision-
language model-based classifier that substantially outperforms existing open-source and proprietary models across comprehensive evaluation metrics (54.6% improvement in Macro F1 over the strongest proprietary baseline). Our analysis shows that frontier VLMs remain unreliable overall and fail catastrophically on the full four-class task, underscoring the need for specialized, preference-aware models. Beyond the
dataset, our reward model DesignSense yields tangible downstream gains in layout generation. Using our judge during RL based training improves generator win rate by about 3%, while inference-time scaling, which involves generating multiple candidates and selecting the best one, provides a 3.6% improvement. These results highlight the practical impact of specialized, layout-aware preference modeling on real-world layout generation quality.
Read the full abstract ▾
All authors Varun Gopal, Rishabh Jain, Aradhya Mathur, Nikitha SR, Sohan Patnaik, Sudhir Yarram, Mayur Hemani, Balaji Krishnamurthy, Mausoom Sarkar
Computer Vision · Cross-listed from cs.RO
Demystifying Action Space Design for Robotic Manipulation Policies
Yuchun Feng, Jinliang Zheng, Zhihao Wang et al. (+5 more)
The specification of the action space plays a pivotal role in imitation-based robotic manipulation policy learning, fundamentally shaping the optimization landscape of policy learning. While recent advances have focused heavily on scaling training data and model capacity, the choice of action space remains guided by ad-hoc heuristics or legacy designs, leading to an ambiguous understanding of robotic policy design philosophies. To address this ambiguity, we conducted a large-scale and systematic empirical study, confirming that the action space does have significant and complex impacts on robotic policy learning. We dissect the action design space along temporal and spatial axes, facilitating a structured analysis of how these choices govern both policy learnability and control stability. Based on 13,000+ real-world rollouts on a bimanual
robot and evaluation on 500+ trained models over four scenarios, we examine the trade-offs between absolute vs. delta representations, and joint-space vs. task-space parameterizations. Our large-scale results suggest that properly designing the policy to predict delta actions consistently improves performance, while joint-space and task-space representations offer complementary strengths, favoring control stability and generalization, respectively.
Read the full abstract ▾
All authors Yuchun Feng, Jinliang Zheng, Zhihao Wang, Dongxiu Liu, Jianxiong Li, Jiangmiao Pang, Tai Wang, Xianyuan Zhan
Computer Vision
MediX-R1: Open Ended Medical Reinforcement Learning
Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed et al. (+5 more)
We introduce MediX-R1, an open-ended Reinforcement Learning (RL) framework for medical
multimodal large
language models (MLLMs) that enables clinically grounded, free-form answers beyond multiple-choice formats. MediX-R1 fine-tunes a baseline
vision-
language backbone with Group Based RL and a composite reward tailored for medical
reasoning: an LLM-based accuracy reward that judges semantic correctness with a strict YES/NO decision, a medical
embedding-based semantic reward to capture paraphrases and terminology variants, and lightweight format and modality rewards that enforce interpretable
reasoning and modality recognition. This multi-signal design provides stable, informative feedback for open-ended outputs where traditional verifiable or MCQ-only rewards fall short. To measure progress, we propose a unified evaluation framework for both text-only and image+text tasks that uses a Reference-based LLM-as-judge in place of brittle string-overlap metrics, capturing semantic correctness,
reasoning, and contextual
alignment. Despite using only $\sim51$K instruction examples, MediX-R1 achieves excellent results across standard medical LLM (text-only) and VLM (image + text)
benchmarks, outperforming strong open-source baselines and delivering particularly large gains on open-ended clinical tasks. Our results demonstrate that open-ended RL with comprehensive reward signals and LLM-based evaluation is a practical path toward reliable medical
reasoning in
multimodal models. Our trained models, curated
datasets and source code are available at https://medix.cvmbzuai.com
Read the full abstract ▾
All authors Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed, Mohamed Zidan, Fahad Khan, Salman Khan, Rao Anwer, Hisham Cholakkal
Computer Vision
VGG-T$^3$: Offline Feed-Forward 3D Reconstruction at Scale
Sven Elflein, Ruilong Li, Sérgio Agostinho et al. (+4 more)
We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and
memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T$^3$ (Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a $1k$ image collection in just $54$ seconds, achieving a $11.6\times$ speed-up over baselines that rely on softmax
attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.
Read the full abstract ▾
All authors Sven Elflein, Ruilong Li, Sérgio Agostinho, Zan Gojcic, Laura Leal-Taixé, Qunjie Zhou, Aljosa Osep
Computer Vision
SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation
Vaibhav Agrawal, Rishubh Parihar, Pradhaan Bhat et al. (+2 more)
We identify occlusion
reasoning as a fundamental yet overlooked aspect for 3D layout-conditioned generation. It is essential for synthesizing partially occluded objects with depth-consistent geometry and scale. While existing methods can generate realistic scenes that follow input layouts, they often fail to model precise inter-object occlusions. We propose SeeThrough3D, a model for 3D layout conditioned generation that explicitly models occlusions. We introduce an occlusion-aware 3D scene representation (OSCR), where objects are depicted as translucent 3D boxes placed within a virtual environment and rendered from desired camera viewpoint. The transparency encodes hidden object regions, enabling the model to reason about occlusions, while the rendered viewpoint provides explicit camera control during generation. We condition a pretrained flow based text-to-image image generation model by introducing a set of visual
tokens derived from our rendered 3D representation. Furthermore, we apply masked self-
attention to accurately bind each object bounding box to its corresponding textual description, enabling accurate generation of multiple objects without object attribute mixing. To train the model, we construct a synthetic
dataset with diverse multi-object scenes with strong inter-object occlusions. SeeThrough3D generalizes effectively to unseen object categories and enables precise 3D layout control with realistic occlusions and consistent camera control.
Read the full abstract ▾
All authors Vaibhav Agrawal, Rishubh Parihar, Pradhaan Bhat, Ravi Kiran Sarvadevabhatla, R. Venkatesh Babu
Computer Vision · Cross-listed from cs.LG
A Dataset is Worth 1 MB
Elad Kimchi Shoshani, Leeyam Gabay and Yedid Hoshen
A
dataset server must often distribute the same large payload to many clients, incurring massive communication costs. Since clients frequently operate on diverse hardware and software frameworks, transmitting a pre-trained model is often infeasible; instead,
agents require raw data to train their own task-specific models locally. While
dataset distillation attempts to compress training signals, current methods struggle to scale to high-resolution data and rarely achieve sufficiently small files. In this paper, we propose Pseudo-Labels as Data (PLADA), a method that completely eliminates pixel transmission. We assume
agents are preloaded with a large, generic, unlabeled reference
dataset (e.g., ImageNet-1K, ImageNet-21K) and communicate a new task by transmitting only the class labels for specific images. To address the distribution mismatch between the reference and target
datasets, we introduce a
pruning mechanism that filters the reference
dataset to retain only the labels of the most semantically relevant images for the target task. This selection process simultaneously maximizes training efficiency and minimizes transmission payload. Experiments on 10 diverse
datasets demonstrate that our approach can transfer task knowledge with a payload of less than 1 MB while retaining high classification accuracy, offering a promising solution for efficient
dataset serving.
Read the full abstract ▾
Computer Vision
Sensor Generalization for Adaptive Sensing in Event-based Object Detection via Joint Distribution Training
Aheli Saha, René Schuster and Didier Stricker
Bio-inspired event cameras have recently attracted significant research due to their asynchronous and low-latency capabilities. These features provide a high dynamic range and significantly reduce motion blur. However, because of the novelty in the nature of their output signals, there is a gap in the variability of available data and a lack of extensive analysis of the parameters characterizing their signals. This paper addresses these issues by providing readers with an in-depth understanding of how intrinsic parameters affect the performance of a model trained on event data, specifically for object detection. We also use our findings to expand the capabilities of the downstream model towards sensor-agnostic robustness.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
Amita Kamath, Jack Hessel, Khyathi Chandu et al. (+3 more)
The lack of
reasoning capabilities in
Vision-
Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of
reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four
reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated
benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of
reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple
languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of
reasoning capabilities.
Read the full abstract ▾
All authors Amita Kamath, Jack Hessel, Khyathi Chandu, Jena D. Hwang, Kai-Wei Chang, Ranjay Krishna
Computer Vision
Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
Tilemachos Aravanis, Vladan Stojnić, Bill Psomas et al. (+2 more)
Open-vocabulary
segmentation (OVS) extends the zero-shot recognition capabilities of
vision-
language models (VLMs) to pixel-level prediction, enabling
segmentation of arbitrary categories specified by text
prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural
language. We address these limitations by introducing a few-shot setting that augments textual
prompts with a support set of pixel-annotated images. Building on this, we propose a
retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized
segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised
segmentation while preserving open-vocabulary ability.
Read the full abstract ▾
All authors Tilemachos Aravanis, Vladan Stojnić, Bill Psomas, Nikos Komodakis, Giorgos Tolias
Computer Vision
ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decoding
Yiran Guan, Sifan Tu, Dingkang Liang et al. (+6 more)
Omni-modal
reasoning is essential for intelligent systems to understand and draw inferences from diverse data sources. While existing omni-modal large
language models (OLLM) excel at perceiving diverse modalities, they lack the complex
reasoning abilities of recent large
reasoning models (LRM). However, enhancing the
reasoning ability of OLLMs through additional training presents significant challenges, including the need for high-quality data, task-specific adaptation, and substantial computational costs. To address these limitations, we propose ThinkOmni, a training-free and data-free framework that lifts textual
reasoning to omni-modal scenarios. ThinkOmni introduces two key components: 1) LRM-as-a-Guide, which leverages off-the-shelf LRMs to guide the OLLM decoding process; 2) Stepwise Contrastive Scaling, which adaptively balances perception and
reasoning signals without manual hyperparameter tuning. Experiments on six multi-modal
reasoning benchmarks demonstrate that ThinkOmni consistently delivers performance improvements, with main results achieving 70.2 on MathVista and 75.5 on MMAU. Overall, ThinkOmni offers a flexible and generalizable solution for omni-modal
reasoning and provides new insights into the generalization and application of
reasoning capabilities. Code is publicly available at https://github.com/1ranGuan/thinkomni
Read the full abstract ▾
All authors Yiran Guan, Sifan Tu, Dingkang Liang, Linghao Zhu, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, Xiang Bai
Computer Vision
PRIMA: Pre-Training with Risk-Integrated Image–Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation
Yiqing Wang, Chunming He, Ziyun Yang et al. (+5 more)
Medical diagnosis requires the effective synthesis of visual manifestations and clinical metadata. However, existing methods often treat metadata as isolated tags, failing to exploit the rich semantic knowledge embedded in clinical descriptions. We propose PRIMA (Pre-training with Risk-integrated Image-Metadata
Alignment), a framework that integrates domain-specific knowledge into multi-modal representation learning. We first curate a corpus of risk–disease correlations via
Retrieval-Augmented Generation (RAG) to refine Clinical ModernBERT,
embedding diagnostic priors into the text encoder. To bridge the modality gap, we introduce a dual-encoder pre-training strategy utilizing DINOv3 and our refined Clinical ModernBERT, optimized by a suite of four complementary loss functions. These losses are designed to capture multi-granular semantic
alignment and handle the ambiguity of clinical correlations through soft labels. Finally, we leverage Qwen3 to fuse these aligned features for precise disease classification. Extensive experiments demonstrate that PRIMA effectively harmonizes pixel-level features with abstract clinical expertise. Across PAD-UFES-20 and AQUA, PRIMA achieves average F1-scores of 72.04% and 85.22%, outperforming strong image-only, metadata-fusion, and medical
vision-
language baselines. Notably, our framework achieves strong performance without requiring massive data collection or exhaustive computational resources. Our code is available at https://github.com/yqwang01/PRIMA.
Read the full abstract ▾
All authors Yiqing Wang, Chunming He, Ziyun Yang, Maria Woodward, Ming-Chen Lu, Mercy Pawar, Leslie Niziol, Sina Farsiu
Computer Vision
Ayush Roy, Wei-Yang Alex Lee, Rudrasis Chakraborty et al. (+1 more)
In recent times, large
datasets hinder efficient model training while also containing redundant concepts.
Dataset distillation aims to synthesize compact
datasets that preserve the knowledge of large-scale training sets while drastically reducing storage and computation. Recent advances in
diffusion models have enabled training-free
distillation by leveraging pre-trained generative priors; however, existing guidance strategies remain limited. Current score-based methods either perform unguided denoising or rely on simple mode-based guidance toward instance prototype centroids (IPC centroids), which often are rudimentary and suboptimal. We propose
Manifold-Guided
Distillation (ManifoldGD), a training-free
diffusion-based framework that integrates
manifold consistent guidance at every denoising timestep. Our method employs IPCs computed via a hierarchical, divisive clustering of VAE latent features, yielding a multi-scale coreset of IPCs that captures both coarse semantic modes and fine intra-class variability. Using a local neighborhood of the extracted IPC centroids, we create the latent
manifold for each
diffusion denoising timestep. At each denoising step, we project the mode-
alignment vector onto the local tangent space of the estimated latent
manifold, thus constraining the generation trajectory to remain
manifold-faithful while preserving semantic consistency. This formulation improves representativeness, diversity, and image fidelity without requiring any model retraining. Empirical results demonstrate consistent gains over existing training-free and training-based baselines in terms of FID, l2 distance among real and synthetic
dataset embeddings, and classification accuracy, establishing ManifoldGD as the first geometry-aware training-free data
distillation framework.
Read the full abstract ▾
All authors Ayush Roy, Wei-Yang Alex Lee, Rudrasis Chakraborty, Vishnu Suresh Lokhande
Computer Vision
Towards Long-Form Spatio-Temporal Video Grounding
Xin Gu, Bing Fan, Jiali Yao et al. (+5 more)
In real scenarios,
videos can span several minutes or even hours. However, existing research on spatio-temporal
video grounding (STVG), given a textual query, mainly focuses on localizing targets in short
videos of tens of seconds, typically less than one minute, which limits real-world applications. In this paper, we explore Long-Form STVG (LF-STVG), which aims to locate targets in long-term
videos. Compared with short
videos, long-term
videos contain much longer temporal spans and more irrelevant information, making it difficult for existing STVG methods that process all frames at once. To address this challenge, we propose an AutoRegressive
Transformer architecture for LF-STVG, termed ART-STVG. Unlike conventional STVG methods that require the entire
video sequence to make predictions at once, ART-STVG treats the
video as streaming input and processes frames sequentially, enabling efficient handling of long
videos. To model spatio-temporal context, we design spatial and temporal
memory banks and apply them to the decoders. Since
memories from different moments are not always relevant to the current frame, we introduce simple yet effective
memory selection strategies to provide more relevant information to the decoders, significantly improving performance. Furthermore, instead of parallel spatial and temporal localization, we propose a cascaded spatio-temporal design that connects the spatial decoder to the temporal decoder, allowing fine-grained spatial cues to assist complex temporal localization in long
videos. Experiments on newly extended LF-STVG
datasets show that ART-STVG significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG. Our code is at: https://github.com/HengLan/ART-STVG.
Read the full abstract ▾
All authors Xin Gu, Bing Fan, Jiali Yao, Zhipeng Zhang, Yan Huang, Cheng Han, Heng Fan, Libo Zhang
Computer Vision
PGVMS: A Prompt-Guided Unified Framework for Virtual Multiplex IHC Staining with Pathological Semantic Learning
Fuqiang Chen, Ranran Zhang, Wanming Hu et al. (+6 more)
Immunohistochemical (IHC) staining enables precise molecular profiling of
protein expression, with over 200 clinically available antibody-based tests in modern pathology. However, comprehensive IHC analysis is frequently limited by insufficient tissue quantities in small biopsies. Therefore, virtual multiplex staining emerges as an innovative solution to digitally transform H&E images into multiple IHC representations, yet current methods still face three critical challenges: (1) inadequate semantic guidance for multi-staining, (2) inconsistent distribution of immunochemistry staining, and (3) spatial misalignment across different stain modalities. To overcome these limitations, we present a
prompt-guided framework for virtual multiplex IHC staining using only uniplex training data (PGVMS). Our framework introduces three key innovations corresponding to each challenge: First, an adaptive
prompt guidance mechanism employing a pathological visual
language model dynamically adjusts staining
prompts to resolve semantic guidance limitations (Challenge 1). Second, our
protein-aware learning strategy (PALS) maintains precise
protein expression patterns by direct quantification and constraint of
protein distributions (Challenge 2). Third, the prototype-consistent learning strategy (PCLS) establishes cross-image semantic interaction to correct spatial misalignments (Challenge 3).
Read the full abstract ▾
All authors Fuqiang Chen, Ranran Zhang, Wanming Hu, Deboch Eyob Abera, Yue Peng, Boyun Zheng, Yiwen Sun, Jing Cai, Wenjian Qin
Computer Vision
LineGraph2Road: Structural Graph Reasoning on Line Graphs for Road Network Extraction
Zhengyang Wei, Renzhi Jing, Yiyi He et al. (+1 more)
Extracting routable road networks from satellite imagery requires accurate
topology recovery beyond pixel-level
segmentation. Recent methods decompose the task into keypoint detection and connectivity prediction, but reliably inferring road connectivity under structural ambiguity remains challenging. Overpasses introduce non-planar crossings that can create false shortcuts, and occlusions break visual continuity, requiring long-range structural
reasoning. We formulate road extraction as fully unobserved connectivity inference and construct a global but sparse Euclidean graph from detected keypoints to better leverage the long-term context. To improve link prediction, we transform this graph into its line graph and perform
reasoning using a Graph
Transformer. We propose an end-to-end pipeline that integrates
vision-based
segmentation, sparse graph construction, and structured inference. Our method explicitly models overpasses and uses
topology-preserving vertex extraction to mitigate routing-critical errors. This approach achieves state-of-the-art performance on City-scale, SpaceNet, and Global-scale
benchmarks in
topology metrics including TOPO-F1 and APLS. In addition to quantitative gains, our method improves reconstruction of multi-level road structures crucial for real-world routing reliability. The code is available at: https://github.com/wzzzzzzy/LineGraph2Road.
Read the full abstract ▾
All authors Zhengyang Wei, Renzhi Jing, Yiyi He, Jenny Suckale
Computer Vision
Decomposing Private Image Generation via Coarse-to-Fine Wavelet Modeling
Jasmine Bayrooti, Weiwei Kong, Natalia Ponomareva et al. (+3 more)
Generative models trained on sensitive image
datasets risk memorizing and reproducing individual training examples, making strong
privacy guarantees essential. While differential
privacy (DP) provides a principled framework for such guarantees, standard DP finetuning (e.g., with DP-SGD) often results in severe degradation of image quality, particularly in high-frequency textures, due to the indiscriminate addition of noise across all model parameters. In this work, we propose a spectral DP framework based on the hypothesis that the most
privacy-sensitive portions of an image are often low-frequency components in the wavelet space (e.g., facial features and object shapes) while high-frequency components are largely generic and public. Based on this hypothesis, we propose the following two-stage framework for DP image generation with coarse image intermediaries: (1) DP finetune an autoregressive spectral image tokenizer model on the low-resolution wavelet coefficients of the sensitive images, and (2) perform high-resolution upsampling using a publicly pretrained super-resolution model. By restricting the
privacy budget to the global structures of the image in the first stage, and leveraging the post-processing property of DP for detail refinement, we achieve promising trade-offs between
privacy and utility. Experiments on the MS-COCO and MM-CelebA-HQ
datasets show that our method generates images with improved quality and style capture relative to other leading DP image frameworks.
Read the full abstract ▾
All authors Jasmine Bayrooti, Weiwei Kong, Natalia Ponomareva, Carlos Esteves, Ameesh Makadia, Amanda Prorok
Computer Vision
Risk-Aware World Model Predictive Control for Generalizable End-to-End Autonomous Driving
Jiangxin Sun, Feng Xue, Teng Long et al. (+4 more)
With advances in imitation learning (IL) and large-scale driving
datasets, end-to-end autonomous driving (E2E-AD) has made great progress recently. Currently, IL-based methods have become a mainstream paradigm: models rely on standard driving behaviors given by experts, and learn to minimize the discrepancy between their actions and expert actions. However, this objective of "only driving like the expert" suffers from limited generalization: when encountering rare or unseen long-tail scenarios outside the distribution of expert demonstrations, models tend to produce unsafe decisions in the absence of prior experience. This raises a fundamental question: Can an E2E-AD system make reliable decisions without any expert action supervision? Motivated by this, we propose a unified framework named Risk-aware World Model Predictive Control (RaWMPC) to address this generalization dilemma through robust control, without reliance on expert demonstrations. Practically, RaWMPC leverages a world model to predict the consequences of multiple candidate actions and selects low-risk actions through explicit risk evaluation. To endow the world model with the ability to predict the outcomes of risky driving behaviors, we design a risk-aware interaction strategy that systematically exposes the world model to hazardous behaviors, making catastrophic outcomes predictable and thus avoidable. Furthermore, to generate low-risk candidate actions at test time, we introduce a self-evaluation
distillation method to distill riskavoidance capabilities from the well-trained world model into a generative action proposal network without any expert demonstration. Extensive experiments show that RaWMPC outperforms state-of-the-art methods in both in-distribution and out-of-distribution scenarios, while providing superior decision interpretability.
Read the full abstract ▾
All authors Jiangxin Sun, Feng Xue, Teng Long, Chang Liu, Jian-Fang Hu, Wei-Shi Zheng, Nicu Sebe
Computer Vision
Spatio-Temporal Token Pruning for Efficient High-Resolution GUI Agents
Zhou Xu, Bowen Zhou, Qi Wang et al. (+2 more)
Pure-
vision GUI
agents provide universal interaction capabilities but suffer from severe efficiency bottlenecks due to the massive spatiotemporal redundancy inherent in high-resolution screenshots and historical trajectories. We identify two critical misalignments in existing compression paradigms: the temporal mismatch, where uniform history encoding diverges from the
agent's "fading
memory"
attention pattern, and the spatial
topology conflict, where unstructured
pruning compromises the grid integrity required for precise coordinate grounding, inducing spatial
hallucinations. To address these challenges, we introduce GUIPruner, a training-free framework tailored for high-resolution GUI
navigation. It synergizes Temporal-Adaptive Resolution (TAR), which eliminates historical redundancy via decay-based resizing, and Stratified Structure-aware
Pruning (SSP), which prioritizes interactive foregrounds and semantic anchors while safeguarding global layout. Extensive evaluations across diverse
benchmarks demonstrate that GUIPruner consistently achieves state-of-the-art performance, effectively preventing the collapse observed in large-scale models under high compression. Notably, on Qwen2-VL-2B, our method delivers a 3.4x reduction in FLOPs and a 3.3x speedup in
vision encoding latency while retaining over 94% of the original performance, enabling real-time, high-precision
navigation with minimal resource consumption.
Read the full abstract ▾
All authors Zhou Xu, Bowen Zhou, Qi Wang, Shuwen Feng, Jingyu Xiao
Computer Vision
Skarimva: Skeleton-based Action Recognition is a Multi-view Application
Daniel Bermuth, Alexander Poeppel and Wolfgang Reif
Human action recognition plays an important role when developing intelligent interactions between humans and machines. While there is a lot of active research on improving the machine learning
algorithms for skeleton-based action recognition, not much
attention has been given to the quality of the input skeleton data itself. This work demonstrates that by making use of multiple camera views to triangulate more accurate 3D~skeletons, the performance of state-of-the-art action recognition models can be improved significantly. This suggests that the quality of the input data is currently a limiting factor for the performance of these models. Based on these results, it is argued that the cost-benefit ratio of using multiple cameras is very favorable in most practical use-cases, therefore future research in skeleton-based action recognition should consider multi-view applications as the standard setup.
Read the full abstract ▾
Computer Vision
Large Multimodal Models as General In-Context Classifiers
Marco Garosi, Matteo Farina, Alessandro Conti et al. (+2 more)
Which
multimodal model should we use for classification? Previous studies suggest that the answer lies in CLIP-like contrastive
Vision-
Language Models (VLMs), due to their remarkable performance in zero-shot classification. In contrast, Large
Multimodal Models (LMM) are more suitable for complex tasks. In this work, we argue that this answer overlooks an important capability of LMMs: in-context learning. We
benchmark state-of-the-art LMMs on diverse
datasets for closed-world classification and find that, although their zero-shot performance is lower than CLIP's, LMMs with a few in-context examples can match or even surpass contrastive VLMs with cache-based adapters, their "in-context" equivalent. We extend this analysis to the open-world setting, where the generative nature of LMMs makes them more suitable for the task. In this challenging scenario, LMMs struggle whenever provided with imperfect context information. To address this issue, we propose CIRCLE, a simple training-free method that assigns pseudo-labels to in-context examples, iteratively refining them with the available context itself. Through extensive experiments, we show that CIRCLE establishes a robust baseline for open-world classification, surpassing VLM counterparts and highlighting the potential of LMMs to serve as unified classifiers, and a flexible alternative to specialized models.
Read the full abstract ▾
All authors Marco Garosi, Matteo Farina, Alessandro Conti, Massimiliano Mancini, Elisa Ricci
Computer Vision
MovieTeller: Tool-augmented Movie Synopsis with ID Consistent Progressive Abstraction
Yizhi Li, Xiaohan Chen, Miao Jiang et al. (+2 more)
With the explosive growth of digital entertainment, automated
video summarization has become indispensable for applications such as content indexing, personalized recommendation, and efficient media archiving. Automatic synopsis generation for long-form
videos, such as movies and TV series, presents a significant challenge for existing
Vision-
Language Models (VLMs). While proficient at single-image captioning, these general-purpose models often exhibit critical failures in long-duration contexts, primarily a lack of ID-consistent character identification and a fractured narrative coherence. To overcome these limitations, we propose MovieTeller, a novel framework for generating movie synopses via tool-augmented progressive abstraction. Our core contribution is a training-free, tool-augmented, fact-grounded generation process. Instead of requiring costly model fine-tuning, our framework directly leverages off-the-shelf models in a plug-and-play manner. We first invoke a specialized face recognition model as an external "tool" to establish Factual Groundings–precise character identities and their corresponding bounding boxes. These groundings are then injected into the
prompt to steer the VLM's
reasoning, ensuring the generated scene descriptions are anchored to verifiable facts. Furthermore, our progressive abstraction pipeline decomposes the summarization of a full-length movie into a multi-stage process, effectively mitigating the context length limitations of current VLMs. Experiments demonstrate that our approach yields significant improvements in factual accuracy, character consistency, and overall narrative coherence compared to end-to-end baselines.
Read the full abstract ▾
All authors Yizhi Li, Xiaohan Chen, Miao Jiang, Wentao Tang, Gaoang Wang
Computer Vision
UniScale: Unified Scale-Aware 3D Reconstruction for Multi-View Understanding via Prior Injection for Robotic Perception
Mohammad Mahdavian, Gordon Tan, Binbin Xu et al. (+3 more)
We present UniScale, a unified, scale-aware multi-view 3D reconstruction framework for robotic applications that flexibly integrates
geometric priors through a modular, semantically informed design. In
vision-based robotic
navigation, the accurate extraction of environmental structure from raw image sequences is critical for downstream tasks. UniScale addresses this challenge with a single feed-forward network that jointly estimates camera intrinsics and extrinsics, scale-invariant depth and point maps, and the metric scale of a scene from multi-view images, while optionally incorporating auxiliary
geometric priors when available. By combining global contextual
reasoning with camera-aware feature representations, UniScale is able to recover the metric-scale of the scene. In robotic settings where camera intrinsics are known, they can be easily incorporated to improve performance, with additional gains obtained when camera poses are also available. This co-design enables robust, metric-aware 3D reconstruction within a single unified model. Importantly, UniScale does not require training from scratch, and leverages world priors exhibited in pre-existing models without
geometric encoding strategies, making it particularly suitable for resource-constrained robotic teams. We evaluate UniScale on multiple
benchmarks, demonstrating strong generalization and consistent performance across diverse environments. We will release our implementation upon acceptance.
Read the full abstract ▾
All authors Mohammad Mahdavian, Gordon Tan, Binbin Xu, Yuan Ren, Dongfeng Bai, Bingbing Liu