Computer Vision
Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions
Kecheng Zhang, Zongxin Yang, Mingfei Han et al. (+6 more)
Visual
agents operating in the wild must respond to queries precisely when sufficient evidence first appears in a
video stream, a critical capability that is overlooked by conventional
video LLMs evaluated in offline settings. The shift to an online, streaming paradigm introduces significant challenges: a lack of decision transparency, the difficulty of aligning response timing with visual evidence, and the need to maintain a global, causally consistent understanding under tight computational budgets. To address these issues, we propose a novel framework that decouples
reasoning control from
memory integration. We introduce \textbf{\model{}}, an instantiation of this framework with two core components. First, the Active Thinking Decision Maker (ATDM) is a transparent
reasoning controller that externalizes its decision process using observable progress ($\boldsymbol{\rho}$) and confidence ($\boldsymbol{c}$) metrics. This allows it to precisely time its response $t_r$ to match the first-sufficient-evidence timestamp $t^\star$ while streaming its
reasoning to the user. Second, the Hierarchical Progressive Semantic Integration (HPSI) module acts as an efficient
memory system. It employs a set of learnable, multi-level aggregation
tokens that are propagated across clips to build a rich, global cognitive state without exceeding
token budgets. %Our approach sets a new standard on key online
video understanding
benchmarks, achieving strong performance of 71.6% on StreamingBench and 46.9% on OVOBench, demonstrating a robust solution for evidence-aligned and transparent online
video analysis. Extensive experiments demonstrate the effectiveness of ATDM and HPSI, e.g., Thinking-QwenVL improves the accuracy of the previous state-of-the-art from 67.63% to 71.60% on the StreamingBench
benchmark.
Read the full abstract ▾
All authors Kecheng Zhang, Zongxin Yang, Mingfei Han, Haihong Hao, Yunzhi Zhuge, Changlin Li, Junhan Zhao, Zhihui Li, Xiaojun Chang
Computer Vision
ESsEN: Training Compact Discriminative Vision-Language Transformers in a Low-Resource Setting
Clayton Fields and Casey Kennington
Vision-
language modeling is rapidly increasing in popularity with an ever expanding list of available models. In most cases, these
vision-
language models have parameters in the tens of billions, which is necessary for some needs, but in many cases smaller models are necessary (e.g., on edge devices or independent robotic platforms). Unfortunately, there is little research in producing light-weight models or in training them with small
datasets. Inspired by the
language learning progression and data sparsity in child development, in this paper, we address both of these goals in a systematic fashion. We show that two-tower encoder models are superior to one-tower encoders in low-resource settings for discriminative English tasks. We show also that incorporating traditional convolutional networks into the two-tower
transformer architecture can help produce parameter efficient
vision-
language models. Finally, we show that the cross-modal fusion module of two-tower encoders can vary significantly in shape and size while producing the same results. In addition, we present ESsEN, a compact
vision-
language model that can be trained end-to-end with relatively few resources that performs as well on several tasks with only a fraction of the parameters compared to other models. The experimental results and the tools we present here make
vision-
language modeling more accessible to a wider variety of researchers.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification
Florian Kittler, Sheethal Bhat and Andreas Maier
Zero-shot
vision-
language models (VLMs) have shown promise for chest radiograph classification, but their performance is often limited by confounding label co-occurrence, long-tail class imbalance, and transfer instability under domain shift. We propose ProtoCLIP, a refinement strategy for CLIP-style VLMs that improves zero-shot discrimination through targeted data curation and distilled anchor
alignment. Specifically, we construct pathology-focused training subsets with curated negative samples to reduce co-occurrence bias. We also introduce a representation-preserving
distillation objective to stabilize adaptation while maintaining semantic structure and improving discrimination of clinically relevant co-occurring pathologies. Evaluated on an unseen
dataset VinDr-CXR, ProtoCLIP improves AUC by 2-10 percentage points over a strong CLIP-based baseline across multiple findings. For pneumothorax specifically, ProtoCLIP achieves a state-of-the-art AUC of 0.94. These results demonstrate that anchor-guided refinement, coupled with curated supervision and controlled adaptation, can mitigate common zero-shot transfer failures in medical VLMs without requiring large-scale retraining.
Read the full abstract ▾
Computer Vision
Revisiting Change VQA in Remote Sensing with Structured and Native Multimodal Qwen Models
Yakoub Bazi, Mohamad M. Al Rahhal, Mansour Zuair et al. (+1 more)
Change visual question answering (Change VQA) addresses the problem of answering natural-
language questions about semantic changes between bi-temporal remote sensing (RS) images. Although
vision-
language models (VLMs) have recently been studied for temporal RS image understanding, Change VQA remains underexplored in the context of modern
multimodal models. In this letter, we revisit the CDVQA
benchmark using recent Qwen models under a unified low-rank adaptation (LoRA) setting. We compare Qwen3-VL, which follows a structured
vision-
language pipeline with multi-depth visual conditioning and a full-
attention decoder, with Qwen3.5, a native
multimodal model that combines a single-stage
alignment with a hybrid decoder backbone. Experimental results on the official CDVQA test splits show that recent VLMs improve over earlier specialized baselines. They further show that performance does not scale monotonically with model size, and that native
multimodal models are more effective than structured
vision-
language pipelines for this task. These findings indicate that tightly integrated
multimodal backbones contribute more to performance than scale or explicit multi-depth visual conditioning for
language-driven semantic change
reasoning in RS imagery.
Read the full abstract ▾
All authors Yakoub Bazi, Mohamad M. Al Rahhal, Mansour Zuair, Faroun Mohamed
Computer Vision
MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline
Jiyao Liu, Jianghan Shen, Sida Song et al. (+19 more)
Recent advances in deep research systems enable large
language models to retrieve, synthesize, and reason over large-scale external knowledge. In medicine, developing clinical guidelines critically depends on such deep evidence integration. However, existing
benchmarks fail to evaluate this capability in realistic workflows requiring multi-step evidence integration and expert-level judgment. To address this gap, we introduce MedProbeBench, the first
benchmark leveraging high-quality clinical guidelines as expert-level references. Medical guidelines, with their rigorous standards in neutrality and verifiability, represent the pinnacle of medical expertise and pose substantial challenges for deep research
agents. For evaluation, we propose MedProbe-Eval, a comprehensive evaluation framework featuring: (1) Holistic Rubrics with 1,200+ task-adaptive rubric criteria for comprehensive quality assessment, and (2) Fine-grained Evidence Verification for rigorous validation of evidence precision, grounded in 5,130+ atomic claims. Evaluation of 17 LLMs and deep research
agents reveals critical gaps in evidence integration and guideline generation, underscoring the substantial distance between current capabilities and expert-level clinical guideline development. Project: https://github.com/uni-medical/MedProbeBench
Read the full abstract ▾
All authors Jiyao Liu, Jianghan Shen, Sida Song, Tianbin Li, Xiaojia Liu, Rongbin Li, Ziyan Huang, Jiashi Lin, Junzhi Ning, Changkai Ji, Siqi Luo, Wenjie Li, Chenglong Ma, Ming Hu, Jing Xiong, Jin Ye, Bin Fu, Ningsheng Xu, Yirong Chen, Lei Jin, Hong Chen, Junjun He
Computer Vision
One-Step Diffusion with Inverse Residual Fields for Unsupervised Industrial Anomaly Detection
Boan Zhang, Wen Li, Guanhua Yu et al. (+3 more)
Diffusion models have achieved outstanding performance in unsupervised industrial anomaly detection (uIAD) by learning a
manifold of normal data under the common assumption that off-
manifold anomalies are harder to generate, resulting in larger reconstruction errors in data space or lower probability densities in the tractable latent space. However, their iterative denoising and noising nature leads to slow inference. In this paper, we propose OSD-IRF, a novel one-step
diffusion with inverse residual fields, to address this limitation for uIAD task. We first train a deep
diffusion probabilistic model (DDPM) on normal data without any conditioning. Then, for a test sample, we predict its inverse residual fields (IRF) based on the noise estimated by the well-trained parametric noise function of the DDPM. Finally, uIAD is performed by evaluating the probability density of the IRF under a Gaussian distribution and comparing it with a threshold. Our key observation is that anomalies become distinguishable in this IRF space, a finding that has seldom been reported in prior works. Moreover, OSD-IRF requires only single step
diffusion for uIAD, thanks to the property that IRF holds for any neighboring time step in the denoising process. Extensive experiments on three widely used uIAD
benchmarks show that our model achieves SOTA or competitive performance across six metrics, along with roughly a 2X inference speedup without
distillation.
Read the full abstract ▾
All authors Boan Zhang, Wen Li, Guanhua Yu, Xiyang Liu, Wenchao Chen, Long Tian
Computer Vision
Towards Robust Text-to-Image Person Retrieval: Multi-View Reformulation for Semantic Compensation
Chao Yuan, Yujian Zhao, Haoxuan Xu et al. (+1 more)
In text-to-image person
retrieval tasks, the diversity of natural
language expressions and the implicitness of visual semantics often lead to the problem of Expression Drift, where semantically equivalent texts exhibit significant feature discrepancies in the
embedding space due to phrasing variations, thereby degrading the robustness of image-text
alignment. This paper proposes a semantic compensation framework (MVR) driven by Large
Language Models (LLMs), which enhances cross-modal representation consistency through multi-view semantic reformulation and feature compensation. The core methodology comprises three components: Multi-View Reformulation (MVR): A dual-branch prompting strategy combines key feature guidance (extracting visually critical components via feature similarity) and diversity-aware rewriting to generate semantically equivalent yet distributionally diverse textual variants; Textual Feature Robustness Enhancement: A training-free latent space compensation mechanism suppresses noise interference through multi-view feature mean-pooling and residual connections, effectively capturing "Semantic Echoes"; Visual Semantic Compensation: VLM generates multi-perspective image descriptions, which are further enhanced through shared text reformulation to address visual semantic gaps. Experiments demonstrate that our method can improve the accuracy of the original model well without training and performs SOTA on three text-to-image person
retrieval datasets.
Read the full abstract ▾
All authors Chao Yuan, Yujian Zhao, Haoxuan Xu, Guanglin Niu
Computer Vision
DSA-CycleGAN: A Domain Shift Aware CycleGAN for Robust Multi-Stain Glomeruli Segmentation
Zeeshan Nisar, Friedrich Feuerhake and Thomas Lampert
A key challenge in
segmentation in digital histopathology is inter- and intra-stain variations as it reduces model performance. Labelling each stain is expensive and time-consuming so methods using stain transfer via CycleGAN, have been developed for training multi-stain
segmentation models using labels from a single stain. Nevertheless, CycleGAN tends to introduce noise during translation because of the one-to-many nature of some stain pairs, which conflicts with its cycle consistency loss. To address this, we propose the Domain Shift Aware CycleGAN, which reduces the presence of such noise. Furthermore, we evaluate several advances from the field of machine learning aimed at resolving similar problems and compare their effectiveness against DSA-CycleGAN in the context of multi-stain glomeruli
segmentation. Experiments demonstrate that DSA-CycleGAN not only improves
segmentation performance in glomeruli
segmentation but also outperforms other methods in reducing noise. This is particularly evident when translating between biologically distinct stains. The code is publicly available at https://github.com/zeeshannisar/DSA-CycleGAN.
Read the full abstract ▾
Computer Vision
EAST: Early Action Prediction Sampling Strategy with Token Masking
Iva Sović, Ivan Martinović and Marin Oršić
Early action prediction seeks to anticipate an action before it fully unfolds, but limited visual evidence makes this task especially challenging. We introduce EAST, a simple and efficient framework that enables a model to reason about incomplete observations. In our empirical study, we identify key components when training early action prediction models. Our key contribution is a randomized training strategy that samples a time step separating observed and unobserved
video frames, enabling a single model to generalize seamlessly across all test-time observation ratios. We further show that joint learning on both observed and future (oracle) representations significantly boosts performance, even allowing an encoder-only model to excel. To improve scalability, we propose a
token masking procedure that cuts
memory usage in half and accelerates training by 2x with negligible accuracy loss. Combined with a
forecasting decoder, EAST sets a new state of the art on NTU60, SSv2, and UCF101, surpassing previous best work by 10.1, 7.7, and 3.9 percentage points, respectively.
Read the full abstract ▾
Computer Vision
LBFTI: Layer-Based Facial Template Inversion for Identity-Preserving Fine-Grained Face Reconstruction
Zixuan Shen, Zhihua Xia, Kaikai Gan et al. (+1 more)
In face recognition systems, facial templates are widely adopted for identity authentication due to their compliance with the data minimization principle. However, facial template inversion technologies have posed a severe
privacy leakage risk by enabling face reconstruction from templates. This paper proposes a Layer-Based Facial Template Inversion (LBFTI) method to reconstruct identity-preserving fine-grained face images. Our scheme decomposes face images into three layers: foreground layers (including eyebrows, eyes, nose, and mouth), midground layers (skin), and background layers (other parts). LBFTI leverages dedicated generators to produce these layers, adopting a rigorous three-stage training strategy: (1) independent refined generation of foreground and midground layers, (2) fusion of foreground and midground layers with template secondary injection to produce complete panoramic face images with background layers, and (3) joint fine-tuning of all modules to optimize inter-layer coordination and identity consistency. Experiments demonstrate that our LBFTI not only outperforms state-of-the-art methods in machine authentication performance, with a 25.3% improvement in TAR, but also achieves better similarity in human perception, as validated by both quantitative metrics and a questionnaire survey.
Read the full abstract ▾
All authors Zixuan Shen, Zhihua Xia, Kaikai Gan, Peipeng Yu
Computer Vision
AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation
Haoyue Tan, Shengnan Wang, Yulin Qiao et al. (+5 more)
Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic
attention complexity. Existing sparse
attention methods either overlook semantic similarity or fail to adapt to heterogeneous
token distributions across layers, leading to model performance degradation. We propose AdaCluster, a training-free adaptive clustering framework that accelerates the generation of DiTs while preserving accuracy. AdaCluster applies an angle-similarity-preserving clustering method to query vectors for higher compression, and designs a euclidean-similarity-preserving clustering method for keys, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection. Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 on one A40 GPU demonstrate up to 1.67-4.31x speedup with negligible quality degradation.
Read the full abstract ▾
All authors Haoyue Tan, Shengnan Wang, Yulin Qiao, Juncheng Zhang, Youhui Bai, Ping Gong, Zewen Jin, Cheng Li
Computer Vision · Cross-listed from cs.RO
Enhancing Glass Surface Reconstruction via Depth Prior for Robot Navigation
Jiamin Zheng, Jingwen Yu, Guangcheng Chen et al. (+1 more)
Indoor
robot navigation is often compromised by glass surfaces, which severely corrupt depth sensor measurements. While foundation models like Depth Anything 3 provide excellent
geometric priors, they lack an absolute metric scale. We propose a training-free framework that leverages depth foundation models as a structural prior, employing a robust local RANSAC-based
alignment to fuse it with raw sensor depth. This naturally avoids contamination from erroneous glass measurements and recovers an accurate metric scale. Furthermore, we introduce \ti{GlassRecon}, a novel RGB-D
dataset with geometrically derived ground truth for glass regions. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art baselines, especially under severe sensor depth corruption. The
dataset and related code will be released at https://github.com/jarvisyjw/GlassRecon.
Read the full abstract ▾
All authors Jiamin Zheng, Jingwen Yu, Guangcheng Chen, Hong Zhang
Computer Vision
OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation
Lei Zhu, Xing Cai, Yingjie Chen et al. (+6 more)
Recent advancements in audio-
video joint generation models have demonstrated impressive capabilities in content creation. However, generating high-fidelity human-centric
videos in complex, real-world physical scenes remains a significant challenge. We identify that the root cause lies in the structural deficiencies of existing
datasets across three dimensions: limited global scene and camera diversity, sparse interaction modeling (both person-person and person-object), and insufficient individual attribute
alignment. To bridge these gaps, we present OmniHuman, a large-scale, multi-scene
dataset designed for fine-grained human modeling. OmniHuman provides a hierarchical annotation covering
video-level scenes, frame-level interactions, and individual-level attributes. To facilitate this, we develop a fully automated pipeline for high-quality data collection and multi-modal annotation. Complementary to the
dataset, we establish the OmniHuman
Benchmark (OHBench), a three-level evaluation system that provides a scientific diagnosis for human-centric audio-
video synthesis. Crucially, OHBench introduces metrics that are highly consistent with human perception, filling the gaps in existing
benchmarks by providing a comprehensive diagnosis across global scenes, relational interactions, and individual attributes.
Read the full abstract ▾
All authors Lei Zhu, Xing Cai, Yingjie Chen, Yiheng Li, Binxin Yang, Hao Liu, Jie Chen, Chen Li, Jing LYu
Computer Vision
EVE: Verifiable Self-Evolution of MLLMs via Executable Visual Transformations
Yongrui Heng, Chaoya Jiang, Han Yang et al. (+2 more)
Self-
evolution of
multimodal large
language models (MLLMs) remains a critical challenge: pseudo-label-based methods suffer from progressive quality degradation as model predictions drift, while template-based methods are confined to a static set of transformations that cannot adapt in difficulty or diversity. We contend that robust, continuous self-improvement requires not only deterministic external feedback independent of the model's internal certainty, but also a mechanism to perpetually diversify the training distribution. To this end, we introduce EVE (Executable Visual transformation-based self-
Evolution), a novel framework that entirely bypasses pseudo-labels by harnessing executable visual transformations continuously enriched in both variety and complexity. EVE adopts a Challenger-Solver dual-policy architecture. The Challenger maintains and progressively expands a queue of visual transformation code examples, from which it synthesizes novel Python scripts to perform dynamic visual transformations. Executing these scripts yields VQA problems with absolute, execution-verified ground-truth answers, eliminating any reliance on model-generated supervision. A multi-dimensional reward system integrating semantic diversity and dynamic difficulty calibration steers the Challenger to enrich its code example queue while posing progressively more challenging tasks, preventing mode collapse and fostering reciprocal co-
evolution between the two policies. Extensive experiments demonstrate that EVE consistently surpasses existing self-
evolution methods, establishing a robust and scalable paradigm for verifiable MLLM self-
evolution. The code is available at https://github.com/0001Henry/EVE .
Read the full abstract ▾
All authors Yongrui Heng, Chaoya Jiang, Han Yang, Shikun Zhang, Wei Ye
Computer Vision
Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection
Sa Zhu, Wanqian Zhang, Lin Wang et al. (+3 more)
Open-Vocabulary Temporal Action Detection (OV-TAD) aims to localize and classify action segments of unseen categories in untrimmed
videos, where effective
alignment between action semantics and
video representations is critical for accurate detection. However, existing methods struggle to mitigate the semantic imbalance between concise, abstract action labels and rich, complex
video contents, inevitably introducing semantic noise and misleading cross-modal
alignment. To address this challenge, we propose DFAlign, the first framework that leverages
diffusion-based denoising to generate foreground knowledge for the guidance of action-
video alignment. Following the 'conditioning, denoising and aligning' manner, we first introduce the Semantic-Unify Conditioning (SUC) module, which unifies action-shared and action-specific semantics as conditions for
diffusion denoising. Then, the Background-Suppress Denoising (BSD) module generates foreground knowledge by progressively removing background redundancy from
videos through denoising process. This foreground knowledge serves as effective intermediate semantic anchor between
video and text representations, mitigating the semantic gap and enhancing the discriminability of action-relevant segments. Furthermore, we introduce the Foreground-
Prompt Alignment (FPA) module to inject extracted foreground knowledge as
prompt tokens into text representations, guiding model's
attention towards action-relevant segments and enabling precise cross-modal
alignment. Extensive experiments demonstrate that our method achieves state-of-the-art performance on two OV-TAD
benchmarks. The code repository is provided as follows: https://anonymous.4open.science/r/Code-2114/.
Read the full abstract ▾
All authors Sa Zhu, Wanqian Zhang, Lin Wang, Jinchao Zhang, Cong Wang, Bo Li
Computer Vision · Cross-listed from cs.RO
Relative State Estimation using Event-Based Propeller Sensing
Ravi Kumar Thakur, Luis Granados Segura, Jan Klivan et al. (+4 more)
Autonomous swarms of multi-Unmanned Aerial Vehicle (
UAV) system requires an accurate and fast relative state estimation. Although
monocular frame-based camera methods perform well in ideal conditions, they are slow, suffer scale ambiguity, and often struggle in visually challenging conditions. The advent of event cameras addresses these challenging tasks by providing low latency, high dynamic range, and microsecond-level temporal resolution. This paper proposes a framework for relative state estimation for quadrotors using event-based propeller sensing. The propellers in the event stream are tracked by detection to extract the region-of-interests. The event streams in these regions are processed in temporal chunks to estimate per-propeller frequencies. These frequency measurements drive a kinematic state estimation module as a thrust input, while camera-derived position measurements provide the update step. Additionally, we use
geometric primitives derived from event streams to estimate the orientation of the quadrotor by fitting an ellipse over a propeller and backprojecting it to recover body-frame tilt-axis. The existing event-based approaches for quadrotor state estimation use the propeller frequency in simulated flight sequences. Our approach estimates the propeller frequency under 3% error on a test
dataset of five real-world outdoor flight sequences, providing a method for decentralized relative localization for multi-
robot systems using event camera.
Read the full abstract ▾
All authors Ravi Kumar Thakur, Luis Granados Segura, Jan Klivan, Radim Špetlík, Tobiáš Vinklárek, Matouš Vrba, Martin Saska
Computer Vision
Spike-NVPT: Learning Robust Visual Prompts via Bio-Inspired Temporal Filtering and Discretization
Qiugang Zhan, Anning Jiang, Ran Tao et al. (+4 more)
Pre-trained
vision models have found widespread application across diverse domains.
Prompt tuning-based methods have emerged as a parameter-efficient paradigm for adapting pre-trained
vision models. While effective on standard
benchmarks, the continuous and dense nature of learned
prompts can lead to sensitivity against input noise, as the high-capacity
prompts tend to overfit task-irrelevant details. To address this trade-off, we propose Spike-NVPT, a noise-robust visual
prompt tuning method. Specifically, we design a Signal Filtering Layer based on spiking
neurons, which uses the integrate-and-fire (IF) mechanism to accumulate task-relevant signals over time and filter transient noise fluctuations. A subsequent Spike Discretization Unit converts filtered signals into sparse binary
prompts. This discretization acts as a strong regularizer, forcing the model to anchor decision boundaries on the most discriminative and robust features. Notably, the resulting binary
prompts remain static during deployment, ensuring zero additional computational overhead during inference. Experimental results demonstrate that Spike-NVPT achieves superior robustness performance, with a maximum improvement of 11.2% over conventional methods, and retains competitive accuracy on clean
datasets. To the best of our knowledge, this is the first attempt to leverage spiking
neurons for fine-tuning traditional artificial neural network (ANN)-based visual models.
Read the full abstract ▾
All authors Qiugang Zhan, Anning Jiang, Ran Tao, Ao Ma, Xiangyu Zhang, Xiurui Xie, Guisong Liu
Computer Vision
LiquidTAD: Efficient Temporal Action Detection via Parallel Liquid-Inspired Temporal Relaxation
Zepeng Sun, Naichuan Zheng, Hailun Xia et al. (+3 more)
Temporal Action Detection (TAD) requires precise localization of action boundaries within long, untrimmed
video sequences. While current high-performing methods achieve strong accuracy, they are often characterized by excessive parameter counts, substantial computational overhead, and a reliance on specialized operators that hinder deployment across diverse hardware platforms. This paper presents LiquidTAD, a framework that distills the exponential relaxation prior of
liquid neural dynamics into a parallel temporal operator, rather than reproducing full
Liquid Neural Network (LNN) dynamics. By introducing a Parallel
Liquid-inspired Relaxation mechanism, sequential ODE solving is avoided through a fully vectorized, non-recursive formulation built entirely upon standard neural operations, enabling hardware-agnostic deployment with linear complexity with respect to the temporal length. A complementary Hierarchical Decay-Rate Sharing Strategy further adapts this relaxation prior across feature pyramid levels, stabilizing optimization and implicitly compensating for temporal compression in deeper layers. Experimental evaluations on THUMOS-14 and ActivityNet-1.3 demonstrate that LiquidTAD achieves accuracy competitive with strong baselines while substantially lowering the model footprint. Specifically, on THUMOS-14, LiquidTAD achieves 69.46% average mAP with only 10.82M parameters and 27.17G FLOPs, reducing the parameter count by over 60% compared with ActionFormer.
Read the full abstract ▾
All authors Zepeng Sun, Naichuan Zheng, Hailun Xia, Junjie Wu, Liwei Bao, Xiaotai Zhang
Computer Vision
MARCO: Navigating the Unseen Space of Semantic Correspondence
Claudia Cuttano, Gabriele Trivigno, Carlo Masone et al. (+1 more)
Recent advances in semantic correspondence rely on dual-encoder architectures, combining DINOv2 with
diffusion backbones. While accurate, these billion-parameter models generalize poorly beyond training keypoints, revealing a gap between
benchmark performance and real-world usability, where queried points rarely match those seen during training. Building upon DINOv2, we introduce MARCO, a unified model for generalizable correspondence driven by a novel training framework that enhances both fine-grained localization and semantic generalization. By coupling a coarse-to-fine objective that refines spatial precision with a self-
distillation framework, which expands sparse supervision beyond annotated regions, our approach transforms a handful of keypoints into dense, semantically coherent correspondences. MARCO sets a new state of the art on SPair-71k, AP-10K, and PF-PASCAL, with gains that amplify at fine-grained localization thresholds (+8.9 PCK@0.01), strongest generalization to unseen keypoints (+5.1, SPair-U) and categories (+4.7, MP-100), while remaining 3x smaller and 10x faster than
diffusion-based approaches. Code is available at https://github.com/visinf/MARCO .
Read the full abstract ▾
All authors Claudia Cuttano, Gabriele Trivigno, Carlo Masone, Stefan Roth
Computer Vision
Han Li, Zehao Huang, Jiahui Fu et al. (+2 more)
Multimodal large
language models have demonstrated remarkable capabilities in 2D
vision, motivating their extension to 3D scene understanding. Recent studies represent 3D scenes as 3D spatial
videos composed of image sequences with depth and camera pose information, enabling pre-trained
video-
language models to perform 3D
reasoning tasks. However, the large number of visual
tokens in spatial
videos remains a major bottleneck for efficient inference and context management. Existing
pruning methods overlook the view consistency of spatial
videos and the spatial diversity of the remaining
tokens, which prevents them from effectively removing inter-frame redundancy and preserving scene completeness. In this paper, we propose Geo3DPruner, a Geometry-Guided 3D Visual
Token Pruning framework. Geo3DPruner first models cross-frame relevance through geometry-aware global
attention, and then performs a two-stage
pruning process. The intra-voxel stage selects representative multi-view features within each voxel, while the inter-voxel stage preserves spatial diversity by selecting a globally distributed subset of voxels. Extensive experiments on multiple 3D scene understanding
benchmarks demonstrate that Geo3DPruner retains over 90% of the original performance while
pruning 90% of visual
tokens, significantly outperforming existing text-guided and
vision-guided
pruning methods.
Read the full abstract ▾
All authors Han Li, Zehao Huang, Jiahui Fu, Naiyan Wang, Si Liu
Computer Vision
Long-Text-to-Image Generation via Compositional Prompt Decomposition
Jen-Yuan Huang, Tong Lin and Yilun Du
While modern text-to-image (T2I) models excel at generating images from intricate
prompts, they struggle to capture the key details when the inputs are descriptive paragraphs. This limitation stems from the prevalence of concise captions that shape their training distributions. Existing methods attempt to bridge this gap by either fine-tuning T2I models on long
prompts, which generalizes poorly to longer lengths; or by projecting the oversize inputs into normal-
prompt space and compromising fidelity. We propose
Prompt Refraction for Intricate Scene Modeling (PRISM), a compositional approach that enables pre-trained T2I models to process long sequence inputs. PRISM uses a lightweight module to extract constituent representations from the long
prompts. The T2I model makes independent noise predictions for each component, and their outputs are merged into a single denoising step using energy-based conjunction. We evaluate PRISM across a wide range of model architectures, showing comparable performances to models fine-tuned on the same training data. Furthermore, PRISM demonstrates superior generalization, outperforming baseline models by 7.4% on
prompts over 500
tokens in a challenging public
benchmark.
Read the full abstract ▾
Computer Vision
Domain-Specialized Object Detection via Model-Level Mixtures of Experts
Svetlana Pavlitska, Malte Stüven, Beyza Keskin et al. (+1 more)
Mixture-of-Experts (MoE) models provide a structured approach to combining specialized neural networks and offer greater interpretability than conventional ensembles. While MoEs have been successfully applied to image classification and semantic
segmentation, their use in object detection remains limited due to challenges in merging dense and structured predictions. In this work, we investigate model-level mixtures of object detectors and analyze their suitability for improving performance and interpretability in object detection. We propose an MoE architecture that combines YOLO-based detectors trained on semantically disjoint data subsets, with a learned gating network that dynamically weights expert contributions. We study different strategies for fusing detection outputs and for training the gating mechanism, including balancing losses to prevent expert collapse. Experiments on the BDD100K
dataset demonstrate that the proposed MoE consistently outperforms standard ensemble approaches and provides insights into expert specialization across domains, highlighting model-level MoEs as a viable alternative to traditional ensembling for object detection. Our code is available at https://github.com/KASTEL-MobilityLab/mixtures-of-experts/.
Read the full abstract ▾
All authors Svetlana Pavlitska, Malte Stüven, Beyza Keskin, J. Marius Zöllner
Computer Vision
Style-Based Neural Architectures for Real-Time Weather Classification
Hamed Ouattara, Pascal Houssam Salmane, Pierre Duthon et al. (+2 more)
In this paper, we present three neural network architectures designed for real-time classification of weather conditions (sunny, rain, snow, fog) from images. These models, inspired by recent advances in style transfer, aim to capture the stylistic elements present in images. One model, called "Multi-PatchGAN", is based on PatchGANs used in well-known architectures such as Pix2Pix and CycleGAN, but here adapted with multiple patch sizes for detection tasks. The second model, "Truncated ResNet50", is a simplified version of ResNet50 retaining only its first nine layers. This truncation, determined by an evolutionary
algorithm, facilitates the extraction of high-frequency features essential for capturing subtle stylistic details. Finally, we propose "Truncated ResNet50 with Gram
Matrix and
Attention", which computes Gram
matrices for each layer during training and automatically weights them via an
attention mechanism, thus optimizing the extraction of the most relevant stylistic expressions for classification. These last two models outperform the state of the art and demonstrate remarkable generalization capability on several public databases. Although developed for weather detection, these architectures are also suitable for other appearance-based classification tasks, such as animal species recognition, texture classification, disease detection in medical imaging, or industrial defect identification.
Read the full abstract ▾
All authors Hamed Ouattara, Pascal Houssam Salmane, Pierre Duthon, Frédéric Bernardin, Omar Ait Aider
Computer Vision
Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuning
Xixi Liu, Jorge Lazo, Andreas Hallqvist et al. (+8 more)
Accurate prognostication and risk estimation are essential for guiding clinical decision-making and optimizing patient management. While radiologist-assessed features from CT scans provide valuable indicators of disease severity and outcomes, interpreting such images requires expert knowledge, and translating rich visual information into textual summaries inevitably leads to information loss. In this work, we propose a
vision-
language framework for 3D CT image understanding that leverages large-scale open-sourced CT images paired with radiology reports through visual instruction tuning. This pre-training enables the model to learn clinically meaningful visual-textual representations, which can then be adapted to downstream survival prediction tasks. By incorporating a survival prediction head on top of the pre-trained model, our approach improves survival prediction from CT images and clinical data while generating clinically meaningful
language responses to predefined questions. Experimental results demonstrate that our method outperforms baseline methods in survival prediction, particularly, when clinical data alone is less predictive. The code will be released upon acceptance.
Read the full abstract ▾
All authors Xixi Liu, Jorge Lazo, Andreas Hallqvist, Mikael Johansson, Åse Johnsson, Jonas S Andersson, Ella Äng Eklund, Patrik Sund, Nasser Hosseini, Jennifer Alvén, Ida Häggström
Computer Vision
Is SAM3 ready for pathology segmentation?
Qiuyu Kong, Shakiba Sharifi, Yiming Wang et al. (+2 more)
Is Segment Anything Model 3 (SAM3) capable in segmenting Any Pathology Images? Digital pathology
segmentation spans tissue-level and
nuclei-level scales, where traditional methods often suffer from high annotation costs and poor generalization. SAM3 introduces Promptable Concept
Segmentation, offering a potential automated interface via text
prompts. With this work, we propose a systematic evaluation protocol to explore the capability space of SAM3 in a structured manner. Specifically, we evaluate SAM3 under different supervision settings including zero-shot, few-shot, and supervised with varying prompting strategies. Our extensive evaluation on pathological
datasets including NuInsSeg, PanNuke and GlaS, reveals that: (1) text-only
prompts poorly activate
nuclear concepts; (2) performance is highly sensitive to visual
prompt types and budgets; (3) few-shot learning offers gains, but SAM3 lacks robustness against visual
prompt noise; and (4) a significant gap persists between
prompt-based usage and task-trained adapter-based reference. Our study delineates SAM3's boundaries in pathology image
segmentation and provides practical guidance on the necessity of pathology domain adaptation.
Read the full abstract ▾
All authors Qiuyu Kong, Shakiba Sharifi, Yiming Wang, Marco Cristani, Zanxi Ruan
Computer Vision
Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation
Zhen Liu, Yuhan Liu, Jinjun Wang et al. (+3 more)
Vision-and-
Language Navigation requires
agents to follow natural-
language instructions in visually changing environments. A central challenge is the dynamic
entanglement between
language and observations: the meaning of instruction shifts as the
agent's field of view and spatial context evolve. However, many existing models encode the instruction as a static global representation, limiting their ability to adapt instruction meaning to the current visual context. We therefore model instruction understanding as an Instruction-as-State variable: a decision-relevant,
token-level instruction state that evolves step by step conditioned on the
agent's perceptual state, where the perceptual state denotes the observation-grounded
navigation context at each step. To realize this principle, we introduce State-Entangled Environment-Guided Instruction Understanding (S-EGIU), a coarse-to-fine framework for state-conditioned segment activation and
token-level semantic refinement. At the coarse level, S-EGIU activates the instruction segment whose semantics align with the current observation. At the fine level, it refines the activated segment through observation-guided
token grounding and contextual modeling, sharpening its internal semantics under the current observation. Together, these stages maintain an instruction state that is continuously updated according to the
agent's perceptual state during
navigation. S-EGIU delivers strong performance on several key metrics, including a +2.68% SPL gain on REVERIE Test Unseen, and demonstrates consistent efficiency gains across multiple VLN
benchmarks, underscoring the value of dynamic instruction–perception
entanglement.
Read the full abstract ▾
All authors Zhen Liu, Yuhan Liu, Jinjun Wang, Jianyi Liu, Wei Song, Jingwen Fu
Computer Vision
Memorize When Needed: Decoupled Memory Control for Spatially Consistent Long-Horizon Video Generation
Yanjun Guo, Zhengqiang Zhang, Pengfei Wang et al. (+3 more)
Spatially consistent long-horizon
video generation aims to maintain temporal and spatial consistency along predefined camera trajectories. Existing methods mostly entangle
memory modeling with
video generation, leading to inconsistent content during scene revisits and diminished generative capacity when exploring novel regions, even when trained on extensive annotated data. To address these limitations, we propose a decoupled framework that separates
memory conditioning from generation. Our approach significantly reduces training costs while simultaneously enhancing spatial consistency and preserving the generative capacity for novel scene exploration. Specifically, we employ a lightweight, independent
memory branch to learn precise spatial consistency from historical observations. We first introduce a hybrid
memory representation to capture complementary temporal and spatial cues from generated frames, then leverage a per-frame cross-
attention mechanism to ensure each frame is conditioned exclusively on the most spatially relevant historical information, which is injected into the generative model to ensure spatial consistency. When generating new scenes, a camera-aware gating mechanism is proposed to mediate the interaction between
memory and generation modules, enabling
memory conditioning only when meaningful historical references exist. Compared with the existing method, our method is highly data-efficient, yet the experiments demonstrate that our approach achieves state-of-the-art performance in terms of both visual quality and spatial consistency. Project page can be found at https://iguoyanjun.github.io/memorize-when-needed-project-page/.
Read the full abstract ▾
All authors Yanjun Guo, Zhengqiang Zhang, Pengfei Wang, Xinyue Liang, Zhiyuan Ma, Lei Zhang
Computer Vision
Towards Symmetry-sensitive Pose Estimation: A Rotation Representation for Symmetric Object Classes
Andreas Kriegler, Csaba Beleznai and Margrit Gelautz
Symmetric objects are common in daily life and industry, yet their inherent orientation ambiguities that impede the training of deep learning networks for pose estimation are rarely discussed in the literature. To cope with these ambiguities, existing solutions typically require the design of specific loss functions and network architectures or resort to
symmetry-invariant evaluation metrics. In contrast, we focus on the numeric representation of the rotation itself, modifying trigonometric identities with the degrees of
symmetry derived from the objects' shapes. We use our representation, SARR, to obtain canonic (
symmetry-resolved) poses for the symmetric objects in two popular 6D pose estimation
datasets, T-LESS and ITODD, where SARR is unique and continuous w.r.t. the visual appearance. This allows us to use a standard CNN for 3D orientation estimation whose performance is evaluated with the
symmetry-sensitive cosine distance $\text{AR}_{\text{C}}$. Our networks outperform the state of the art using $\text{AR}_{\text{C}}$ and achieve satisfactory performance when using conventional
symmetry-invariant measures. Our method does not require any 3D models but only depth, or, as part of an additional experiment, texture-less RGB/grayscale images as input. We also show that networks trained on SARR outperform the same networks trained on rotation
matrices, Euler angles, quaternions, standard trigonometrics or the recently popular 6d representation – even in inference scenarios where no prior knowledge of the objects'
symmetry properties is available. Code and a visualization toolkit are available at https://github.com/akriegler/SARR .
Read the full abstract ▾
Computer Vision
A Comparative Evaluation of Geometric Accuracy in NeRF and Gaussian Splatting
Mikolaj Zielinski, Eryk Vykysaly, Bartlomiej Biesiada et al. (+3 more)
Recent advances in neural rendering have introduced numerous 3D scene representations. Although standard computer
vision metrics evaluate the visual quality of generated images, they often overlook the fidelity of surface geometry. This limitation is particularly critical in robotics, where accurate geometry is essential for tasks such as grasping and object manipulation. In this paper, we present an evaluation pipeline for neural rendering methods that focuses on
geometric accuracy, along with a
benchmark comprising 19 diverse scenes. Our approach enables a systematic assessment of reconstruction methods in terms of surface and shape fidelity, complementing traditional visual metrics.
Read the full abstract ▾
All authors Mikolaj Zielinski, Eryk Vykysaly, Bartlomiej Biesiada, Jan Baturo, Mateusz Capala, Dominik Belter
Computer Vision
DiffuSAM: Diffusion Guided Zero-Shot Object Grounding for Remote Sensing Imagery
Geet Sethi, Panav Shah, Ashutosh Gandhe et al. (+1 more)
Diffusion models have emerged as powerful tools for a wide range of
vision tasks, including text-guided image generation and editing. In this work, we explore their potential for object grounding in remote sensing imagery. We propose a hybrid pipeline that integrates
diffusion-based localization cues with state-of-the-art
segmentation models such as RemoteSAM and SAM3 to obtain more accurate bounding boxes. By leveraging the complementary strengths of generative
diffusion models and foundational
segmentation models, our approach enables robust and adaptive object localization across complex scenes. Experiments demonstrate that our pipeline significantly improves localization performance, achieving over a 14% increase in Acc@0.5 compared to existing state-of-the-art methods.
Read the full abstract ▾
All authors Geet Sethi, Panav Shah, Ashutosh Gandhe, Soumitra Darshan Nayak