Computer Vision
CLIP-Map: Structured Matrix Mapping for Parameter-Efficient CLIP Compression
Kangjie Zhang, Wenxuan Huang, Xin Zhou et al. (+9 more)
Contrastive
Language-Image Pre-training (CLIP) has achieved widely applications in various computer
vision tasks, e.g., text-to-image generation, Image-Text
retrieval and Image captioning. However, CLIP suffers from high
memory and computation cost, which prohibits its usage to the resource-limited application scenarios. Existing CLIP compression methods typically reduce the size of pre-trained CLIP weights by selecting their subset as weight inheritance for further retraining via mask optimization or important weight measurement. However, these select-based weight inheritance often compromises the feature presentation ability, especially on the extreme compression. In this paper, we propose a novel mapping-based CLIP compression framework, CLIP-Map. It leverages learnable
matrices to map and combine pretrained weights by Full-Mapping with Kronecker Factorization, aiming to preserve as much information from the original weights as possible. To mitigate the optimization challenges introduced by the learnable mapping, we propose Diagonal Inheritance Initialization to reduce the distribution shifting problem for efficient and effective mapping learning. Extensive experimental results demonstrate that the proposed CLIP-Map outperforms select-based frameworks across various compression ratios, with particularly significant gains observed under high compression settings.
Read the full abstract ▾
All authors Kangjie Zhang, Wenxuan Huang, Xin Zhou, Boxiang Zhou, Dejia Song, Yuan Xie, Baochang Zhang, Lizhuang Ma, Nemo Chen, Xu Tang, Yao Hu, Shaohui Lin
Computer Vision
Neural Implicit 3D Cardiac Shape Reconstruction from Sparse CT Angiography Slices Mimicking 2D Transthoracic Echocardiography Views
Gino E. Jansen, Carolina Brás, R. Nils Planken et al. (+3 more)
Accurate 3D representations of cardiac structures allow quantitative analysis of anatomy and function. In this work, we propose a method for reconstructing complete 3D cardiac shapes from
segmentations of sparse planes in CT angiography (CTA) for application in 2D transthoracic echocardiography (TTE). Our method uses a neural implicit function to reconstruct the 3D shape of the cardiac chambers and left-ventricle myocardium from sparse CTA planes. To investigate the feasibility of achieving 3D reconstruction from 2D TTE, we select planes that mimic the standard apical 2D TTE views. During training, a multi-layer perceptron learns shape priors from 3D
segmentations of the target structures in CTA. At test time, the network reconstructs 3D cardiac shapes from
segmentations of TTE-mimicking CTA planes by jointly optimizing the latent code and the rigid transforms that map the observed planes into 3D space. For each heart, we simulate four realistic apical views, and we compare reconstructed multi-class volumes with the reference CTA volumes. On a held-out set of CTA
segmentations, our approach achieves an average Dice coefficient of 0.86 $\pm$ 0.04 across all structures. Our method also achieves markedly lower volume errors than the clinical standard, Simpson's biplane rule: 4.88 $\pm$ 4.26 mL vs. 8.14 $\pm$ 6.04 mL, respectively, for the left ventricle; and 6.40 $\pm$ 7.37 mL vs. 37.76 $\pm$ 22.96 mL, respectively, for the left atrium. This suggests that our approach offers a viable route to more accurate 3D chamber quantification in 2D transthoracic echocardiography.
Read the full abstract ▾
All authors Gino E. Jansen, Carolina Brás, R. Nils Planken, Mark J. Schuuring, Berto J. Bouma, Ivana Išgum
Computer Vision
EoCD: Encoder only Remote Sensing Change Detection
Mubashir Noman, Mustansar Fiaz, Hiyam Debary et al. (+4 more)
Being a cornerstone of temporal analysis, change detection has been playing a pivotal role in modern earth observation. Existing change detection methods rely on the Siamese encoder to individually extract temporal features followed by temporal fusion. Subsequently, these methods design sophisticated decoders to improve the change detection performance without taking into consideration the complexity of the model. These aforementioned issues intensify the overall computational cost as well as the network's complexity which is undesirable. Alternatively, few methods utilize the early fusion scheme to combine the temporal images. These methods prevent the extra overhead of Siamese encoder, however, they also rely on sophisticated decoders for better performance. In addition, these methods demonstrate inferior performance as compared to late fusion based methods. To bridge these gaps, we introduce encoder only change detection (EoCD) that is a simple and effective method for the change detection task. The proposed method performs the early fusion of the temporal data and replaces the decoder with a parameter-free multiscale feature fusion module thereby significantly reducing the overall complexity of the model. EoCD demonstrate the optimal balance between the change detection performance and the prediction speed across a variety of encoder architectures. Additionally, EoCD demonstrate that the performance of the model is predominantly dependent on the encoder network, making the decoder an additional component. Extensive experimentation on four challenging change detection
datasets reveals the effectiveness of the proposed method.
Read the full abstract ▾
All authors Mubashir Noman, Mustansar Fiaz, Hiyam Debary, Abdul Hannan, Shah Nawaz, Fahad Shahbaz Khan, Salman Khan
Computer Vision
Contour Refinement using Discrete Diffusion in Low Data Regime
Fei Yu Guan, Ian Keefe, Sophie Wilkinson et al. (+2 more)
Boundary detection of irregular and translucent objects is an important problem with applications in medical imaging, environmental monitoring and manufacturing, where many of these applications are plagued with scarce labeled data and low in situ computational resources. While recent image
segmentation studies focus on
segmentation mask
alignment with ground-truth, the task of boundary detection remains understudied, especially in the low data regime. In this work, we present a lightweight discrete
diffusion contour refinement pipeline for robust boundary detection in the low data regime. We use a Convolutional Neural Network(CNN) architecture with self-
attention layers as the core of our pipeline, and condition on a
segmentation mask, iteratively denoising a sparse contour representation. We introduce multiple novel adaptations for improved low-data efficacy and inference efficiency, including using a simplified
diffusion process, a customized model architecture, and minimal post processing to produce a dense, isolated contour given a
dataset of size <500 training images. Our method outperforms several SOTA baselines on the medical imaging
dataset KVASIR, is competitive on HAM10K and our custom wildfire
dataset, Smoke, while improving inference framerate by 3.5X.
Read the full abstract ▾
All authors Fei Yu Guan, Ian Keefe, Sophie Wilkinson, Daniel D.B. Perrakis, Steven Waslander
Computer Vision
Pathwise Test-Time Correction for Autoregressive Long Video Generation
Xunzhi Xiang, Zixuan Duan, Guiyu Zhang et al. (+7 more)
Distilled autoregressive
diffusion models facilitate real-time short
video synthesis but suffer from severe error accumulation during long-sequence generation. While existing Test-Time Optimization (TTO) methods prove effective for images or short clips, we identify that they fail to mitigate drift in extended sequences due to unstable reward landscapes and the hypersensitivity of distilled parameters. To overcome these limitations, we introduce Test-Time Correction (TTC), a training-free alternative. Specifically, TTC utilizes the initial frame as a stable reference anchor to calibrate intermediate
stochastic states along the sampling trajectory. Extensive experiments demonstrate that our method seamlessly integrates with various distilled models, extending generation lengths with negligible overhead while matching the quality of resource-intensive training-based methods on 30-second
benchmarks.
Read the full abstract ▾
All authors Xunzhi Xiang, Zixuan Duan, Guiyu Zhang, Haiyu Zhang, Zhe Gao, Junta Wu, Shaofeng Zhang, Tengfei Wang, Qi Fan, Chunchao Guo
Computer Vision · Cross-listed from cs.AI
OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention
Zhangquan Chen, Jiale Tao, Ruihuang Li et al. (+10 more)
While humans perceive the world through diverse modalities that operate synergistically to support a holistic understanding of their surroundings, existing omnivideo models still face substantial challenges on audio-visual understanding tasks. In this paper, we propose OmniVideo-R1, a novel reinforced framework that improves mixed-modality
reasoning. OmniVideo-R1 empowers models to "think with omnimodal cues" by two key strategies: (1) query-intensive grounding based on self-supervised learning paradigms; and (2) modality-attentive fusion built upon contrastive learning paradigms. Extensive experiments on multiple
benchmarks demonstrate that OmniVideo-R1 consistently outperforms strong baselines, highlighting its effectiveness and robust generalization capabilities.
Read the full abstract ▾
All authors Zhangquan Chen, Jiale Tao, Ruihuang Li, Yihao Hu, Ruitao Chen, Zhantao Yang, Xinlei Yu, Haodong Jing, Manyuan Zhang, Shuai Shao, Biao Wang, Qinglin Lu, Ruqi Huang
Computer Vision
Self-Supervised Learning with a Multi-Task Latent Space Objective
Pierre-François De Plaen, Abhishek Jha, Luc Van Gool et al. (+2 more)
We propose a multi-task formulation of self-predictive Siamese SSL in which each spatial transformation defines a distinct latent-space
alignment task, solved by a dedicated predictor over a shared encoder. This perspective directly explains a long-standing failure of multi-crop training in self-predictive methods such as BYOL, SimSiam, and MoCo v3: a shared predictor is forced to solve heterogeneous
alignment tasks simultaneously, leading to unstable optimization. Assigning one predictor per view type resolves this interference, unlocking linear evaluation gains of 3.8-4% across frameworks. This perspective also suggests a principled way to enrich pre-training by introducing additional spatial transformations as complementary tasks. We demonstrate this by introducing asymmetric cutout views, in which a masked online view is aligned with a complete target, forming a semantic inpainting objective. The resulting framework is stable, backbone-agnostic, and consistently improves the performance of ResNet and ViT models on ImageNet and COCO.
Read the full abstract ▾
All authors Pierre-François De Plaen, Abhishek Jha, Luc Van Gool, Tinne Tuytelaars, Marc Proesmans
Computer Vision
UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents
Han Xiao, Guozhi Wang, Hao Wang et al. (+7 more)
Online Reinforcement Learning (RL) offers a promising paradigm for enhancing GUI
agents through direct environment interaction. However, its effectiveness is severely hindered by inefficient credit assignment in long-horizon tasks and repetitive errors across tasks due to the lack of experience transfer. To address these challenges, we propose UI-Mem, a novel framework that enhances GUI online RL with a Hierarchical Experience
Memory. Unlike traditional replay buffers, our
memory accumulates structured knowledge, including high-level workflows, subtask skills, and failure patterns. These experiences are stored as parameterized templates that enable cross-task and cross-application transfer. To effectively integrate
memory guidance into online RL, we introduce Stratified Group Sampling, which injects varying levels of guidance across trajectories within each rollout group to maintain outcome diversity, driving the unguided policy toward internalizing guided behaviors. Furthermore, a Self-Evolving Loop continuously abstracts novel strategies and errors to keep the
memory aligned with the
agent's evolving policy. Experiments on online GUI
benchmarks demonstrate that UI-Mem significantly outperforms traditional RL baselines and static reuse strategies, with strong generalization to unseen applications. Project page: https://ui-mem.github.io
Read the full abstract ▾
All authors Han Xiao, Guozhi Wang, Hao Wang, Shilong Liu, Yuxiang Chai, Yue Pan, Yufeng Zhou, Xiaoxin Chen, Yafei Wen, Hongsheng Li
Computer Vision
Weaver: End-to-End Agentic System Training for Video Interleaved Reasoning
Yudi Shi, Shangzhe Di, Qirui Chen et al. (+5 more)
Video reasoning constitutes a comprehensive assessment of a model's capabilities, as it demands robust perceptual and interpretive skills, thereby serving as a means to explore the boundaries of model performance. While recent research has leveraged text-centric Chain-of-Thought
reasoning to augment these capabilities, such approaches frequently suffer from representational mismatch and restricted by limited perceptual acuity. To address these limitations, we propose Weaver, a novel, end-to-end trainable
multimodal reasoning agentic system. Weaver empowers its policy model to dynamically invoke diverse tools throughout the
reasoning process, enabling progressive acquisition of crucial visual cues and construction of authentic
multimodal reasoning trajectories. Furthermore, we integrate a reinforcement learning
algorithm to allow the system to freely explore strategies for employing and combining these tools with trajectory-free data. Extensive experiments demonstrate that our system, Weaver, enhances performance on several complex
video reasoning benchmarks, particularly those involving long
videos.
Read the full abstract ▾
All authors Yudi Shi, Shangzhe Di, Qirui Chen, Qinian Wang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Weidi Xie
Computer Vision
Hai Zhang, Siqi Liang, Li Chen et al. (+5 more)
Why must
vision-
language navigation be bound to detailed and verbose
language instructions? While such details ease decision-making, they fundamentally contradict the goal for
navigation in the real-world. Ideally,
agents should possess the autonomy to navigate in unknown environments guided solely by simple and high-level intents. Realizing this ambition introduces a formidable challenge: Beyond-the-View
Navigation (BVN), where
agents must locate distant, unseen targets without dense and step-by-step guidance. Existing large
language model (LLM)-based methods, though adept at following dense instructions, often suffer from short-sighted behaviors due to their reliance on short-horimzon supervision. Simply extending the supervision horizon, however, destabilizes LLM training. In this work, we identify that
video generation models inherently benefit from long-horizon supervision to align with
language instructions, rendering them uniquely suitable for BVN tasks. Capitalizing on this insight, we propose introducing the
video generation model into this field for the first time. Yet, the prohibitive latency for generating
videos spanning tens of seconds makes real-world deployment impractical. To bridge this gap, we propose SparseVideoNav, achieving sub-second trajectory inference guided by a generated sparse future spanning a 20-second horizon. This yields a remarkable 27x speed-up compared to the unoptimized counterpart. Extensive real-world zero-shot experiments demonstrate that SparseVideoNav achieves 2.5x the success rate of state-of-the-art LLM baselines on BVN tasks and marks the first realization of such capability in challenging night scenes.
Read the full abstract ▾
All authors Hai Zhang, Siqi Liang, Li Chen, Yuxian Li, Yukuan Xu, Yichao Zhong, Fu Zhang, Hongyang Li
Computer Vision
NVS-HO: A Benchmark for Novel View Synthesis of Handheld Objects
Musawar Ali, Manuel Carranza-García, Nicola Fioraio et al. (+2 more)
We propose NVS-HO, the first
benchmark designed for novel view synthesis of handheld objects in real-world environments using only RGB inputs. Each object is recorded in two complementary RGB sequences: (1) a handheld sequence, where the object is manipulated in front of a static camera, and (2) a board sequence, where the object is fixed on a ChArUco board to provide accurate camera poses via marker detection. The goal of NVS-HO is to learn a NVS model that captures the full appearance of an object from (1), whereas (2) provides the ground-truth images used for evaluation. To establish baselines, we consider both a classical SfM pipeline and a state-of-the-art pre-trained feed-forward neural network (VGGT) as pose
estimators, and train NVS models based on NeRF and Gaussian Splatting. Our experiments reveal significant performance gaps in current methods under unconstrained handheld conditions, highlighting the need for more robust approaches. NVS-HO thus offers a challenging real-world
benchmark to drive progress in RGB-based novel view synthesis of handheld objects.
Read the full abstract ▾
All authors Musawar Ali, Manuel Carranza-García, Nicola Fioraio, Samuele Salti, Luigi Di Stefano
Computer Vision
Focus-Scan-Refine: From Human Visual Perception to Efficient Visual Token Pruning
Enwei Tong, Yuanchao Bai, Yao Zhu et al. (+2 more)
Vision-
language models (VLMs) often generate massive visual
tokens that greatly increase inference latency and
memory footprint; while training-free
token pruning offers a practical remedy, existing methods still struggle to balance local evidence and global context under aggressive compression. We propose Focus-Scan-Refine (FSR), a human-inspired, plug-and-play
pruning framework that mimics how humans answer visual questions: focus on key evidence, then scan globally if needed, and refine the scanned context by aggregating relevant details. FSR first focuses on key evidence by combining visual importance with instruction relevance, avoiding the bias toward visually salient but query-irrelevant regions. It then scans for complementary context conditioned on the focused set, selecting
tokens that are most different from the focused evidence. Finally, FSR refines the scanned context by aggregating nearby informative
tokens into the scan anchors via similarity-based assignment and score-weighted merging, without increasing the
token budget. Extensive experiments across multiple VLM backbones and
vision-
language benchmarks show that FSR consistently improves the accuracy-efficiency trade-off over existing state-of-the-art
pruning methods. The source codes can be found at https://github.com/ILOT-code/FSR.
Read the full abstract ▾
All authors Enwei Tong, Yuanchao Bai, Yao Zhu, Junjun Jiang, Xianming Liu
Computer Vision
Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiation
Hengyi Wang, Ruiqiang Zhang, Chang Liu et al. (+4 more)
With the rising need for spatially grounded tasks such as
Vision-
Language Navigation/Action, allocentric perception capabilities in
Vision-
Language Models (VLMs) are receiving growing focus. However, VLMs remain brittle on allocentric spatial queries that require explicit perspective shifts, where the answer depends on
reasoning in a target-centric frame rather than the observed camera view. Thus, we introduce Allocentric Perceiver, a training-free strategy that recovers metric 3D states from one or more images with off-the-shelf
geometric experts, and then instantiates a query-conditioned allocentric reference frame aligned with the instruction's semantic intent. By deterministically transforming reconstructed geometry into the target frame and prompting the backbone VLM with structured, geometry-grounded representations, Allocentric Perceriver offloads mental rotation from implicit
reasoning to explicit computation. We evaluate Allocentric Perciver across multiple backbone families on spatial
reasoning benchmarks, observing consistent and substantial gains ($\sim$10%) on allocentric tasks while maintaining strong
egocentric performance, and surpassing both spatial-perception-finetuned models and state-of-the-art open-source and proprietary models.
Read the full abstract ▾
All authors Hengyi Wang, Ruiqiang Zhang, Chang Liu, Guanjie Wang, Zehua Ma, Han Fang, Weiming Zhang
Computer Vision
ReText: Text Boosts Generalization in Image-Based Person Re-identification
Timur Mamedov, Karina Kvanchiani, Anton Konushin et al. (+1 more)
Generalizable image-based person re-identification (Re-ID) aims to recognize individuals across cameras in unseen domains without retraining. While visual features are sensitive to domain shifts, natural
language descriptions remain consistent across domains, regardless of lighting changes and other biases. Since well-known multi-camera
datasets lack textual descriptions, we update them with captions through a fine-tuned
Vision-
Language Model (VLM). We propose ReText, a novel method trained with textual supervision to enrich visual features with semantic cues. Besides, inspired by recent findings showing that incorporating single-camera data into training improves generalization, we also adopt this strategy. However, in contrast to such prior works, we exploit not only visual information but also the textual annotations naturally accompanying these samples. Finally, during training, ReText jointly optimizes three tasks: (1) Re-ID, (2) image-text matching, and (3) image reconstruction guided by text. Experiments demonstrate that ReText achieves strong generalization and significantly outperforms state-of-the-art methods on cross-domain Re-ID
benchmarks. The expanded multi-camera
datasets and VLM weights are publicly available at https://github.com/timmzak/ReText.
Read the full abstract ▾
All authors Timur Mamedov, Karina Kvanchiani, Anton Konushin, Vadim Konushin
Computer Vision
FMPose3D: monocular 3D pose estimation via flow matching
Ti Wang, Xiaohang Yu and Mackenzie Weygandt Mathis
Monocular 3D pose estimation is fundamentally ill-posed due to depth ambiguity and occlusions, thereby motivating probabilistic methods that generate multiple plausible 3D pose hypotheses. In particular,
diffusion-based models have recently demonstrated strong performance, but their iterative denoising process typically requires many timesteps for each prediction, making inference computationally expensive. In contrast, we leverage Flow Matching (FM) to learn a velocity field defined by an Ordinary Differential Equation (ODE), enabling efficient generation of 3D pose samples with only a few integration steps. We propose a novel generative pose estimation framework, FMPose3D, that formulates 3D pose estimation as a conditional distribution transport problem. It continuously transports samples from a standard Gaussian prior to the distribution of plausible 3D poses conditioned only on 2D inputs. Although ODE trajectories are deterministic, FMPose3D naturally generates various pose hypotheses by sampling different noise seeds. To obtain a single accurate prediction from those hypotheses, we further introduce a Reprojection-based Posterior Expectation Aggregation (RPEA) module, which approximates the
Bayesian posterior expectation over 3D hypotheses. FMPose3D surpasses existing methods on the widely used human pose estimation
benchmarks Human3.6M and MPI-INF-3DHP, and further achieves state-of-the-art performance on the 3D animal pose
datasets Animal3D and CtrlAni3D, demonstrating strong performance across both 3D pose domains. The code is available at https://github.com/AdaptiveMotorControlLab/FMPose3D.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Disc-Centric Contrastive Learning for Lumbar Spine Severity Grading
Sajjan Acharya and Pralisha Kansakar
This work examines a disc-centric approach for automated severity grading of lumbar spinal stenosis from sagittal T2-weighted MRI. The method combines contrastive pretraining with disc-level fine-tuning, using a single anatomically localized region of interest per intervertebral disc. Contrastive learning is employed to help the model focus on meaningful disc features and reduce sensitivity to irrelevant differences in image appearance. The framework includes an auxiliary regression task for disc localization and applies weighted focal loss to address class imbalance. Experiments demonstrate a 78.1% balanced accuracy and a reduced severe-to-normal misclassification rate of 2.13% compared with supervised training from scratch. Detecting discs with moderate severity can still be challenging, but focusing on disc-level features provides a practical way to assess the lumbar spinal stenosis.
Read the full abstract ▾
Computer Vision
Neuro-Inspired Visual Pattern Recognition via Biological Reservoir Computing
Luca Ciampi, Ludovico Iannello, Fabrizio Tonelli et al. (+4 more)
In this paper, we present a neuro-inspired approach to reservoir computing (RC) in which a network of in vitro cultured cortical
neurons serves as the physical reservoir. Rather than relying on artificial recurrent models to approximate neural dynamics, our biological reservoir computing (BRC) system leverages the spontaneous and stimulus-evoked activity of living neural
circuits as its computational substrate. A high-density multi-electrode array (HD-MEA) provides simultaneous stimulation and readout across hundreds of channels: input patterns are delivered through selected electrodes, while the remaining ones capture the resulting high-dimensional neural responses, yielding a biologically grounded feature representation. A linear readout layer (single-layer perceptron) is then trained to classify these reservoir states, enabling the living neural network to perform static visual pattern-recognition tasks within a computer-
vision framework. We evaluate the system across a sequence of tasks of increasing difficulty, ranging from pointwise stimuli to oriented bars, clock-digit-like shapes, and handwritten digits from the MNIST
dataset. Despite the inherent variability of biological neural responses-arising from noise, spontaneous activity, and inter-session differences-the system consistently generates high-dimensional representations that support accurate classification. These results demonstrate that in vitro cortical networks can function as effective reservoirs for static visual pattern recognition, opening new avenues for integrating living neural substrates into neuromorphic computing frameworks. More broadly, this work contributes to the effort to incorporate biological principles into machine learning and supports the goals of neuro-inspired
vision by illustrating how living neural systems can inform the design of efficient and biologically grounded computational models.
Read the full abstract ▾
All authors Luca Ciampi, Ludovico Iannello, Fabrizio Tonelli, Gabriele Lagani, Angelo Di Garbo, Federico Cremisi, Giuseppe Amato
Computer Vision
Depth as Prior Knowledge for Object Detection
Moussa Kassem Sbeyti and Nadja Klein
Detecting small and distant objects remains challenging for object detectors due to scale variation, low resolution, and background clutter. Safety-critical applications require reliable detection of these objects for safe planning. Depth information can improve detection, but existing approaches require complex, model-specific architectural modifications. We provide a theoretical analysis followed by an empirical investigation of the depth-detection relationship. Together, they explain how depth causes systematic performance degradation and why depth-informed supervision mitigates it. We introduce DepthPrior, a framework that uses depth as prior knowledge rather than as a fused feature, providing comparable benefits without modifying detector architectures. DepthPrior consists of Depth-Based Loss Weighting (DLW) and Depth-Based Loss Stratification (DLS) during training, and Depth-Aware Confidence Thresholding (DCT) during inference. The only overhead is the initial cost of depth estimation. Experiments across four
benchmarks (KITTI, MS COCO, VisDrone, SUN RGB-D) and two detectors (YOLOv11, EfficientDet) demonstrate the effectiveness of DepthPrior, achieving up to +9% mAP$_S$ and +7% mAR$_S$ for small objects, with inference recovery rates as high as 95:1 (true vs. false detections). DepthPrior offers these benefits without additional sensors, architectural changes, or performance costs. Code is available at https://github.com/mos-ks/DepthPrior.
Read the full abstract ▾
Computer Vision
Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplification
Lexiang Hu, Youze Xue, Dian Li et al. (+2 more)
Multimodal embeddings serve as a bridge for aligning
vision and
language, with the two primary implementations – CLIP-based and MLLM-based
embedding models – both limited to capturing only global semantic information. Although numerous studies have focused on fine-grained understanding, we observe that complex scenarios currently targeted by MLLM
embeddings often involve a hybrid perceptual pattern of both global and fine-grained elements, thus necessitating a compatible fusion mechanism. In this paper, we propose Adaptive Global and Fine-grained perceptual Fusion for MLLM
Embeddings (AGFF-Embed), a method that
prompts the MLLM to generate multiple
embeddings focusing on different dimensions of semantic information, which are then adaptively and smoothly aggregated. Furthermore, we adapt AGFF-Embed with the Explicit Gradient Amplification (EGA) technique to achieve in-batch hard negatives enhancement without requiring fine-grained editing of the
dataset. Evaluation on the MMEB and MMVP-VLM
benchmarks shows that AGFF-Embed comprehensively achieves state-of-the-art performance in both general and fine-grained understanding compared to other
multimodal embedding models.
Read the full abstract ▾
All authors Lexiang Hu, Youze Xue, Dian Li, Gang Liu, Zhouchen Lin
Computer Vision
Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization
Yunchuan Ma, Laiyun Qing, Guorong Li et al. (+3 more)
Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (i.e., labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed
videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across
videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four
benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.
Read the full abstract ▾
All authors Yunchuan Ma, Laiyun Qing, Guorong Li, Yuqing Liu, Yuankai Qi, Qingming Huang
Computer Vision · Cross-listed from cs.CY
Ethology of Latent Spaces
Philippe Boisnard
This study challenges the presumed neutrality of latent spaces in
vision language models (VLMs) by adopting an ethological perspective on their algorithmic behaviors. Rather than constituting spaces of homogeneous indeterminacy, latent spaces exhibit model-specific algorithmic sensitivities, understood as differential regimes of perceptual salience shaped by training data and architectural choices. Through a comparative analysis of three models (OpenAI CLIP, OpenCLIP LAION, SigLIP) applied to a corpus of 301 artworks (15th to 20th), we reveal substantial divergences in the attribution of political and cultural categories. Using bipolar semantic axes derived from vector analogies (Mikolov et al., 2013), we show that SigLIP classifies 59.4% of the artworks as politically engaged, compared to only 4% for OpenCLIP. African masks receive the highest political scores in SigLIP while remaining apolitical in OpenAI CLIP. On an aesthetic colonial axis, inter-model discrepancies reach 72.6 percentage points. We introduce three operational concepts: computational latent politicization, describing the emergence of political categories without intentional encoding; emergent bias, irreducible to statistical or normative bias and detectable only through contrastive analysis; and three algorithmic scopic regimes: entropic (LAION), institutional (OpenAI), and semiotic (SigLIP), which structure distinct modes of visibility. Drawing on Foucault's notion of the archive, Jameson's ideologeme, and Simondon's theory of individuation, we argue that training
datasets function as quasi-archives whose discursive formations crystallize within latent space. This work contributes to a critical reassessment of the conditions under which VLMs are applied to digital art history and calls for methodologies that integrate learning architectures into any delegation of cultural interpretation to algorithmic
agents.
Read the full abstract ▾
Computer Vision
Poster: Camera Tampering Detection for Outdoor IoT Systems
Shadi Attarha, Kanaga Shanmugi and Anna Förster
Recently, the use of smart cameras in outdoor settings has grown to improve surveillance and security. Nonetheless, these systems are susceptible to tampering, whether from deliberate vandalism or harsh environmental conditions, which can undermine their monitoring effectiveness. In this context, detecting camera tampering is more challenging when a camera is capturing still images rather than
video as there is no sequence of continuous frames over time. In this study, we propose two approaches for detecting tampered images: a rule-based method and a deep-learning-based method. The aim is to evaluate how each method performs in terms of accuracy, computational demands, and the data required for training when applied to real-world scenarios. Our results show that the deep-learning model provides higher accuracy, while the rule-based method is more appropriate for scenarios where resources are limited and a prolonged calibration phase is impractical. We also offer publicly available
datasets with normal, blurred, and rotated images to support the development and evaluation of camera tampering detection methods, addressing the need for such resources.
Read the full abstract ▾
Computer Vision
ShapeUP: Scalable Image-Conditioned 3D Editing
Inbar Gat, Dana Cohen-Bar, Guy Levy et al. (+2 more)
Recent advancements in 3D foundation models have enabled the generation of high-fidelity assets, yet precise 3D manipulation remains a significant challenge. Existing 3D editing frameworks often face a difficult trade-off between visual controllability,
geometric consistency, and scalability. Specifically, optimization-based methods are prohibitively slow, multi-view 2D propagation techniques suffer from visual drift, and training-free latent manipulation methods are inherently bound by frozen priors and cannot directly benefit from scaling. In this work, we present ShapeUP, a scalable, image-conditioned 3D editing framework that formulates editing as a supervised latent-to-latent translation within a native 3D representation. This formulation allows ShapeUP to build on a pretrained 3D foundation model, leveraging its strong generative prior while adapting it to editing through supervised training. In practice, ShapeUP is trained on triplets consisting of a source 3D shape, an edited 2D image, and the corresponding edited 3D shape, and learns a direct mapping using a 3D
Diffusion Transformer (DiT). This image-as-
prompt approach enables fine-grained visual control over both local and global edits and achieves implicit, mask-free localization, while maintaining strict structural consistency with the original asset. Our extensive evaluations demonstrate that ShapeUP consistently outperforms current trained and training-free baselines in both identity preservation and edit fidelity, offering a robust and scalable paradigm for native 3D content creation.
Read the full abstract ▾
All authors Inbar Gat, Dana Cohen-Bar, Guy Levy, Elad Richardson, Daniel Cohen-Or
Computer Vision
Enhancing Personality Recognition by Comparing the Predictive Power of Traits, Facets, and Nuances
Amir Ansari, Jana Subirana, Bruna Silva et al. (+3 more)
Personality is a complex, hierarchical construct typically assessed through item-level questionnaires aggregated into broad trait scores. Personality recognition models aim to infer personality traits from different sources of behavioral data. However, reliance on broad trait scores as ground truth, combined with limited training data, poses challenges for generalization, as similar trait scores can manifest through diverse, context dependent behaviors. In this work, we explore the predictive impact of the more granular hierarchical levels of the Big-Five Personality Model, facets and nuances, to enhance personality recognition from audiovisual interaction data. Using the UDIVA v0.5
dataset, we trained a
transformer-based model including cross-modal (audiovisual) and cross-subject (dyad-aware)
attention mechanisms. Results show that nuance-level models consistently outperform facet and trait-level models, reducing mean squared error by up to 74% across interaction scenarios.
Read the full abstract ▾
All authors Amir Ansari, Jana Subirana, Bruna Silva, Sergio Escalera, David Gallardo-Pujol, Cristina Palmero
Computer Vision
SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
Jinlin Wu, Felix Holm, Chuxi Chen et al. (+18 more)
While foundation models have advanced
surgical video analysis, current approaches rely predominantly on pixel-level reconstruction objectives that waste model capacity on low-level visual details, such as smoke, specular reflections, and fluid motion, rather than semantic structures essential for
surgical understanding. We present SurgMotion, a
video-native foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction. Built on the
Video Joint
Embedding Predictive Architecture (V-JEPA), SurgMotion introduces three key technical innovations tailored to
surgical videos: (1) motion-guided latent masked prediction to prioritize semantically meaningful regions, (2) spatiotemporal affinity self-
distillation to enforce relational consistency, and (3) spatiotemporal feature diversity regularization (SFDR) to prevent representation collapse in texture-sparse
surgical scenes. To enable large-scale pretraining, we curate SurgMotion-15M, the largest
surgical video dataset to date, comprising 3,658 hours of
video from 50 sources across 13 anatomical regions. Extensive experiments across 17
benchmarks demonstrate that SurgMotion significantly outperforms state-of-the-art methods on
surgical workflow recognition, achieving 14.6 percent improvement in F1 score on EgoSurgery and 10.3 percent on PitVis; on action triplet recognition with 39.54 percent mAP-IVT on CholecT50; as well as on skill assessment, polyp
segmentation, and depth estimation. These results establish SurgMotion as a new standard for universal, motion-oriented
surgical video understanding.
Read the full abstract ▾
All authors Jinlin Wu, Felix Holm, Chuxi Chen, An Wang, Yaxin Hu, Xiaofan Ye, Zelin Zang, Miao Xu, Lihua Zhou, Huarui Wu, Huai Liao, Danny T. M. Chan, Ming Feng, Wai S. Poon, Hongliang Ren, Dong Yi, Nassir Navab, Gaofeng Meng, Jiebo Luo, Hongbin Liu, Zhen Lei
Computer Vision · Cross-listed from cs.SE
ROMAN: Reward-Orchestrated Multi-Head Attention Network for Autonomous Driving System Testing
Jianlei Chi, Yuzhen Wu, Jiaxuan Hou et al. (+7 more)
Automated Driving System (ADS) acts as the
brain of autonomous vehicles, responsible for their safety and efficiency. Safe deployment requires thorough testing in diverse real-world scenarios and compliance with traffic laws like speed limits, signal obedience, and right-of-way rules. Violations like running red lights or speeding pose severe safety risks. However, current testing approaches face significant challenges: limited ability to generate complex and high-risk law-breaking scenarios, and failing to account for complex interactions involving multiple vehicles and critical situations. To address these challenges, we propose ROMAN, a novel scenario generation approach for ADS testing that combines a multi-head
attention network with a traffic law weighting mechanism. ROMAN is designed to generate high-risk violation scenarios to enable more thorough and targeted ADS evaluation. The multi-head
attention mechanism models interactions among vehicles, traffic signals, and other factors. The traffic law weighting mechanism implements a workflow that leverages an LLM-based risk weighting module to evaluate violations based on the two dimensions of severity and occurrence. We have evaluated ROMAN by testing the Baidu Apollo ADS within the CARLA simulation platform and conducting extensive experiments to measure its performance. Experimental results demonstrate that ROMAN surpassed state-of-the-art tools ABLE and LawBreaker by achieving 7.91% higher average violation count than ABLE and 55.96% higher than LawBreaker, while also maintaining greater scenario diversity. In addition, only ROMAN successfully generated violation scenarios for every clause of the input traffic laws, enabling it to identify more high-risk violations than existing approaches.
Read the full abstract ▾
All authors Jianlei Chi, Yuzhen Wu, Jiaxuan Hou, Xiaodong Zhang, Ming Fan, Suhui Sun, Weijun Dai, Bo Li, Jianguo Sun, Jun Sun
Computer Vision
Unified Sensor Simulation for Autonomous Driving
Nikolay Patakin, Arsenii Shirokov, Anton Konushin et al. (+1 more)
In this work, we introduce XSIM, a sensor simulation framework for autonomous driving. XSIM extends 3DGUT splatting with a generalized rolling-shutter modeling tailored for autonomous driving applications. Our framework provides a unified and flexible formulation for appearance and
geometric sensor modeling, enabling rendering of complex sensor distortions in dynamic environments. We identify spherical cameras, such as
LiDARs, as a critical edge case for existing 3DGUT splatting due to cyclic projection and time discontinuities at azimuth boundaries leading to incorrect particle projection. To address this issue, we propose a phase modeling mechanism that explicitly accounts temporal and shape discontinuities of Gaussians projected by the Unscented Transform at azimuth borders. In addition, we introduce an extended 3D Gaussian representation that incorporates two distinct opacity parameters to resolve mismatches between geometry and color distributions. As a result, our framework provides enhanced scene representations with improved
geometric consistency and photorealistic appearance. We evaluate our framework extensively on multiple autonomous driving
datasets, including Waymo Open
Dataset, Argoverse 2, and PandaSet. Our framework consistently outperforms strong recent baselines and achieves state-of-the-art performance across all
datasets. The source code is publicly available at \href{https://github.com/whesense/XSIM}{https://github.com/whesense/XSIM}.
Read the full abstract ▾
All authors Nikolay Patakin, Arsenii Shirokov, Anton Konushin, Dmitry Senushkin
Computer Vision · Cross-listed from cs.LG
Shiva-DiT: Residual-Based Differentiable Top-$k$ Selection for Efficient Diffusion Transformers
Jiaji Zhang, Hailiang Zhao, Jiaju Wu et al. (+3 more)
Diffusion Transformers (DiTs) are costly at high resolution because self-
attention scales quadratically with
token sequence length. Existing
pruning methods do not jointly provide end-to-end learnability, low training overhead, and deterministic
token counts for predictable
token-dependent computation. We propose Shiva-DiT, based on Residual-Based Differentiable Top-k Selection. Its forward pass executes hard top-k selection, while a residual-aware straight-through
estimator propagates gradients to both
token scores and the budget k without evaluating a second backbone path. A Context-Aware Router and Adaptive Ratio Policy learn layer- and timestep-dependent retention schedules under a target average budget. Experiments on SD3-Medium, Flux.1-dev, and PixArt-{\Sigma} show consistent reductions in FLOPs and measured latency. On SD3-Medium, Shiva-DiT provides four fidelity-latency operating points and reaches a 1.54x wall-clock speedup with competitive fidelity.
Read the full abstract ▾
All authors Jiaji Zhang, Hailiang Zhao, Jiaju Wu, Ruichao Sun, Xinkui Zhao, Shuiguang Deng
Computer Vision
Multi-instance robust fitting for non-classical geometric models
Zongliang Zhang, Shuxiang Li, Xingwang Huang et al. (+1 more)
Most existing robust fitting methods are designed for classical models, such as lines, circles, and planes. In contrast, fewer methods have been developed to robustly handle non-classical models, such as spiral curves, procedural character models, and free-form surfaces. Furthermore, existing methods primarily focus on reconstructing a single instance of a non-classical model. This paper aims to reconstruct multiple instances of non-classical models from noisy data. We formulate this multi-instance fitting task as an optimization problem, which comprises an
estimator and an optimizer. Specifically, we propose a novel
estimator based on the model-to-data error, capable of handling outliers without a predefined error threshold. Since the proposed
estimator is non-differentiable with respect to the model parameters, we employ a meta-heuristic
algorithm as the optimizer to seek the global optimum. The effectiveness of our method are demonstrated through experimental results on various non-classical models. The code is available at https://github.com/zhangzongliang/fitting.
Read the full abstract ▾
All authors Zongliang Zhang, Shuxiang Li, Xingwang Huang, Zongyue Wang
Computer Vision
CAViT – Channel-Aware Vision Transformer for Dynamic Feature Fusion
Aon Safdar and Mohamed Saadeldin
Vision Transformers (ViTs) have demonstrated strong performance across a range of computer
vision tasks by modeling long-range spatial interactions via self-
attention. However, channel-wise mixing in ViTs remains static, relying on fixed multilayer perceptrons (MLPs) that lack adaptability to input content. We introduce 'CAViT', a dual-
attention architecture that replaces the static MLP with a dynamic,
attention-based mechanism for feature interaction. Each
Transformer block in CAViT performs spatial self-
attention followed by channel-wise self-
attention, allowing the model to dynamically recalibrate feature representations based on global image context. This unified and content-aware
token mixing strategy enhances representational expressiveness without increasing depth or complexity. We validate CAViT across five
benchmark datasets spanning both natural and medical domains, where it outperforms the standard ViT baseline by up to +3.6% in accuracy, while reducing parameter count and FLOPs by over 30%. Qualitative
attention maps reveal sharper and semantically meaningful activation patterns, validating the effectiveness of our
attention-driven
token mixing.
Read the full abstract ▾