Computer Vision
4D Monocular Surgical Reconstruction under Arbitrary Camera Motions
Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh et al. (+5 more)
Reconstructing deformable
surgical scenes from endoscopic
videos is challenging and clinically important. Recent state-of-the-art methods based on implicit neural representations or 3D Gaussian splatting have made notable progress. However, most are designed for deformable scenes with fixed endoscope viewpoints and rely on stereo depth priors or accurate structure-from-motion for initialization and optimization, limiting their ability to handle
monocular sequences with large camera motion in real clinical settings. To address this, we propose Local-EndoGS, a high-quality 4D reconstruction framework for
monocular endoscopic sequences with arbitrary camera motion. Local-EndoGS introduces a progressive, window-based global representation that allocates local deformable scene models to each observed window, enabling scalability to long sequences with substantial motion. To overcome unreliable initialization without stereo depth or accurate structure-from-motion, we design a coarse-to-fine strategy integrating multi-view geometry, cross-window information, and
monocular depth priors, providing a robust foundation for optimization. We further incorporate long-range 2D pixel trajectory constraints and physical motion priors to improve deformation plausibility. Experiments on three public endoscopic
datasets with deformable scenes and varying camera motions show that Local-EndoGS consistently outperforms state-of-the-art methods in appearance quality and geometry. Ablation studies validate the effectiveness of our key designs. Code will be released upon acceptance at: https://github.com/IRMVLab/Local-EndoGS.
Read the full abstract ▾
All authors Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh, Yirui Li, Hao Liu, Lijun Han, Hesheng Wang, Shing Shin Cheng
Computer Vision
EAGLE: Expert-Augmented Attention Guidance for Tuning-Free Industrial Anomaly Detection in Multimodal Large Language Models
Xiaomeng Peng, Xilang Huang and Seon Han Choi
Multimodal large
language models (MLLMs) can enrich industrial anomaly detection with semantic descriptions and anomaly
reasoning, but they still lag specialist anomaly detectors in binary detection accuracy. Existing approaches address this gap by fine-tuning MLLMs or training bridging modules to align expert outputs with MLLM inputs, limiting flexibility across backbones. We propose EAGLE, a tuning-free framework that integrates expert anomaly detectors with frozen MLLMs. EAGLE consists of Threshold-Guided
Prompt Selection (TGPS), which estimates a decision threshold from expert model statistics and selects textual and visual
prompts, and Confidence-Aware
Attention Sharpening (CAAS), which shifts MLLM
attention toward visual evidence when expert confidence is low. Beyond improving accuracy, we analyze MLLM
attention and find that correct anomaly predictions are associated with stronger focus on ground-truth defect regions; EAGLE consistently strengthens this
alignment. On MVTec-AD and VisA, EAGLE improves five MLLM backbones without parameter updates, reaching up to 94.4% and 88.1% in anomaly discrimination accuracy, respectively, and achieving performance competitive with fine-tuning-based methods while largely preserving MLLM semantic
reasoning ability.
Read the full abstract ▾
Computer Vision
A High-Level Survey of Optical Remote Sensing
Panagiotis Koletsis, Vasilis Efthymiou, Maria Vakalopoulou et al. (+3 more)
In recent years, significant advances in computer
vision have also propelled progress in remote sensing. Concurrently, the use of drones has expanded, with many organizations incorporating them into their operations. Most drones are equipped by default with RGB cameras, which are both robust and among the easiest sensors to use and interpret. The body of literature on optical remote sensing is vast, encompassing diverse tasks, capabilities, and methodologies. Each task or methodology could warrant a dedicated survey. This work provides a comprehensive overview of the capabilities of the field, while also presenting key information, such as
datasets and insights. It aims to serve as a guide for researchers entering the field, offering high-level insights and helping them focus on areas most relevant to their interests. To the best of our knowledge, no existing survey addresses this holistic perspective.
Read the full abstract ▾
All authors Panagiotis Koletsis, Vasilis Efthymiou, Maria Vakalopoulou, Nikos Komodakis, Anastasios Doulamis, Georgios Th. Papadopoulos
Computer Vision
SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery
Lorenzo Caselli, Marco Mistretta, Simone Magistri et al. (+1 more)
Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent
multimodal approaches improve performance by incorporating textual information. However, they treat modalities independently and incur high computational cost. We propose SpectralGCD, an efficient and effective
multimodal approach to GCD that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation. Each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, which anchors learning to explicit semantics and reduces reliance on spurious visual cues. To maintain the semantic quality of representations learned by an efficient student, we introduce Spectral Filtering which exploits a cross-modal covariance
matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge
distillation from the same teacher ensures that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six
benchmarks, SpectralGCD delivers accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost. The code is publicly available at: https://github.com/miccunifi/SpectralGCD.
Read the full abstract ▾
All authors Lorenzo Caselli, Marco Mistretta, Simone Magistri, Andrew D. Bagdanov
Computer Vision
DRetHTR: Linear-Time Decoder-Only Retentive Network for Handwritten Text Recognition
Changhun Kim, Martin Mayr, Thomas Gorges et al. (+4 more)
State-of-the-art handwritten text recognition (HTR) systems commonly use
Transformers, whose growing key-value (KV) cache makes decoding slow and
memory-intensive. We introduce DRetHTR, a decoder-only model built on Retentive Networks (RetNet). Compared to an equally sized decoder-only
Transformer baseline, DRetHTR delivers 1.6-1.9x faster inference with 38-42% less
memory usage, without loss of accuracy. By replacing softmax
attention with softmax-free retention and injecting multi-scale sequential priors, DRetHTR avoids a growing KV cache: decoding is linear in output length in both time and
memory. To recover the local-to-global inductive bias of
attention, we propose layer-wise gamma scaling, which progressively enlarges the effective retention horizon in deeper layers. This encourages early layers to model short-range dependencies and later layers to capture broader context, mitigating the flexibility gap introduced by removing softmax. Consequently, DRetHTR achieves best reported test character error rates of 2.26% (IAM-A, en), 1.81% (RIMES, fr), and 3.46% (Bentham, en), and is competitive on READ-2016 (de) with 4.21%. This demonstrates that decoder-only RetNet enables
Transformer-level HTR accuracy with substantially improved decoding speed and
memory efficiency.
Read the full abstract ▾
All authors Changhun Kim, Martin Mayr, Thomas Gorges, Fei Wu, Mathias Seuret, Andreas Maier, Vincent Christlein
Computer Vision
Tree crop mapping of South America reveals links to deforestation and conservation
Yuchang Jiang, Anton Raichuk, Xiaoye Tong et al. (+6 more)
Monitoring tree crop expansion is vital for zero-deforestation policies like the European Union's Regulation on Deforestation-free Products (EUDR). However, these efforts are hindered by a lack of highresolution data distinguishing diverse agricultural systems from forests. Here, we present the first 10m-resolution tree crop map for South America, generated using a multi-modal, spatio-temporal deep learning model trained on Sentinel-1 and Sentinel-2 satellite imagery time series. The map identifies approximately 11 million hectares of tree crops, 23% of which is linked to 2000-2020 forest cover loss. Critically, our analysis reveals that existing regulatory maps supporting the EUDR often classify established agriculture, particularly smallholder agroforestry, as "forest". This discrepancy risks false deforestation alerts and unfair penalties for small-scale farmers. Our work mitigates this risk by providing a high-resolution baseline, supporting conservation policies that are effective, inclusive, and equitable.
Read the full abstract ▾
All authors Yuchang Jiang, Anton Raichuk, Xiaoye Tong, Vivien Sainte Fare Garnot, Daniel Ortiz-Gonzalo, Dan Morris, Konrad Schindler, Jan Dirk Wegner, Maxim Neumann
Computer Vision · Cross-listed from math.NA
Application and Evaluation of the Common Circles Method
Michael Quellmalz, Mia Kvåle Løvmo, Simon Moser et al. (+2 more)
We investigate the application of the common circle method for estimating sample motion in optical diffraction
tomography (ODT) of sub-millimeter sized biological tissue. When samples are confined via contact-free acoustical force fields, their motion must be estimated from the captured images. The common circle method identifies intersections of Ewald spheres in
Fourier space to determine rotational motion. This paper presents a practical implementation, incorporating temporal consistency constraints to achieve stable reconstructions. Our results on both simulated and real-world data demonstrate that the common circle method provides a computationally efficient alternative to full optimization methods for motion detection.
Read the full abstract ▾
All authors Michael Quellmalz, Mia Kvåle Løvmo, Simon Moser, Franziska Strasser, Monika Ritsch-Marte
Computer Vision
Polaffini: A feature-based approach for robust affine and polyaffine image registration
Antoine Legouhy, Cosimo Campo, Ross Callaghan et al. (+2 more)
In this work we present Polaffini, a robust and versatile framework for anatomically grounded registration. Medical image registration is dominated by intensity-based registration methods that rely on surrogate measures of
alignment quality. In contrast, feature-based approaches that operate by identifying explicit anatomical correspondences, while more desirable in theory, have largely fallen out of favor due to the challenges of reliably extracting features. However, such challenges are now significantly overcome thanks to recent advances in deep learning, which provide pre-trained
segmentation models capable of instantly delivering reliable, fine-grained anatomical delineations. We aim to demonstrate that these advances can be leveraged to create new anatomically-grounded image registration
algorithms. To this end, we propose Polaffini, which obtains, from these segmented regions, anatomically grounded feature points with 1-to-1 correspondence in a particularly simple way: extracting their centroids. These enable efficient global and local affine matching via closed-form solutions. Those are used to produce an overall transformation ranging from affine to polyaffine with tunable smoothness. Polyaffine transformations can have many more degrees of freedom than affine ones allowing for finer
alignment, and their
embedding in the log-Euclidean framework ensures diffeomorphic properties. Polaffini has applications both for standalone registration and as pre-
alignment for subsequent non-linear registration, and we evaluate it against popular intensity-based registration techniques. Results demonstrate that Polaffini outperforms competing methods in terms of structural
alignment and provides improved initialisation for downstream non-linear registration. Polaffini is fast, robust, and accurate, making it particularly well-suited for integration into medical image processing pipelines.
Read the full abstract ▾
All authors Antoine Legouhy, Cosimo Campo, Ross Callaghan, Hojjat Azadbakht, Hui Zhang
Computer Vision
Leveraging Contrastive Learning for a Similarity-Guided Tampered Document Data Generation Pipeline
Mohamed Dhouib, Davide Buscaldi, Sonia Vanier et al. (+1 more)
Detecting tampered text in document images is a challenging task due to data scarcity. To address this, previous work has attempted to generate tampered documents using rule-based methods. However, the resulting documents often suffer from limited variety and poor visual quality, typically leaving highly visible artifacts that are rarely observed in real-world manipulations. This undermines the model's ability to learn robust, generalizable features and results in poor performance on real-world data. Motivated by this discrepancy, we propose a novel method for generating high-quality tampered document images. We first train an auxiliary network to compare text crops, leveraging contrastive learning with a novel strategy for defining positive pairs and their corresponding negatives. We also train a second auxiliary network to evaluate whether a crop tightly encloses the intended characters, without cutting off parts of characters or including parts of adjacent ones. Using a carefully designed generation pipeline that leverages both networks, we introduce a framework capable of producing diverse, high-quality tampered document images. We assess the effectiveness of our data generation pipeline by training multiple models on
datasets derived from the same source images, generated using our method and existing approaches, under identical training protocols. Evaluating these models on various open-source
datasets shows that our pipeline yields consistent performance improvements across architectures and
datasets.
Read the full abstract ▾
All authors Mohamed Dhouib, Davide Buscaldi, Sonia Vanier, Aymen Shabou
Computer Vision · Cross-listed from cs.LG
The Sound of Death: Deep Learning Reveals Vascular Damage from Carotid Ultrasound
Christoph Balada, Aida Romano-Martinez, Payal Varshney et al. (+10 more)
Cardiovascular diseases (CVDs) remain the leading cause of mortality worldwide, yet early risk detection is often limited by available diagnostics. Carotid ultrasound, a non-invasive and widely accessible modality, encodes rich structural and hemodynamic information that is largely untapped. Here, we present a machine learning (ML) framework that extracts clinically meaningful representations of vascular damage (VD) from carotid ultrasound
videos, using hypertension as a weak proxy label. The model learns robust features that are biologically plausible, interpretable, and strongly associated with established cardiovascular risk factors, comorbidities, and laboratory measures. High VD stratifies individuals for myocardial infarction, cardiac death, and all-cause mortality, matching or outperforming conventional risk models such as SCORE2. Explainable AI analyses reveal that the model relies on vessel morphology and perivascular tissue characteristics, uncovering novel functional and anatomical signatures of vascular damage. This work demonstrates that routine carotid ultrasound contains far more prognostic information than previously recognized. Our approach provides a scalable, non-invasive, and cost-effective tool for population-wide cardiovascular risk assessment, enabling earlier and more personalized prevention strategies without reliance on laboratory tests or complex clinical inputs.
Read the full abstract ▾
All authors Christoph Balada, Aida Romano-Martinez, Payal Varshney, Vincent ten Cate, Katharina Geschke, Jonas Tesarz, Paul Claßen, Alexander K. Schuster, Dativa Tibyampansha, Karl-Patrik Kresoja, Philipp S. Wild, Sheraz Ahmed, Andreas Dengel
Computer Vision
Attachment Anchors: A Novel Framework for Laparoscopic Grasping Point Prediction in Colorectal Surgery
Dennis N. Schneider, Lars Wagner, Daniel Rueckert et al. (+1 more)
Accurate grasping point prediction is a key challenge for autonomous tissue manipulation in minimally invasive surgery, particularly in complex and variable procedures such as colorectal interventions. Due to their complexity and prolonged duration, colorectal procedures have been underrepresented in current research. At the same time, they pose a particularly interesting learning environment due to repetitive tissue manipulation, making them a promising entry point for autonomous, machine learning-driven support. Therefore, in this work, we introduce attachment anchors, a structured representation that encodes the local
geometric and mechanical relationships between tissue and its anatomical attachments in colorectal surgery. This representation reduces uncertainty in grasping point prediction by normalizing
surgical scenes into a consistent local reference frame. We demonstrate that attachment anchors can be predicted from laparoscopic images and incorporated into a grasping framework based on machine learning. Experiments on a
dataset of 90 colorectal surgeries demonstrate that attachment anchors improve grasping point prediction compared to image-only baselines. There are particularly strong gains in out-of-distribution settings, including unseen procedures and operating surgeons. These results suggest that attachment anchors are an effective intermediate representation for learning-based tissue manipulation in colorectal surgery.
Read the full abstract ▾
All authors Dennis N. Schneider, Lars Wagner, Daniel Rueckert, Dirk Wilhelm
Computer Vision
Physics Encoded Spatial and Temporal Generative Adversarial Network for Tropical Cyclone Image Super-resolution
Ruoyi Zhang, Jiawei Yuan, Lujia Ye et al. (+2 more)
High-resolution satellite imagery is indispensable for tracking the genesis, intensification, and trajectory of tropical cyclones (TCs). However, existing deep learning-based super-resolution (SR) methods often treat satellite image sequences as generic
videos, neglecting the underlying atmospheric physical laws governing cloud motion. To address this, we propose a Physics Encoded Spatial and Temporal Generative
Adversarial Network (PESTGAN) for TC image super-resolution. Specifically, we design a disentangled generator architecture incorporating a PhyCell module, which approximates the vorticity equation via constrained convolutions and encodes the resulting approximate physical dynamics as implicit latent representations to separate physical dynamics from visual textures. Furthermore, a dual-discriminator framework is introduced, employing a temporal discriminator to enforce motion consistency alongside spatial realism. Experiments on the Digital Typhoon
dataset for 4$\times$ upscaling demonstrate that PESTGAN establishes a better performance in structural fidelity and perceptual quality. While maintaining competitive pixel-wise accuracy compared to existing approaches, our method significantly excels in reconstructing meteorologically plausible cloud structures with superior physical fidelity.
Read the full abstract ▾
All authors Ruoyi Zhang, Jiawei Yuan, Lujia Ye, Runling Yu, Liling Zhao
Computer Vision · Cross-listed from cs.LG
Unified Latents (UL): How to train your latents
Jonathan Heek, Emiel Hoogeboom, Thomas Mensink et al. (+1 more)
We present Unified Latents (UL), a framework for learning latent representations that are jointly regularized by a
diffusion prior and decoded by a
diffusion model. By linking the encoder's output noise to the prior's minimum noise level, we obtain a simple training objective that provides a tight upper bound on the latent bitrate. On ImageNet-512, our approach achieves competitive FID of 1.4, with high reconstruction quality (PSNR) while requiring fewer training FLOPs than models trained on Stable
Diffusion latents. On Kinetics-600, we set a new state-of-the-art FVD of 1.3.
Read the full abstract ▾
All authors Jonathan Heek, Emiel Hoogeboom, Thomas Mensink, Tim Salimans
Computer Vision
EA-Swin: An Embedding-Agnostic Swin Transformer for AI-Generated Video Detection
Hung Mai, Loi Dinh, Duc Hai Nguyen et al. (+6 more)
Recent advances in foundation
video generators such as Sora2, Veo3, and other commercial systems have produced highly realistic synthetic
videos, exposing the limitations of existing detection methods that rely on shallow
embedding trajectories, image-based adaptation, or computationally heavy MLLMs. We propose EA-Swin, an
Embedding-Agnostic Swin
Transformer that models spatiotemporal dependencies directly on pretrained
video embeddings via a factorized windowed
attention design, making it compatible with generic ViT-style patch-based encoders. Moreover, we construct the EA-
Video dataset, a
benchmark dataset comprising 130K
videos that integrates newly collected samples with curated existing
datasets, covering diverse commercial and open-source generators and including unseen-generator splits for rigorous cross-distribution evaluation. Extensive experiments show that EA-Swin achieves 0.97-0.99 accuracy across major generators, outperforming prior SoTA methods (typically 0.8-0.9) by a margin of 5-20%, while maintaining strong generalization to unseen distributions, establishing a scalable and robust solution for modern AI-generated
video detection.
Read the full abstract ▾
All authors Hung Mai, Loi Dinh, Duc Hai Nguyen, Dat Do, Luong Doan, Khanh Nguyen Quoc, Huan Vu, Naeem Ul Islam, Tuan Do
Computer Vision
A Multi-modal Detection System for Infrastructure-based Freight Signal Priority
Ziyan Zhang, Chuheng Wei, Xuanpeng Zhao et al. (+6 more)
Freight vehicles approaching signalized intersections require reliable detection and motion estimation to support infrastructure-based Freight Signal Priority (FSP). Accurate and timely perception of vehicle type, position, and speed is essential for enabling effective priority control strategies. This paper presents the design, deployment, and evaluation of an infrastructure-based multi-modal freight vehicle detection system integrating
LiDAR and camera sensors. A hybrid sensing architecture is adopted, consisting of an intersection-mounted subsystem and a midblock subsystem, connected via wireless communication for synchronized data transmission. The perception pipeline incorporates both clustering-based and deep learning-based detection methods with Kalman filter tracking to achieve stable real-time performance.
LiDAR measurements are registered into geodetic reference frames to support lane-level localization and consistent vehicle tracking. Field evaluations demonstrate that the system can reliably monitor freight vehicle movements at high spatio-temporal resolution. The design and deployment provide practical insights for developing infrastructure-based sensing systems to support FSP applications.
Read the full abstract ▾
All authors Ziyan Zhang, Chuheng Wei, Xuanpeng Zhao, Siyan Li, Will Snyder, Mike Stas, Peng Hao, Kanok Boriboonsomsin, Guoyuan Wu
Computer Vision
Inferring Height from Earth Embeddings: First insights using Google AlphaEarth
Alireza Hamoudzadeh, Valeria Belloni and Roberta Ravanelli
This study investigates whether the geospatial and
multimodal features encoded in Earth
Embeddings can effectively guide deep learning (DL) regression models for regional surface height mapping. In particular, we focused on AlphaEarth
Embeddings at 10 m spatial resolution and evaluated their capability to support terrain height inference using a high-quality Digital Surface Model (DSM) as reference. U-Net and U-Net++ architectures were thus employed as lightweight convolutional decoders to assess how well the geospatial information distilled in the
embeddings can be translated into accurate surface height estimates. Both architectures achieved strong training performance (both with $R^2 = 0.97$), confirming that the
embeddings encode informative and decodable height-related signals. On the test set, performance decreased due to distribution shifts in height frequency between training and testing areas. Nevertheless, U-Net++ shows better generalization ($R^2 = 0.84$, median difference = -2.62 m) compared with the standard U-Net ($R^2 = 0.78$, median difference = -7.22 m), suggesting enhanced robustness to distribution mismatch. While the testing RMSE (approximately 16 m for U-Net++) and residual bias highlight remaining challenges in generalization, strong correlations indicate that the
embeddings capture transferable topographic patterns. Overall, the results demonstrate the promising potential of AlphaEarth
Embeddings to guide DL-based height mapping workflows, particularly when combined with spatially aware convolutional architectures, while emphasizing the need to address bias for improved regional transferability.
Read the full abstract ▾
Computer Vision
HiMAP: History-aware Map-occupancy Prediction with Fallback
Yiming Xu, Yi Yang, Hao Cheng et al. (+1 more)
Accurate motion
forecasting is critical for autonomous driving, yet most predictors rely on multi-object tracking (MOT) with identity association, assuming that objects are correctly and continuously tracked. When tracking fails due to, e.g., occlusion, identity switches, or missed detections, prediction quality degrades and safety risks increase. We present HiMAP, a tracking-free, trajectory prediction framework that remains reliable under MOT failures. HiMAP converts past detections into spatiotemporally invariant historical occupancy maps and introduces a historical query module that conditions on the current
agent state to iteratively retrieve
agent-specific history from unlabeled occupancy representations. The retrieved history is summarized by a temporal map
embedding and, together with the final query and map context, drives a DETR-style decoder to produce multi-modal future trajectories. This design lifts identity reliance, supports streaming inference via reusable encodings, and serves as a robust fallback when tracking is unavailable. On Argoverse~2, HiMAP achieves performance comparable to tracking-based methods while operating without IDs, and it substantially outperforms strong baselines in the no-tracking setting, yielding relative gains of 11% in FDE, 12% in ADE, and a 4% reduction in MR over a fine-tuned QCNet. Beyond aggregate metrics, HiMAP delivers stable forecasts for all
agents simultaneously without waiting for tracking to recover, highlighting its practical value for safety-critical autonomy. The code is available under: https://github.com/XuYiMing83/HiMAP.
Read the full abstract ▾
All authors Yiming Xu, Yi Yang, Hao Cheng, Monika Sester
Computer Vision
GASS: Geometry-Aware Spherical Sampling for Disentangled Diversity Enhancement in Text-to-Image Generation
Ye Zhu, Kaleb S. Newman, Johannes F. Lutzeyer et al. (+3 more)
Despite high semantic
alignment, modern text-to-image (T2I) generative models still struggle to synthesize diverse images from a given
prompt. In this work, we enhance the T2I diversity through a
geometric lens. Unlike most existing methods that rely primarily on
entropy-based guidance to increase sample dissimilarity, we introduce Geometry-Aware Spherical Sampling (GASS) to enhance diversity by explicitly controlling both
prompt-dependent and
prompt-independent sources of variation. Specifically, we decompose the diversity measure in CLIP
embeddings using two orthogonal directions: the text
embedding, which captures semantic variation related to the
prompt, and an identified orthogonal direction that captures
prompt-independent variation (e.g., backgrounds). Based on this decomposition, GASS increases the
geometric projection spread of generated image
embeddings along both axes and guides the T2I sampling process via expanded predictions along the generation trajectory. Our experiments on different frozen T2I backbones (U-Net and DiT,
diffusion and flow) and
benchmarks demonstrate the effectiveness of disentangled diversity enhancement with minimal impact on image fidelity and semantic
alignment.
Read the full abstract ▾
All authors Ye Zhu, Kaleb S. Newman, Johannes F. Lutzeyer, Adriana Romero-Soriano, Michal Drozdzal, Olga Russakovsky
Computer Vision
Yahong Wang, Juncheng Wu, Zhangkai Ni et al. (+6 more)
Multimodal large
language models (MLLMs) incur substantial inference cost due to the processing of hundreds of visual
tokens per image. Although
token pruning has proven effective for accelerating inference, determining when and where to prune remains largely heuristic. Existing approaches typically rely on static, empirically selected layers, which limit interpretability and transferability across models. In this work, we introduce a
matrix-
entropy perspective and identify an "
Entropy Collapse Layer" (ECL), where the information content of visual representations exhibits a sharp and consistent drop, which provides a principled criterion for selecting the
pruning stage. Building on this observation, we propose EntropyPrune, a novel
matrix-
entropy-guided
token pruning framework that quantifies the information value of individual visual
tokens and prunes redundant ones without relying on
attention maps. Moreover, to enable efficient computation, we exploit the spectral equivalence of dual Gram
matrices, reducing the complexity of
entropy computation and yielding up to a 64x theoretical speedup. Extensive experiments on diverse
multimodal benchmarks demonstrate that EntropyPrune consistently outperforms state-of-the-art
pruning methods in both accuracy and efficiency. On LLaVA-1.5-7B, our method achieves a 68.2% reduction in FLOPs while preserving 96.0% of the original performance. Furthermore, EntropyPrune generalizes effectively to high-resolution and
video-based models, highlighting the strong robustness and scalability in practical MLLM acceleration. The code will be publicly available at https://github.com/YahongWang1/EntropyPrune.
Read the full abstract ▾
All authors Yahong Wang, Juncheng Wu, Zhangkai Ni, Chengmei Yang, Yihang Liu, Longzhen Yang, Yuyin Zhou, Ying Wen, Lianghua He
Computer Vision · Cross-listed from cs.AI
Texo: Formula Recognition within 20M Parameters
Sicheng Mao
In this paper we present Texo, a minimalist yet highperformance formula recognition model that contains only 20 million parameters. By attentive design,
distillation and transfer of the vocabulary and the tokenizer, Texo achieves comparable performance to state-of-the-art models such as UniMERNet-T and PPFormulaNet-S, while reducing the model size by 80% and 65%, respectively. This enables real-time inference on consumer-grade hardware and even in-browser deployment. We also developed a web application to demonstrate the model capabilities and facilitate its usage for end users.
Read the full abstract ▾
Computer Vision
Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
Seulbi Lee and Sangheum Hwang
Large
Vision Language Models (LVLMs) have achieved remarkable progress, yet they often suffer from
language bias, producing answers without relying on visual evidence. While prior work attempts to mitigate this issue through decoding strategies, architectural modifications, or curated instruction data, they typically lack a quantitative measure of how much individual training samples or
tokens actually benefit from the image. In this work, we introduce Visual Information Gain (VIG), a perplexity-based metric that measures the reduction in prediction uncertainty provided by visual input. VIG enables fine-grained analysis at both sample and
token levels, effectively highlighting visually grounded elements such as colors, spatial relations, and attributes. Leveraging this, we propose a VIG-guided selective training scheme that prioritizes high-VIG samples and
tokens. This approach improves visual grounding and mitigates
language bias, achieving superior performance with significantly reduced supervision by focusing exclusively on visually informative samples and
tokens.
Read the full abstract ▾
Computer Vision
NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting
Jiwei Shan, Zeyu Cai, Yirui Li et al. (+5 more)
Visual simultaneous localization and mapping (V-SLAM) is a fundamental capability for autonomous perception and
navigation. However, endoscopic scenes violate the
rigidity assumption due to persistent soft-tissue deformations, creating a strong coupling ambiguity between camera ego-motion and intrinsic deformation. Although recent
monocular non-rigid SLAM methods have made notable progress, they often lack effective decoupling mechanisms and rely on sparse or low-fidelity scene representations, which leads to tracking drift and limited reconstruction quality. To address these limitations, we propose NRGS-SLAM, a
monocular non-rigid SLAM system for endoscopy based on 3D Gaussian Splatting. To resolve the coupling ambiguity, we introduce a deformation-aware 3D Gaussian map that augments each Gaussian primitive with a learnable deformation probability, optimized via a
Bayesian self-supervision strategy without requiring external non-
rigidity labels. Building on this representation, we design a deformable tracking module that performs robust coarse-to-fine pose estimation by prioritizing low-deformation regions, followed by efficient per-frame deformation updates. A carefully designed deformable mapping module progressively expands and refines the map, balancing representational capacity and computational efficiency. In addition, a unified robust
geometric loss incorporates external
geometric priors to mitigate the inherent ill-posedness of
monocular non-rigid SLAM. Extensive experiments on multiple public endoscopic
datasets demonstrate that NRGS-SLAM achieves more accurate camera pose estimation (up to 50% reduction in RMSE) and higher-quality photo-realistic reconstructions than state-of-the-art methods. Comprehensive ablation studies further validate the effectiveness of our key design choices. Source code will be publicly available upon paper acceptance.
Read the full abstract ▾
All authors Jiwei Shan, Zeyu Cai, Yirui Li, Yongbo Chen, Lijun Han, Yun-hui Liu, Hesheng Wang, Shing Shin Cheng
Computer Vision
BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning
Siyuan Liang, Yongcheng Jing, Yingjie Wang et al. (+3 more)
Research on
backdoor attacks against
multimodal contrastive learning models faces two key challenges: stealthiness and persistence. Existing methods often fail under strong detection or continuous fine-tuning, largely due to (1) cross-modal inconsistency that exposes trigger patterns and (2) gradient dilution at low poisoning rates that accelerates
backdoor forgetting. These coupled causes remain insufficiently modeled and addressed. We propose BadCLIP++, a unified framework that tackles both challenges. For stealthiness, we introduce a semantic-fusion QR micro-trigger that embeds imperceptible patterns near task-relevant regions, preserving clean-data statistics while producing compact trigger distributions. We further apply target-aligned subset selection to strengthen signals at low injection rates. For persistence, we stabilize trigger
embeddings via radius shrinkage and centroid
alignment, and stabilize model parameters through
curvature control and elastic weight consolidation, maintaining solutions within a low-
curvature wide basin resistant to fine-tuning. We also provide the first theoretical analysis showing that, within a trust region, gradients from clean fine-tuning and
backdoor objectives are co-directional, yielding a non-increasing upper bound on attack success degradation. Experiments demonstrate that with only 0.3% poisoning, BadCLIP++ achieves 99.99% attack success rate (ASR) in digital settings, surpassing baselines by 11.4 points. Across nineteen defenses, ASR remains above 99.90% with less than 0.8% drop in clean accuracy. The method further attains 65.03% success in physical attacks and shows robustness against watermark removal defenses.
Read the full abstract ▾
All authors Siyuan Liang, Yongcheng Jing, Yingjie Wang, Jiaxing Huang, Ee-chien Chang, Dacheng Tao
Computer Vision
B$^3$-Seg: Camera-Free, Training-Free 3DGS Segmentation via Analytic EIG and Beta-Bernoulli Bayesian Updates
Hiromichi Kamata, Samuel Arthur Munro and Fuminori Homma
Interactive 3D Gaussian Splatting (3DGS)
segmentation is essential for real-time editing of pre-reconstructed assets in film and game production. However, existing methods rely on predefined camera viewpoints, ground-truth labels, or costly retraining, making them impractical for low-latency use. We propose B$^3$-Seg (Beta-Bernoulli
Bayesian Segmentation for 3DGS), a fast and theoretically grounded method for open-vocabulary 3DGS
segmentation under camera-free and training-free conditions. Our approach reformulates
segmentation as sequential Beta-Bernoulli
Bayesian updates and actively selects the next view via analytic Expected Information Gain (EIG). This
Bayesian formulation guarantees the adaptive monotonicity and submodularity of EIG, which produces a greedy $(1{-}1/e)$ approximation to the optimal view sampling policy. Experiments on multiple
datasets show that B$^3$-Seg achieves competitive results to high-cost supervised methods while operating end-to-end
segmentation within a few seconds. The results demonstrate that B$^3$-Seg enables practical, interactive 3DGS
segmentation with provable information efficiency.
Read the full abstract ▾
Computer Vision
3D Scene Rendering with Multimodal Gaussian Splatting
Chi-Shiang Gau, Konstantinos D. Polyzos, Athanasios Bacharis et al. (+2 more)
3D scene reconstruction and rendering are core tasks in computer
vision, with applications spanning industrial monitoring, robotics, and autonomous driving. Recent advances in 3D Gaussian Splatting (GS) and its variants have achieved impressive rendering fidelity while maintaining high computational and
memory efficiency. However, conventional
vision-based GS pipelines typically rely on a sufficient number of camera views to initialize the Gaussian primitives and train their parameters, typically incurring additional processing cost during initialization while falling short in conditions where visual cues are unreliable, such as adverse weather, low illumination, or partial occlusions. To cope with these challenges, and motivated by the robustness of
radio-frequency (RF) signals to weather, lighting, and occlusions, we introduce a
multimodal framework that integrates RF sensing, such as automotive radar, with GS-based rendering as a more efficient and robust alternative to
vision-only GS rendering. The proposed approach enables efficient depth prediction from only sparse RF-based depth measurements, yielding a high-quality 3D point cloud for initializing Gaussian functions across diverse GS architectures. Numerical tests demonstrate the merits of judiciously incorporating RF sensing into GS pipelines, achieving high-fidelity 3D scene rendering driven by RF-informed structural accuracy.
Read the full abstract ▾
All authors Chi-Shiang Gau, Konstantinos D. Polyzos, Athanasios Bacharis, Saketh Madhuvarasu, Tara Javidi
Computer Vision · Cross-listed from cs.RO
Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success
Varun Burde, Pavel Burget and Torsten Sattler
3D reconstruction serves as the foundational layer for numerous robotic perception tasks, including 6D object pose estimation and grasp pose generation. Modern 3D reconstruction methods for objects can produce visually and geometrically impressive meshes from multi-view images, yet standard
geometric evaluations do not reflect how reconstruction quality influences downstream tasks such as robotic manipulation performance. This paper addresses this gap by introducing a large-scale, physics-based
benchmark that evaluates 6D pose
estimators and 3D mesh models based on their functional efficacy in grasping. We analyze the impact of model fidelity by generating grasps on various reconstructed 3D meshes and executing them on the ground-truth model, simulating how grasp poses generated with an imperfect model affect interaction with the real object. This assesses the combined impact of pose error, grasp robustness, and
geometric inaccuracies from 3D reconstruction. Our results show that reconstruction artifacts significantly decrease the number of grasp pose candidates but have a negligible effect on grasping performance given an accurately estimated pose. Our results also reveal that the relationship between grasp success and pose error is dominated by spatial error, and even a simple translation error provides insight into the success of the grasping pose of symmetric objects. This work provides insight into how perception systems relate to object manipulation using
robots.
Read the full abstract ▾
Computer Vision
ComptonUNet: A Deep Learning Model for GRB Localization with Compton Cameras under Noisy and Low-Statistic Conditions
Shogo Sato, Kazuo Tanaka, Shojun Ogasawara et al. (+4 more)
Gamma-ray bursts (GRBs) are among the most energetic transient phenomena in the universe and serve as powerful probes for high-energy astrophysical processes. In particular, faint GRBs originating from a distant universe may provide unique insights into the early stages of
star formation. However, detecting and localizing such weak sources remains challenging owing to low
photon statistics and substantial background noise. Although recent machine learning models address individual aspects of these challenges, they often struggle to balance the trade-off between statistical robustness and noise suppression. Consequently, we propose ComptonUNet, a hybrid deep learning framework that jointly processes raw data and reconstructs images for robust GRB localization. ComptonUNet was designed to operate effectively under conditions of limited
photon statistics and strong background contamination by combining the statistical efficiency of direct reconstruction models with the denoising capabilities of image-based architectures. We perform realistic simulations of GRB-like events embedded in background environments representative of low-Earth orbit missions to evaluate the performance of ComptonUNet. Our results demonstrate that ComptonUNet significantly outperforms existing approaches, achieving improved localization accuracy across a wide range of low-statistic and high-background scenarios.
Read the full abstract ▾
All authors Shogo Sato, Kazuo Tanaka, Shojun Ogasawara, Kazuki Yamamoto, Kazuhiko Murasaki, Ryuichi Tanida, Jun Kataoka
Computer Vision
Cross Pseudo Labeling For Weakly Supervised Video Anomaly Detection
Dayeon Lee, Donghyeong Kim, Chaewon Park et al. (+2 more)
Weakly supervised
video anomaly detection aims to detect anomalies and identify abnormal categories with only
video-level labels. We propose CPL-VAD, a dual-branch framework with cross pseudo labeling. The binary anomaly detection branch focuses on snippet-level anomaly localization, while the category classification branch leverages
vision-
language alignment to recognize abnormal event categories. By exchanging pseudo labels, the two branches transfer complementary strengths, combining temporal precision with semantic discrimination. Experiments on XD-Violence and UCF-Crime demonstrate that CPL-VAD achieves state-of-the-art performance in both anomaly detection and abnormal category classification.
Read the full abstract ▾
All authors Dayeon Lee, Donghyeong Kim, Chaewon Park, Sungmin Woo, Sangyoun Lee
Computer Vision · Cross-listed from cs.LG
Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression
Akira Sakai and Yuma Ichikawa
Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across
Transformers, CNNs, and MLPs, learned sign
matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rademacher baseline. This randomness gives rise to the lower bound of sub-bit model compression – the one-bit wall. Despite this apparent randomness, most weights retain their initialization signs; flips primarily occur via rare near-zero boundary crossings, suggesting that sign-pattern randomness is largely inherited from initialization. We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise. Under bounded updates and a rare re-entry condition into a small neighborhood of zero, the number of effective sign flips exhibits a
geometric tail. Building on this mechanism, we introduce a from-scratch low-rank sign-template training method that prevents the emergence of this one-bit wall.
Read the full abstract ▾
Computer Vision
Cholec80-port: A Geometrically Consistent Trocar Port Segmentation Dataset for Robust Surgical Scene Understanding
Shunsuke Kikuchi, Atsushi Kouno and Hiroki Matsuzaki
Trocar ports are camera-fixed, pseudo-static structures that can persistently occlude laparoscopic views and attract disproportionate feature points due to specular, textured surfaces. This makes ports particularly detrimental to geometry-based downstream pipelines such as image stitching, 3D reconstruction, and visual SLAM, where dynamic or non-anatomical outliers degrade
alignment and tracking stability. Despite this practical importance, explicit port labels are rare in public
surgical datasets, and existing annotations often violate
geometric consistency by masking the central lumen (opening), even when anatomical regions are visible through it. We present Cholec80-port, a high-fidelity trocar port
segmentation dataset derived from Cholec80, together with a rigorous standard operating procedure (SOP) that defines a port-sleeve mask excluding the central opening. We additionally cleanse and unify existing public
datasets under the same SOP. Experiments demonstrate that geometrically consistent annotations substantially improve cross-
dataset robustness beyond what
dataset size alone provides.
Read the full abstract ▾