Computer Vision · Cross-listed from cs.LG
IGLU: The Integrated Gaussian Linear Unit Activation Function
Mingi Kang, Zai Yang and Jeova Farias Sales Rocha Neto
Activation functions are fundamental to deep neural networks, governing gradient flow, optimization stability, and representational capacity. Within historic deep architectures, while ReLU has been the dominant choice for the activation function, modern
transformer-based models increasingly are adopting smoother alternatives such as GELU and other self-gated alternatives. Despite their empirical success, the mathematical relationships among these functions and the principles underlying their effectiveness remains only partially understood. We introduce IGLU, a parametric activation function derived as a scale mixture of GELU gates under a half-normal mixing distribution. This derivation yields a closed-form expression whose gating component is exactly the Cauchy CDF, providing a principled one-parameter family that continuously interpolates between identity-like and ReLU-like behavior via a single sharpness parameter $\sigma$. Unlike GELU's Gaussian gate, IGLU's heavy-tailed Cauchy gate decays polynomially in the negative tail, guaranteeing non-zero gradients for all finite inputs and offering greater robustness to vanishing gradients. We further introduce IGLU-Approx, a computationally efficient rational approximation of IGLU expressed entirely in terms of ReLU operations that eliminates transcendental function evaluation. Through evaluations on CIFAR-10, CIFAR-100, and WikiText-103 across ResNet-20, ViT-Tiny, and GPT-2 Small, IGLU achieves competitive or superior performance on both
vision and
language datasets against ReLU and GELU baselines, with IGLU-Approx recovering this performance at substantially reduced computational cost. In particular, we show that employing a heavy-tailed gate leads to considerable performance gains in heavily imbalanced classification
datasets.
Read the full abstract ▾
Computer Vision
ColonSplat: Reconstruction of Peristaltic Motion in Colonoscopy with Dynamic Gaussian Splatting
Weronika Smolak-Dyżewska, Joanna Kaleta, Diego Dall'Alba et al. (+1 more)
Accurate 3D reconstruction of colonoscopy data, accounting for complex peristaltic movements, is crucial for advanced
surgical navigation and retrospective diagnostics. While recent novel view synthesis and 3D reconstruction methods have demonstrated remarkable success in general endoscopic scenarios, they struggle in the highly constrained environment of the colon. Due to the limited field of view of a camera moving through an actively deforming tubular structure, existing endoscopic methods reconstruct the colon appearance only for initial camera trajectory. However, the underlying anatomy remains largely static; instead of updating Gaussians' spatial coordinates (xyz), these methods encode deformation through either rotation, scale or opacity adjustments. In this paper, we first present a
benchmark analysis of state-of-the-art dynamic endoscopic methods for realistic colonoscopic scenes, showing that they fail to model true anatomical motion. To enable rigorous evaluation of global reconstruction quality, we introduce DynamicColon, a synthetic
dataset with ground-truth point clouds at every timestep. Building on these insights, we propose ColonSplat, a dynamic Gaussian Splatting framework that captures peristaltic-like motion while preserving global
geometric consistency, achieving superior
geometric fidelity on C3VDv2 and DynamicColon
datasets. Project page: https://wmito.github.io/ColonSplat
Read the full abstract ▾
All authors Weronika Smolak-Dyżewska, Joanna Kaleta, Diego Dall'Alba, Przemysław Spurek
Computer Vision
An Extended Topological Model For High-Contrast Optical Flow
Brad Turow and Jose A. Perea
In this paper, we identify low-dimensional models for dense core subsets in the space of $3\times 3$ high-contrast optical flow patches sampled from the Sintel
dataset. In particular, we leverage the theory of approximate and discrete circle bundles to identify a 3-
manifold whose boundary is a previously proposed optical flow
torus, together with disjoint circles corresponding to pairs of binary step-edge range image patches. The 3-
manifold model we introduce provides an explanation for why the previously-proposed
torus model could not be verified with direct methods (e.g., a straightforward persistent homology computation). We also demonstrate that nearly all optical flow patches in the top 1 percent by contrast norm are found near the family of binary step-edge circles described above, rather than the optical flow
torus, and that these frequently occurring patches are concentrated near motion boundaries (which are of particular importance for computer
vision tasks such as object
segmentation and tracking). Our findings offer insights on the subtle interplay between
topology and geometry in inference for visual data.
Read the full abstract ▾
Computer Vision
Active View Selection with Perturbed Gaussian Ensemble for Tomographic Reconstruction
Yulun Wu, Ruyi Zha, Wei Cao et al. (+3 more)
Sparse-view computed
tomography (CT) is critical for reducing radiation exposure to patients. Recent advances in radiative 3D Gaussian Splatting (3DGS) have enabled fast and accurate sparse-view CT reconstruction. Despite these algorithmic advancements, practical reconstruction fidelity remains fundamentally bounded by the quality of the captured data, raising the crucial yet underexplored problem of X-ray active view selection. Existing active view selection methods are primarily designed for natural-light scenes and fail to capture the unique
geometric ambiguities and physical attenuation properties inherent in X-ray imaging. In this paper, we present Perturbed Gaussian Ensemble, an active view selection framework that integrates uncertainty modeling with sequential decision-making, tailored for X-ray Gaussian Splatting. Specifically, we identify low-density Gaussian primitives that are likely to be uncertain and apply
stochastic density scaling to construct an ensemble of plausible Gaussian density fields. For each candidate projection, we measure the structural variance of the ensemble predictions and select the one with the highest variance as the next best view. Extensive experimental results on arbitrary-trajectory CT
benchmarks demonstrate that our density-guided perturbation strategy effectively eliminates
geometric artifacts and consistently outperforms existing baselines in progressive tomographic reconstruction under unified view selection protocols.
Read the full abstract ▾
All authors Yulun Wu, Ruyi Zha, Wei Cao, Yingying Li, Yuanhao Cai, Yaoyao Liu
Computer Vision
MotionBits: Video Segmentation through Motion-Level Analysis of Rigid Bodies
Howard H. Qian, Kejia Ren, Yu Xiang et al. (+2 more)
Rigid bodies constitute the smallest manipulable elements in the real world, and understanding how they physically interact is fundamental to
embodied reasoning and robotic manipulation. Thus, accurate detection,
segmentation, and tracking of moving rigid bodies is essential for enabling
reasoning modules to interpret and act in diverse environments. However, current
segmentation models trained on semantic grouping are limited in their ability to provide meaningful interaction-level cues for completing
embodied tasks. To address this gap, we introduce MotionBit, a novel concept that, unlike prior formulations, defines the smallest unit in motion-based
segmentation through kinematic spatial twist equivalence, independent of semantics. In this paper, we contribute (1) the MotionBit concept and definition, (2) a hand-labeled
benchmark, called MoRiBo, for evaluating moving rigid-body
segmentation across robotic manipulation and human-in-the-wild
videos, and (3) a learning-free graph-based MotionBits
segmentation method that outperforms state-of-the-art
embodied perception methods by 37.3% in macro-averaged mIoU on the MoRiBo
benchmark. Finally, we demonstrate the effectiveness of MotionBits
segmentation for downstream
embodied reasoning and manipulation tasks, highlighting its importance as a fundamental primitive for understanding physical interactions.
Read the full abstract ▾
All authors Howard H. Qian, Kejia Ren, Yu Xiang, Vicente Ordonez, Kaiyu Hang
Computer Vision
Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models
Md Ashikur Rahman, Md Arifur Rahman, Niamul Hassan Samin et al. (+2 more)
We uncover a behavioral law of long-horizon
vision-
language models: models that maintain temporally grounded beliefs generalize better. Standard
benchmarks measure only final-answer accuracy, which obscures how models use visual information; a model can guess correctly while its step-by-step
reasoning is entirely unanchored to the visual input. We formalize this as behavioral faithfulness over long horizons, an empirically measurable property that quantifies whether a model's intermediate
reasoning remains consistent with the evolving visual state. Across eight models on three long-horizon
benchmarks, we demonstrate that temporal grounding quality is a leading indicator of robustness: the Step Grounding Rate (SGR) predicts out-of-distribution retention with $r = 0.83$ (permutation test $p = 0.003$), a relationship that holds within capacity-matched models and cannot be explained by scale or in-distribution accuracy. Critically, grounding quality varies by up to 10.8 percentage points within parameter-matched 7B models despite similar accuracy, revealing it as an independent axis of model capability. Multiple robustness checks confirm the signal reflects genuine visual reliance: counterfactual traces drop SGR by 26–41 percentage points, cross-architecture verifiers agree at $\rho = 0.96$, random
reasoning scores near chance ($\sim 18\%$), and the predictor remains strong even without explicit
reasoning disclosure ($r = 0.78$).
Read the full abstract ▾
All authors Md Ashikur Rahman, Md Arifur Rahman, Niamul Hassan Samin, Abdullah Ibne Hanif Arean, Juena Ahmed Noshin
Computer Vision
A Hybrid Machine Learning Model for Cerebral Palsy Detection
Karan Kumar Singh, Nikita Gajbhiye and Gouri Sankar Mishra
The development of effective treatments for Cerebral Palsy (CP) can begin with the early identification of affected children while they are still in the early stages of the disorder. Pathological issues in the
brain can be better diagnosed with the use of one of many medical imaging techniques.
Magnetic Resonance Imaging (MRI) has revolutionized medical imaging with its unparalleled image resolution. A unique Machine Learning (ML) model that was built to identify CP disorder is presented in this paper. The model is intended to assist in the early diagnosis of CP in newborns. In this study, the
brain MRI images
dataset was first collected, and then the preprocessing techniques were applied to this
dataset to make it ready for use in the proposed model. Following this, the proposed model was constructed by combining three CNN models, specifically VGG 19, Efficient-Net, and the ResNet50 model, to extract features from the image. Following this, a Bi-LSTM was utilized as a classifier to determine whether or not CP was present, and finally, the proposed model was employed for training and testing. The results show that the proposed model achieved an accuracy of 98.83%, which is higher than VGG-19 (96.79%), Efficient-Net (97.29%), and VGG-16 (97.50%).. When the suggested model is compared to other models that have been pre-trained in the past, the accuracy scores seem to be much higher.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
HiDE: Hierarchical Dictionary-Based Entropy Modeling for Learned Image Compression
Haoxuan Xiong, Yuanyuan Xu, Kun Zhu et al. (+2 more)
Learned image compression (LIC) has achieved remarkable coding efficiency, where
entropy modeling plays a pivotal role in minimizing bitrate through informative priors. Existing methods predominantly exploit internal contexts within the input image, yet the rich external priors embedded in large-scale training data remain largely underutilized. Recent advances in dictionary-based
entropy models have demonstrated that incorporating external priors can substantially enhance compression performance. However, current approaches organize heterogeneous external priors within a single-level dictionary, resulting in imbalanced utilization and limited representational capacity. Moreover, effective
entropy modeling requires not only expressive priors but also a parameter estimation network capable of interpreting them. To address these challenges, we propose HiDE, a Hierarchical Dictionary-based
Entropy modeling framework for learned image compression. HiDE decomposes external priors into global structural and local detail dictionaries with cascaded
retrieval, enabling structured and efficient utilization of external information. Moreover, a context-aware parameter
estimator with parallel multi-receptive-field design is introduced to adaptively exploit heterogeneous contexts for accurate conditional probability estimation. Experimental results show that HiDE achieves 18.5%, 21.99%, and 24.01% BD-rate savings over VTM-12.1 on the Kodak, CLIC, and Tecnick
datasets, respectively.
Read the full abstract ▾
All authors Haoxuan Xiong, Yuanyuan Xu, Kun Zhu, Yiming Wang, Baoliu Ye
Computer Vision
EarthBridge: A Solution for 4th Multi-modal Aerial View Image Challenge Translation Track
Zhenyuan Chen, Guanyuan Shen and Feng Zhang
Cross-modal image-to-image translation among Electro-Optical (EO), Infrared (IR), and Synthetic Aperture Radar (SAR) sensors is essential for comprehensive multi-modal aerial-view analysis. However, translating between these modalities is notoriously difficult due to their distinct electromagnetic signatures and
geometric characteristics. This paper presents EarthBridge, a high-fidelity translation framework developed for the 4th Multi-modal Aerial View Image Challenge – Translation (MAVIC-T). We explore two distinct methodologies:
Diffusion Bridge Implicit Models (DBIM), which we generalize using non-Markovian bridge processes for high-quality deterministic sampling, and Contrastive Unpaired Translation (CUT), which utilizes contrastive learning for structural consistency. Our EarthBridge framework employs a channel-concatenated UNet denoiser trained with Karras-weighted bridge scalings and a specialized "booting noise" initialization to handle the inherent ambiguity in cross-modal mappings. We evaluate these methods across all four challenge tasks (SAR$\rightarrow$EO, SAR$\rightarrow$RGB, SAR$\rightarrow$IR, RGB$\rightarrow$IR), achieving superior spatial detail and spectral accuracy. Our solution achieved a composite score of 0.38, securing the second position on the MAVIC-T leaderboard. Code is available at https://github.com/Bili-Sakura/EarthBridge-Preview.
Read the full abstract ▾
Computer Vision
XMACNet: An Explainable Lightweight Attention based CNN with Multi Modal Fusion for Chili Disease Classification
Tapon Kumer Ray, Rajkumar Y, Shalini R et al. (+3 more)
Plant disease classification via imaging is a critical task in precision agriculture. We propose XMACNet, a novel light-weight Convolutional Neural Network (CNN) that integrates self-
attention and multi-modal fusion of visible imagery and vegetation indices for chili disease detection. XMACNet uses an EfficientNetV2S backbone enhanced by a self-
attention module and a fusion branch that processes both RGB images and computed vegetation index maps (NDVI, NPCI, MCARI). We curated a new
dataset of 12,000 chili leaf images across six classes (five disease types plus healthy), augmented synthetically via StyleGAN to mitigate data scarcity. Trained on this
dataset, XMACNet achieves high accuracy, F1-score, and AUC, outperforming baseline models such as ResNet-50, MobileNetV2, and a Swin
Transformer variant. Crucially, XMACNet is explainable: we use Grad-CAM++ and SHAP to visualize and quantify the models focus on disease features. The models compact size and fast inference make it suitable for edge deployment in real-world farming scenarios.
Read the full abstract ▾
All authors Tapon Kumer Ray, Rajkumar Y, Shalini R, Srigayathri K, Jayashree S, Lokeswari P
Computer Vision
ButterflyViT: 354$\times$ Expert Compression for Edge Vision Transformers
Aryan Karmore
Deploying sparse Mixture of Experts(MoE)
Vision Transformers remains a challenge due to linear expert
memory scaling. Linear
memory scaling stores $N$ independent expert weight
matrices requiring $\mathcal{O}(N_E \cdot d^2)$
memory, which exceeds edge devices
memory budget. Current compression methods like quantization,
pruning and low-rank factorization reduce constant factors but leave the scaling bottleneck unresolved. We introduce ButterflyViT, a method that treats experts not as independent weight
matrices but as
geometric reorientations of a unified shared quantized substrate. Diversity among experts arises from viewing different angles of shared capacity, not from redundant storage. By applying learned rotations to a shared ternary prototype, each expert yields $\mathcal{O}(d_{\text{model}} \cdot d_{\text{ff}} + N_E \cdot n_\ell \cdot d)$
memory which is sub-linear in the number of experts. To address the unique challenges of
vision, a spatial smoothness regulariser is introduced that penalises routing irregularities between adjacent patch
tokens, turning patch correlation into a training signal. Across image classification tasks on CIFAR-100, ButterflyViT achieves 354$\times$
memory reduction at 64 experts with negligible accuracy loss. ButterflyViT allows multiple experts to fit on edge-constrained devices showing that
geometric parameterization breaks linear scaling.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Heterogeneous Decentralized Diffusion Models
Zhiying Jiang, Raihan Seraj, Marcos Villagra et al. (+1 more)
Training frontier-scale
diffusion models often requires substantial computational resources concentrated in tightly-coupled clusters, limiting participation to well-resourced institutions. While Decentralized
Diffusion Models (DDM) enable training multiple experts in isolation, existing approaches require 1176 GPU-days and homogeneous training objectives across all experts. We present an efficient framework that dramatically reduces resource requirements while supporting heterogeneous training objectives. Our approach combines three key contributions: (1) a heterogeneous decentralized training paradigm that allows experts to use different objectives (DDPM and Flow Matching), unified at inference time without any retraining; (2) pretrained checkpoint conversion from ImageNet-DDPM to Flow Matching objectives, accelerating convergence and enabling initialization without objective-specific pretraining; and (3) PixArt-$\alpha$'s efficient AdaLN-Single architecture, reducing parameters while maintaining quality. Experiments on LAION-Aesthetics show that, relative to the training scale reported for prior DDM work, our approach reduces the compute by 16$\times$ and data by 14$\times$. Under aligned inference settings, our heterogeneous configuration achieves better FID and higher intra-
prompt diversity than the homogeneous baseline. By eliminating synchronization requirements and enabling mixed DDPM/FM objectives, our framework makes decentralized generative model training accessible to contributors with single GPUs requiring only 24–48GB VRAM.
Read the full abstract ▾
All authors Zhiying Jiang, Raihan Seraj, Marcos Villagra, Bidhan Roy
Computer Vision
Vessel-Aware Deep Learning for OCTA-Based Detection of AMD
Margalit G. Mitzner, Moinak Bhattacharya, Zhilin Zou et al. (+2 more)
Age-related macular degeneration (AMD) is characterized by early micro-vascular alterations that can be captured non-invasively using optical coherence
tomography angiography (OCTA), yet most deep learning (DL) models rely on global features and fail to exploit clinically meaningful vascular biomarkers. We introduce an external multiplicative
attention framework that incorporates vessel-specific tortuosity maps and vasculature dropout maps derived from arteries, veins, and capillaries. These biomarker maps are generated from vessel
segmentations and smoothed across multiple spatial scales to highlight coherent patterns of vascular remodeling and capillary rarefaction. Tortuosity reflects abnormalities in vessel geometry linked to impaired auto-regulation, while dropout maps capture localized perfusion deficits that precede structural retinal damage. The maps are fused with the OCTA projection to guide a deep classifier toward physiologically relevant regions. Arterial tortuosity provided the most consistent discriminative value, while capillary dropout maps performed best among density-based variants, especially at larger smoothing scales. Our proposed method offers interpretable insights aligned with known AMD pathophysiology.
Read the full abstract ▾
All authors Margalit G. Mitzner, Moinak Bhattacharya, Zhilin Zou, Chao Chen, Prateek Prasanna
Computer Vision
HERO: Hierarchical Embedding-Refinement for Open-Vocabulary Temporal Sentence Grounding in Videos
Tingting Han, Xinsong Tao, Yufei Yin et al. (+3 more)
Temporal Sentence Grounding in
Videos (TSGV) aims to temporally localize segments of a
video that correspond to a given natural
language query. Despite recent progress, most existing TSGV approaches operate under closed-vocabulary settings, limiting their ability to generalize to real-world queries involving novel or diverse linguistic expressions. To bridge this critical gap, we introduce the Open-Vocabulary TSGV (OV-TSGV) task and construct the first dedicated
benchmarks–Charades-OV and ActivityNet-OV–that simulate realistic vocabulary shifts and paraphrastic variations. These
benchmarks facilitate systematic evaluation of model generalization beyond seen training concepts. To tackle OV-TSGV, we propose HERO(Hierarchical
Embedding-Refinement for Open-Vocabulary grounding), a unified framework that leverages hierarchical linguistic
embeddings and performs parallel cross-modal refinement. HERO jointly models multi-level semantics and enhances
video-
language alignment via semantic-guided visual filtering and contrastive masked text refinement. Extensive experiments on both standard and open vocabulary
benchmarks demonstrate that HERO consistently surpasses state-of-the-art methods, particularly under open-vocabulary scenarios, validating its strong generalization capability and underscoring the significance of OV-TSGV as a new research direction.
Read the full abstract ▾
All authors Tingting Han, Xinsong Tao, Yufei Yin, Min Tan, Sicheng Zhao, Zhou Yu
Computer Vision
AWPD: Frequency Shield Network for Agnostic Watermark Presence Detection
Xiang Ao, Yilin Du, Zidan Wang et al. (+2 more)
Invisible watermarks, as an essential technology for image copyright protection, have been widely deployed with the rapid development of social media and AIGC. However, existing invisible watermark detection heavily relies on prior knowledge of specific
algorithms, leading to limited detection capabilities for “unknown watermarks” in open environments. To this end, we propose a novel task named Agnostic Watermark Presence Detection (AWPD), which aims to identify whether an image carries a copyright mark without requiring decoding information. We construct the UniFreq-100K
dataset, comprising large-scale samples across various invisible watermark
embedding algorithms. Furthermore, we propose the Frequency Shield Network (FSNet). This model deploys an Adaptive Spectral Perception Module (ASPM) in the shallow layers, utilizing learnable frequency gating to dynamically amplify high-frequency watermark signals while suppressing low-frequency semantics. In the deep layers, the network introduces Dynamic Multi-Spectral
Attention (DMSA) combined with tri-stream extremum pooling to deeply mine watermark energy anomalies, forcing the model to precisely focus on sensitive frequency bands. Extensive experiments demonstrate that FSNet exhibits superior zero-shot detection capabilities on the AWPD task, outperforming existing baseline models. Code and
datasets will be released upon acceptance.
Read the full abstract ▾
All authors Xiang Ao, Yilin Du, Zidan Wang, Mengru Chen, Siyang Lu
Computer Vision
Nikita Kisel, Illia Volkov, Klara Janouskova et al. (+1 more)
Multimodal Large
Language Model (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies comparing MLLMs with supervised and
Vision-
Language Models (VLMs) report conflicting conclusions, and we show these conflicts stem from protocols that either inflate or underestimate performance. Across the most common evaluation protocols, we identify and fix key issues: model outputs that fall outside the provided class list and are discarded, inflated results from weak multiple-choice distractors, and open-world setting that underperforms only due to poor output mapping. We additionally quantify the impact of commonly overlooked design choices - batch size, image ordering, and text encoder selection - showing they substantially affect accuracy. Evaluating on ReGT, our multilabel reannotation of 625 ImageNet-1k classes, reveals that MLLMs benefit most from corrected labels (up to +10.8%), substantially narrowing the perceived gap with supervised models. Much of the reported MLLM underperformance on classification is thus an artifact of noisy ground truth and flawed evaluation protocol rather than genuine model deficiency. Models less reliant on supervised training signals prove most sensitive to annotation quality. Finally, we show that MLLMs can assist human annotators: in a controlled case study, annotators confirmed or integrated MLLM predictions in approximately 50% of difficult cases, demonstrating their potential for large-scale
dataset curation. This work is part of the Aiming for Perfect ImageNet-1k project, see https://klarajanouskova.github.io/ImageNet/.
Read the full abstract ▾
All authors Nikita Kisel, Illia Volkov, Klara Janouskova, Jiri Matas
Computer Vision
Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
Lijiang Li, Zuwei Long, Yunhang Shen et al. (+6 more)
While recent
multimodal large
language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete
diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for
multimodal systems. Drawing inspiration from these pioneering studies, we introduce Omni-
Diffusion, the first any-to-any
multimodal language model built entirely on mask-based discrete
diffusion models, which unifies understanding and generation across text,
speech, and images. Omni-
Diffusion employs a unified mask-based discrete
diffusion model to directly capture the joint distribution over discrete
multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of
benchmarks, our method outperforms or performs on par with existing
multimodal systems that process two or more modalities, highlighting the significant promise of
diffusion models in powering the next generation of
multimodal foundation models. Project webpage: https://omni-
diffusion.github.io.
Read the full abstract ▾
All authors Lijiang Li, Zuwei Long, Yunhang Shen, Heting Gao, Haoyu Cao, Xing Sun, Caifeng Shan, Ran He, Chaoyou Fu
Computer Vision
BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations
Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen et al. (+1 more)
The integration of Large
Language Models (LLMs) into autonomous driving has attracted growing interest for their strong
reasoning and semantic understanding abilities, which are essential for handling complex decision-making and long-tail scenarios. However, existing methods typically feed LLMs with
tokens from multi-view and multi-frame images independently, leading to redundant computation and limited spatial consistency. This separation in visual processing hinders accurate 3D spatial
reasoning and fails to maintain
geometric coherence across views. On the other hand, Bird's-Eye View (BEV) representations learned from geometrically annotated tasks (e.g., object detection) provide spatial structure but lack the semantic richness of foundation
vision encoders. To bridge this gap, we propose BEVLM, a framework that connects a spatially consistent and semantically distilled BEV representation with LLMs. Through extensive experiments, we show that BEVLM enables LLMs to reason more effectively in cross-view driving scenes, improving accuracy by 46.0%, by leveraging BEV features as unified inputs. Furthermore, by distilling semantic knowledge from LLMs into BEV representations, BEVLM significantly improves closed-loop end-to-end driving performance in safety-critical scenarios across UniAD and VAD, with gains of up to 28.2%.
Read the full abstract ▾
All authors Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen, Sihao Ding
Computer Vision
SCOPE: Scene-Contextualized Incremental Few-Shot 3D Segmentation
Vishal Thengane, Zhaochong An, Tianjin Huang et al. (+5 more)
Incremental Few-Shot (IFS)
segmentation aims to learn new categories over time from only a few annotations. Although widely studied in 2D, it remains underexplored for 3D point clouds. Existing methods suffer from catastrophic forgetting or fail to learn discriminative prototypes under sparse supervision, and often overlook a key cue: novel categories frequently appear as unlabelled background in base-training scenes. We introduce SCOPE (Scene-COntextualised Prototype Enrichment), a plug-and-play background-guided prototype enrichment framework that integrates with any prototype-based 3D
segmentation method. After base training, a class-agnostic
segmentation model extracts high-confidence pseudo-instances from background regions to build a prototype pool. When novel classes arrive with few labelled samples, relevant background prototypes are retrieved and fused with few-shot prototypes to form enriched representations without retraining the backbone or adding parameters. Experiments on ScanNet and S3DIS show that SCOPE achieves SOTA performance, improving novel-class IoU by up to 6.98% and 3.61%, and mean IoU by 2.25% and 1.70%, respectively, while maintaining low forgetting. Code is available https://github.com/Surrey-UP-Lab/SCOPE.
Read the full abstract ▾
All authors Vishal Thengane, Zhaochong An, Tianjin Huang, Son Lam Phung, Abdesselam Bouzerdoum, Lu Yin, Na Zhao, Xiatian Zhu
Computer Vision
Alejandra Perez, Anita Rau, Busisiwe Mlambo et al. (+9 more)
A capable
surgical AI model should reason from visual evidence; yet, many existing
surgical benchmarks can be partially solved without visual input, as models may exploit procedural priors or common anatomy-instrument interactions to infer answers from
language alone. To address this, we introduce SUREON, a
video question-answering (QA)
dataset and
benchmark designed to evaluate visually grounded
surgical reasoning. Our pipeline generates QA pairs from narrated
surgical academic
videos, refines incorrect options using multiple models to reduce linguistic biases, and uses trained
adversarial guessers to filter out questions answerable without visual evidence. The resulting
dataset contains 76K training pairs covering 168 procedures and 10 specialties, alongside a
benchmark of 340 questions across 10 categories verified by expert surgeons. Benchmarking leading commercial and open-source models shows that
surgical visual
reasoning remains largely unsolved–no model exceeds 53% accuracy and even expert surgeons reach only 64.9%. We also introduce SUREON-VLM, a baseline trained on our curated data that avoids
language shortcuts, outperforms existing open-source models on SUREON, and matches larger specialist models on multiple standard
surgical benchmarks. Our training ablations show that distractor refinement and blind curation strengthen visual grounding, increasing the
multimodal gain from 8.7 to 29.1 percentage points while reducing the training set from 176.6K to 76K filtered samples. SUREON thus offers both a verified testbed for
surgical visual
reasoning and training data that strengthens visual grounding.
Read the full abstract ▾
All authors Alejandra Perez, Anita Rau, Busisiwe Mlambo, Lee White, Aina Brunet, Juan Verde, Juan Caicedo, Joël L. Lavanchy, Adam Schmidt, Chinedu Nwoye, Omid Mohareri, Muhammad Abdullah Jamal
Computer Vision
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
Boqiang Zhang, Lei Ke, Ruihan Yang et al. (+5 more)
Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and
robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on
vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM
reasoning. To address this issue, we present Penguin-VL, whose
vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for
multimodal understanding. Across various image and
video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical
reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective
video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex
reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: https://github.com/tencent-ailab/Penguin-VL
Read the full abstract ▾
All authors Boqiang Zhang, Lei Ke, Ruihan Yang, Qi Gao, Tianyuan Qu, Rossell Chen, Dong Yu, Leoweiliang
Computer Vision
EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinking
Fangrui Zhu, Yunfeng Xi, Jianmo Ni et al. (+9 more)
Egocentric video understanding is inherently complex due to the dynamic 4D nature of the environment, where camera motion and object displacements necessitate a continuous re-evaluation of spatial relations. In this work, we target a suite of under-explored
egocentric 4D
reasoning tasks, including fixture interaction counting, viewpoint-relative fixture location, object movement itinerary tracking, and stationary object localization, that require fundamentally different cognitive operations: spatial anchoring, temporal tracking, and duration
reasoning. We observe that these structural differences make task-agnostic approaches insufficient: generic Chain-of-Thought methods lack task-appropriate
reasoning primitives, and uniform reinforcement learning actively destabilizes performance on spatial tasks. To address this, we propose EgoReasoner, a two-stage framework that aligns both the
reasoning scaffold and the reward signal to each task's cognitive structure. In the first stage, Task-Adaptive Thinking Templates guide the synthesis of structured CoT traces that teach the model to reason adaptively across task types via supervised fine-tuning. In the second stage, task-aware reward functions verify entity grounding, temporal
alignment, and task-adaptive logical consistency, selectively strengthening each
reasoning pathway via reinforcement fine-tuning with GRPO. Our 3B-parameter model, trained on only 16K samples, achieves 37.5% average accuracy on the challenging HD-EPIC
benchmark, surpassing Qwen2.5-VL-7B (25.7%) by over 10 points.
Read the full abstract ▾
All authors Fangrui Zhu, Yunfeng Xi, Jianmo Ni, Mu Cai, Boqing Gong, Long Zhao, Chen Qu, Ian Miao, Yi Li, Cheng Zhong, Huaizu Jiang, Shwetak Patel
Computer Vision
Modeling and Measuring Redundancy in Multisource Multimodal Data for Autonomous Driving
Yuhan Zhou, Mehri Sattari, Haihua Chen et al. (+1 more)
Next-generation autonomous vehicles (AVs) rely on large volumes of multisource and
multimodal ($M^2$) data to support real-time decision-making. In practice, data quality (DQ) varies across sources and modalities due to environmental conditions and sensor limitations, yet AV research has largely prioritized
algorithm design over DQ analysis. This work focuses on redundancy as a fundamental but underexplored DQ issue in AV
datasets. Using the nuScenes and Argoverse 2 (AV2)
datasets, we model and measure redundancy in multisource camera data and
multimodal image-
LiDAR data, and evaluate how removing redundant labels affects the YOLOv8 object detection task. Experimental results show that selectively removing redundant multisource image object labels from cameras with shared fields of view improves detection. In nuScenes, mAP${50}$ gains from $0.66$ to $0.70$, $0.64$ to $0.67$, and from $0.53$ to $0.55$, on three representative overlap regions, while detection on other overlapping camera pairs remains at the baseline even under stronger
pruning. In AV2, $4.1$-$8.6\%$ of labels are removed, and mAP${50}$ stays near the $0.64$ baseline.
Multimodal analysis also reveals substantial redundancy between image and
LiDAR data. These findings demonstrate that redundancy is a measurable and actionable DQ factor with direct implications for AV performance. This work highlights the role of redundancy as a data quality factor in AV perception and motivates a data-centric perspective for evaluating and improving AV
datasets. Code, data, and implementation details are publicly available at: https://github.com/yhZHOU515/RedundancyAD
Read the full abstract ▾
All authors Yuhan Zhou, Mehri Sattari, Haihua Chen, Kewei Sha
Computer Vision
SurgFormer: Scalable Learning of Organ Deformation with Resection Support and Real-Time Inference
Ashkan Shahbazi, Elaheh Akbari, Kyvia Pereira et al. (+7 more)
We introduce SurgFormer, a multiresolution gated
transformer for data driven soft tissue simulation on volumetric meshes. High fidelity biomechanical solvers are often too costly for interactive use, so we train SurgFormer on solver generated data to predict nodewise displacement fields at near real time rates. SurgFormer builds a fixed mesh hierarchy and applies repeated multibranch blocks that combine local message passing, coarse global self
attention, and pointwise feedforward updates, fused by learned per node, per channel gates to adaptively integrate local and long range information while remaining scalable on large meshes. For cut conditioned simulation, resection information is encoded as a learned cut
embedding and provided as an additional input, enabling a unified model for both standard deformation prediction and
topology altering cases. We also introduce two
surgical simulation
datasets generated under a unified protocol with XFEM based supervision: a cholecystectomy resection
dataset and an appendectomy manipulation and resection
dataset with cut and uncut cases. To our knowledge, this is the first learned volumetric surrogate setting to study XFEM supervised cut conditioned deformation within the same volumetric pipeline as standard deformation prediction. Across diverse baselines, SurgFormer achieves strong accuracy with favorable efficiency, making it a practical backbone for both tasks. {Code, data, and project page: \href{https://mint-vu.github.io/SurgFormer/}{available here}}
Read the full abstract ▾
All authors Ashkan Shahbazi, Elaheh Akbari, Kyvia Pereira, Jon S. Heiselman, Annie C. Benson, Garrison L. H. Johnston, Jie Ying Wu, Nabil Simaan, Michael I. Miga, Soheil Kolouri
Computer Vision
NEGATE: Constrained Semantic Guidance for Linguistic Negation in Text-to-Video Diffusion
Taewon Kang and Ming C. Lin
Negation is a fundamental linguistic operator, yet it remains inadequately modeled in
diffusion-based generative systems. In this work, we present a formal treatment of linguistic negation in
diffusion-based generative models by modeling it as a structured feasibility constraint on semantic guidance within
diffusion dynamics. Rather than introducing heuristics or retraining model parameters, we reinterpret classifier-free guidance as defining a semantic update direction and enforce negation by projecting the update onto a convex constraint set derived from linguistic structure. This novel formulation provides a unified framework for handling diverse negation phenomena, including object absence, graded non-inversion semantics, multi-negation composition, and scope-sensitive disambiguation. Our approach is training-free, compatible with pretrained
diffusion backbones, and naturally extends from image generation to temporally evolving
video trajectories. In addition, we introduce a structured negation-centric
benchmark suite that isolates distinct linguistic failure modes in generative systems, to further research in this area. Experiments demonstrate that our method achieves robust negation compliance while preserving visual fidelity and structural coherence, establishing the first unified formulation of linguistic negation in
diffusion-based generative models beyond representation-level evaluation.
Read the full abstract ▾
Computer Vision
Spatial Calibration of Diffuse LiDARs
Nikhil Behari and Ramesh Raskar
Diffuse direct time-of-flight
LiDARs report per-pixel depth histograms formed by aggregating
photon returns over a wide instantaneous field of view, violating the single-ray assumption behind standard
LiDAR-RGB calibration. We present a simple spatial calibration procedure that estimates, for each diffuse
LiDAR pixel, its footprint (effective support region) and relative spatial sensitivity in a co-located RGB image plane. Using a scanned retroreflective patch with background subtraction, we recover per-pixel response maps that provide an explicit
LiDAR-to-RGB correspondence for cross-modal
alignment and fusion. We demonstrate the method on the ams OSRAM TMF8828.
Read the full abstract ▾
Computer Vision
AV-Unified: A Unified Framework for Audio-visual Scene Understanding
Guangyao Li, Xin Wang and Wenwu Zhu
When humans perceive the world, they naturally integrate multiple audio-visual tasks within dynamic, real-world scenes. However, current works such as event localization, parsing,
segmentation and question answering are mostly explored individually, making it challenging to comprehensively understand complex audio-visual scenes and explore inter-task relationships. Hence, we propose AV-Unified, a unified framework that enables joint learning across a wide range of audio-visual scene understanding tasks. AV-Unified standardizes the diverse input-output formats of each task and incorporates a multi-scale spatiotemporal perception network to effectively capture audio-visual associations. Specifically, we unify the inputs and outputs of all supported tasks by converting them into sequences of discrete
tokens, establishing a shared representation that allows a single architecture to be trained jointly across heterogeneous varied
datasets. Considering the varying temporal granularity of audio-visual events, a multi-scale temporal perception module is designed to capture key cues. Meanwhile, to overcome the lack of auditory supervision in the visual domain, we design a cross-modal guidance-based spatial perception module that models spatial audio-visual associations. Furthermore, task-specific text
prompts are employed to enhance the model's adaptability and task-awareness. Extensive experiments on
benchmark datasets (e.g., AVE, LLP, MUSIC-AVQA, VGG-SS and AVS) demonstrate the effectiveness of AV-Unified across temporal, spatial, and spatiotemporal tasks.
Read the full abstract ▾
Computer Vision
Artificial Intelligence for Detecting Fetal Orofacial Clefts and Advancing Medical Education
Yuanji Zhang, Yuhao Huang, Haoran Dou et al. (+28 more)
Orofacial clefts are among the most common congenital craniofacial abnormalities, yet accurate prenatal detection remains challenging due to the scarcity of experienced specialists and the relative rarity of the condition. Early and reliable diagnosis is essential to enable timely clinical intervention and reduce associated morbidity. Here we show that an artificial intelligence system, trained on over 45,139 ultrasound images from 9,215 fetuses across 22 hospitals, can diagnose fetal orofacial clefts with sensitivity and specificity exceeding 93% and 95% respectively, matching the performance of senior radiologists and substantially outperforming junior radiologists. When used as a medical copilot, the system raises junior radiologists' sensitivity by more than 6%. Beyond direct diagnostic assistance, the system also accelerates the development of clinical expertise. A pilot study involving 24 radiologists and trainees demonstrated that the model can improve the expertise development for rare conditions. This dual-purpose approach offers a scalable solution for improving both diagnostic accuracy and specialist training in settings where experienced radiologists are scarce.
Read the full abstract ▾
All authors Yuanji Zhang, Yuhao Huang, Haoran Dou, Xiliang Zhu, Chen Ling, Zhong Yang, Lianying Liang, Jiuping Li, Siying Liang, Rui Li, Yan Cao, Yuhan Zhang, Jiewei Lai, Yongsong Zhou, Hongyu Zheng, Xinru Gao, Cheng Yu, Liling Shi, Mengqin Yuan, Honglong Li, Xiaoqiong Huang, Chaoyu Chen, Jialin Zhang, Wenxiong Pan, Alejandro F. Frangi, Guangzhi He, Xin Yang, Yi Xiong, Linliang Yin, Xuedong Deng, Dong Ni
Computer Vision · Cross-listed from cs.RO
SG-DOR: Learning Scene Graphs with Direction-Conditioned Occlusion Reasoning for Pepper Plants
Rohit Menon, Niklas Mueller-Goldingen, Sicong Pan et al. (+2 more)
Robotic harvesting in dense crop canopies requires effective interventions that depend not only on geometry, but also on explicit, direction-conditioned relations identifying which organs obstruct a target fruit. We present SG-DOR (Scene Graphs with Direction-Conditioned Occlusion
Reasoning), a relational framework that, given instance-segmented organ point clouds, infers a scene graph encoding physical attachments and direction-conditioned occlusion. We introduce an occlusion ranking task for retrieving and ranking candidate leaves for a target fruit and approach direction, and propose a direction-aware graph neural architecture with per-fruit leaf-set
attention and union-level aggregation. Experiments on a multi-plant synthetic pepper
dataset show improved occlusion prediction (F1=0.73, NDCG@3=0.85) and attachment inference (edge F1=0.83) over strong ablations, yielding a structured relational signal for downstream intervention planning.
Read the full abstract ▾
All authors Rohit Menon, Niklas Mueller-Goldingen, Sicong Pan, Gokul Krishna Chenchani, Maren Bennewitz
Computer Vision
Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis
Hila Chefer, Patrick Esser, Dominik Lorenz et al. (+5 more)
Strong semantic representations improve the convergence and generation quality of
diffusion and flow models. Existing approaches largely rely on external models, which require separate training, operate on misaligned objectives, and exhibit unexpected scaling behavior. We argue that this dependence arises from the model's training objective, which poses a denoising task with little incentive to learn semantic representations. We introduce Self-Flow: a self-supervised flow matching paradigm that integrates representation learning within the generative framework. Our key mechanism, Dual-Timestep Scheduling, applies heterogeneous noise levels across
tokens, creating an information asymmetry that forces the model to infer missing information from corrupted inputs. This drives learning strong representations alongside generative capabilities without external supervision. Our method generalizes across modalities and enables multi-modal training while following expected scaling laws, achieving superior image,
video, and audio generation.
Read the full abstract ▾
All authors Hila Chefer, Patrick Esser, Dominik Lorenz, Dustin Podell, Vikash Raja, Vinh Tong, Antonio Torralba, Robin Rombach