Computer Vision
CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes
Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti et al. (+4 more)
Do
Vision-
Language Models (VLMs) count what they see, or what they know about the world? We introduce CounterCount, a diagnostic
dataset of paired factual and counterfactual (CF) images, in which the CF image contains an atypical count of a subject's attribute (e.g., a rabbit with four ears), together with annotations localizing the counted attributes. Humans answer both types of images nearly perfectly ($98.9\%$ and $97.5\%$), but across 16 VLMs (14 open-weight and 2 closed-source), average accuracy drops from $91.6\%$ on factual to $49.5\%$ on CF images, with most errors stemming from responses that match the canonical count, therefore revealing a bias towards knowledge priors rather than reliance on visual information. Based on these findings, we perform an analysis that reveals two interesting insights behind this gap. First, models exhibit a “System-1”-like response behavior, falling back on canonical counts after a shallow recognition of the subject. Second, model activations partially contain the visual evidence necessary to answer correctly, yet models underutilize it, attending less to the counted attributes when they provide biased answers. Input manipulations that break visual context, introduce cueing or
reasoning, or direct amplification of the model's internal
attention to relevant regions can reduce such biases. CounterCount goes beyond showing that VLMs fall back on knowledge priors, revealing that once a familiar context is recognized, models favor prior knowledge over available visual evidence rather than failing to encode that evidence.
Read the full abstract ▾
All authors Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti, Zaid Alyafeai, Paul Gavrikov, Bernard Ghanem, Abdelrahman Eldesokey
Computer Vision
Why We Look Where We Look: Emergent Human-like Fixations of a Foveated Visual Language Model Maximizing Scene Understanding
Shravan Murlidaran, Ziqi Wen, Sana Shehabi et al. (+1 more)
When humans view scenes without a specific task (free-viewing), they initially direct their eye movements toward the scene center and then fixate on people, text, objects being gazed at or grasped, and semantically meaningful regions. What these signature fixation patterns reflect and whether they optimize an underlying perceptual task remain unknown. We show that a computational
agent with simulated foveation, trained to optimize scene comprehension, exhibits emergent human fixation signature patterns. In contrast, versions of the
agent trained to search or classify scenes, or equipped with peripheral
vision that was better or worse than human
vision, predicted human fixation patterns less accurately. Thus, human free-viewing fixation patterns may emerge as a functional byproduct of optimizing scene comprehension under the biological constraints of foveated
vision.
Read the full abstract ▾
All authors Shravan Murlidaran, Ziqi Wen, Sana Shehabi, Miguel P. Eckstein
Computer Vision
Unleashing the Representational Power of Fourier Shapes for Attacking Infrared Object Detection
Yixing Yong, Jian Wang, Ming Lei et al. (+2 more)
Infrared object detection is crucial for perception in autonomous driving and surveillance but remains vulnerable to physical
adversarial attacks. Unlike in the RGB domain, where attacks rely on color texture, infrared attacks must manipulate thermal signatures, making the geometry shape of heat-blocking materials the primary
adversarial information carrier. Current shape-based methods suffer from a fundamental trade-off between representational capability and optimization power, limiting their attack effectiveness.In this work, we overcome this dilemma by introducing learnable
Fourier shapes to the infrared domain. We utilize an end-to-end differentiable framework where a compact set of
Fourier coefficients, defining the shape boundary, is analytically mapped to a pixel-space mask via the winding number
theorem. This enables efficient gradient-based optimization to generate potent shapes that cause human targets to evade detection. Extensive digital and physical experiments provide a comprehensive evaluation and validate our superior performance. Our resulting physical patch achieves striking robustness, successfully evading detectors across diverse distances, angles, poses, and individuals, and achieves over 88% attack success rate at distances greater than 25m (conf.=0.5). Code is available at https://github.com/Yongyx99/
Fourier-shape-attack.
Read the full abstract ▾
All authors Yixing Yong, Jian Wang, Ming Lei, Lijun He, Fan Li
Computer Vision
Evidence-Guided Unknown Rejection for High-Confidence Near-Known Unknowns
Xi Chen, Yingjun Xiao and Gang Fang
Open-set recognition systems face a neglected failure mode: high-confidence near-known unknowns, which lie outside the known label set but are close enough to known classes that a closed-set classifier accepts them with high confidence. We show that this failure is widespread across scalar-threshold methods, including recent post-hoc detectors, and that stronger encoders can amplify rather than remove the risk. We propose EGUR-A, which changes the decision from “is this sample's score high enough?” to “does this predicted known class have sufficient evidence to accept this sample?” EGUR-A combines class-conditional local acceptance evidence with global residual evidence, and selects their relative weight from known-sample statistics without unknown validation data. Across CUB, FGVC-Aircraft, and ImageNet-hard, EGUR-A substantially reduces high-confidence false known acceptance at matched known-rejection operating points. The result is not a stronger threshold; it is a different question: whether a known class is entitled to accept a sample.
Read the full abstract ▾
Computer Vision
Curriculum Group Policy Optimization: Adaptive Sampling for Unleashing the Potential of Text-to-Image Generation
Baoteng Li, Xianghao Zang, Xinran Wang et al. (+8 more)
Text-to-Image (T2I) generation has achieved remarkable progress in recent years. Meanwhile, reinforcement learning methods, particularly those based on Group Relative Policy Optimization (GRPO), have attracted widespread
attention and been successfully applied to T2I tasks. However, the uniform sampling strategy commonly used during training often ignores the match between sample difficulty and the model's current learning capability, leading to low training efficiency. We argue that improving training efficiency requires continuously prioritizing
prompts that match the model's evolving capability and remain actively learnable. To this end, we propose Curriculum Group Policy Optimization (CGPO), an adaptive curriculum training framework. During training, each
prompt produces a group of images scored by a reward model. We use the variance of group rewards as an online proxy for
prompt inconsistency. A higher variance suggests that the model has partially captured the
prompt requirements but has not yet achieved stable mastery. Such
prompts are more likely to provide useful learning signals, so we increase their sampling probabilities accordingly. Additionally, to address data imbalance in multi-category
datasets, we design a category calibration method based on proportional
fairness optimization, which balances training difficulty across categories. Experiments on GenEval, T2I-CompBench++, and DPG Bench demonstrate that our framework effectively improves generation performance.
Read the full abstract ▾
All authors Baoteng Li, Xianghao Zang, Xinran Wang, Xiangyu Na, Zhixiang He, Hao Sun, Chi Zhang, Zhongjiang He, Tianwei Cao, Kongming Liang, Zhanyu Ma
Computer Vision
Is Complex Training Necessary for Long-Tailed OOD Detection? A Re-think from Feature Geometry
Ningkang Peng, Xuanming Chen and Yanhui Gu
Long-tailed out-of-distribution (LT-OOD) detection is often addressed with specialized training, including auxiliary out-of-distribution (OOD) data, abstention heads, contrastive objectives, energy losses, or gradient-conflict control. We show that these training mechanisms can obscure a simpler issue: frozen long-tailed representations may already contain useful OOD evidence, but raw Mahalanobis distance is distorted by frequency-coupled feature radius and poorly supported tail covariance. We propose Hyperspherical Pooled Mahalanobis (HPM), a post-hoc detector that normalizes features onto the unit sphere and replaces class-specific covariance with a pooled, ridge-regularized metric while keeping class means as semantic anchors. In CIFAR-LT experiments and an ImageNet-100-LT near-OOD boundary analysis, HPM improves raw Mahalanobis scoring; for Prior-Calibrated ERM (PC-ERM), it raises AUROC from 46.49 to 85.67 on CIFAR-10-LT and from 50.40 to 78.35 on CIFAR-100-LT. This simple PC-ERM+HPM pipeline also achieves the best Log Efficiency Score (LES; 3.08) on CIFAR-100-LT, retaining roughly 95% of the best CIFAR-100-LT AUROC observed among the compared post-hoc scores at substantially lower training-time cost. These results argue for evaluating representation quality, detector geometry, and training complexity as separate factors in LT-OOD detection.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
When Accuracy Is Not Enough: Uncertainty Collapse between Noisy Label Learning and Out-of-Distribution Detection
Ningkang Peng, Jingyang Mao, Runhan Zhou et al. (+2 more)
Learning with noisy labels (LNL) is typically benchmarked by closed-set classification accuracy, yet deployment often requires classifiers to reject out-of-distribution (OOD) inputs. We present a learner-agnostic ACC-OOD
benchmark that freezes LNL checkpoints and evaluates them with standardized near-/far-OOD routing and post-hoc scores across synthetic and real label noise. The
benchmark reveals a recurring failure mode: high closed-set accuracy does not ensure OOD reliability, because low-confidence, misclassified in-distribution samples can overlap the score and feature regions occupied by OOD inputs under noisy training. We term this pathology uncertainty collapse. This structural overlap can make high-accuracy LNL methods lose separability at the ID-error/OOD interface under standard OOD scores. As an intervention, we study Virtual Margin Regularization (VMR), a lightweight repair probe demonstrated mainly with PSSCL that synthesizes boundary virtual outliers on trusted ID batches and widens the energy margin. VMR partially reduces the collapse-induced far-OOD failure without replacing the host objective or sacrificing closed-set accuracy in the tested settings. These results support LNL
benchmarks that co-report closed-set generalization, open-world reliability, and structural overlap diagnostics.
Read the full abstract ▾
All authors Ningkang Peng, Jingyang Mao, Runhan Zhou, Peirong Ma, Yanhui Gu
Computer Vision
Network Knowledge Prior Guided Learning for Data-Efficient Surface Defect Detection
Hang-Cheng Dong, Guodong Liu, Dong Ye et al. (+1 more)
Deep learning-based methods have become the de facto standard for industrial defect detection. However, their data-hungry nature and inherent "black-box" characteristics often lead to performance bottlenecks and limited trustworthiness in real-world applications. To address these challenges, this paper proposes a novel knowledge-guided loss function that seamlessly integrates model interpretability into the training process without incurring any additional inference cost. Our method operates in two phases: first, a primary classification network is trained, and its explanations, in the form of saliency maps, are generated as prior knowledge. Second, a multi-task learning framework is established, where the main task performs classification, and an auxiliary task imposes consistency between the saliency maps of the final model and the primary model. This consistency is enforced by a dedicated knowledge-guided loss term, effectively acting as a powerful regularizer to steer the model towards robust feature representations. Extensive experiments on multiple public defect
datasets demonstrate that our approach consistently enhances the performance of baseline models in terms of accuracy and AP. Moreover, visual analysis reveals that the proposed method yields more concentrated and human-intelligible saliency maps. This work presents a simple yet effective paradigm for bridging the gap between model performance and interpretability, paving the way for more reliable and high-performing
vision systems in industrial quality inspection.
Read the full abstract ▾
All authors Hang-Cheng Dong, Guodong Liu, Dong Ye, Bingguo Liu
Computer Vision
Efficient Sparse-to-Dense Visual Localization via Compact Gaussian Scene Representation and Accelerated Dense Pose Estimation
Zizhuo Li, Songchu Deng, Linfeng Tang et al. (+1 more)
This letter presents LiteLoc, a novel and efficient localizer built on 3D Gaussian Splatting (3DGS). The previous state-of-the-art (SoTA) sparse-to-dense localizer, STDLoc, has shown remarkable localization capability but suffers from severe storage redundancy and computational latency. By revisiting its design decisions, we derive two simple yet highly effective improvements that cumulatively make LiteLoc much more efficient in both
memory and computation, while also being easier to train. One key observation is that the color field, inherited directly from Feature 3DGS, is functionally useless for localization. Yet, its reconstruction of high-frequency photometric details necessitates excessive Gaussian primitives, resulting in a tightly coupled color-feature representation with significant
memory overhead and sub-optimal feature field optimization. To resolve this, we propose a color-free decoupled feature field that constructs a compact Gaussian scene representation by retaining only task-essential feature attributes, thereby eliminating approximately 94% of redundant storage with no loss of localization-relevant information. We further find that the primary computational bottleneck lies in the dense Perspective-n-Point (PnP) solver, where most matches contribute saturated
geometric constraints with diminishing accuracy gains. Accordingly, we propose a condensing strategy that distills dense matches into a subset of 5% representative matches, enabling a nearly 19-fold speedup in robust estimation with negligible performance drop. Extensive experiments show that LiteLoc surpasses STDLoc in multiple scenes with considerable efficiency benefits, opening up exciting prospects for latency-sensitive visual localization.
Read the full abstract ▾
All authors Zizhuo Li, Songchu Deng, Linfeng Tang, Jiayi Ma
Computer Vision
PlantPose: Universal Plant Skeleton Estimation via Tree-constrained Graph Generation
Xinpeng Liu, Hiroaki Santo, Yosuke Toda et al. (+1 more)
Accurate estimation of plant skeletal structures (e.g., branching structures) from images is essential for smart agriculture and plant science. Unlike human skeletons with fixed
topology, plant skeleton estimation presents a unique challenge, i.e., estimating arbitrary tree graphs from images. To address this problem, we introduce PlantPose, a universal plant skeleton
estimator via tree-constrained graph generation. PlantPose combines learning-based graph generation with traditional graph
algorithms to enforce tree constraints during the training loop. To enhance the model's generalization capability, we curate a large and diverse
dataset comprising real-world and synthetic plant images, along with simplified representations (e.g., sketches and abstract drawings). This
dataset enables the generalized model to adapt to diverse input styles and categories of plant images while preserving topological consistency. Our approach demonstrates robust and accurate plant skeleton estimation across multiple domains, including previously unseen out-of-domain scenarios. Further analyses highlight the method's strengths and limitations in handling complex, heterogeneous data distributions. All implementations and
datasets are available at https://github.com/huntorochi/PlantPose/.
Read the full abstract ▾
All authors Xinpeng Liu, Hiroaki Santo, Yosuke Toda, Fumio Okura
Computer Vision
Towards Universal Physical Adversarial Attacks via a Joint Multi-Objective and Multi-Model Optimization Framework
Ziyang Liu, Hongyuan Wang, Zijian Wang et al. (+4 more)
Physical
adversarial attacks often overfit single surrogate models and optimization objectives. While ensemble attacks can mitigate this, existing methods struggle with severe gradient conflicts within restricted physical texture spaces, significantly degrading cross-model transferability. To bridge this gap, this paper proposes a Joint Multi-Objective and Multi-Model Optimization Framework (JMOF) that leverages quantitative similarity analysis to select the optimal surrogate model ensemble. Within JMOF, a dual-level mechanism jointly suppresses prediction outputs and flattens intermediate feature distributions, balancing attack efficiency with deep generalization. Additionally, an Orthogonal Gradient
Alignment (OGA) strategy resolves cross-model gradient conflicts, transforming mutually repulsive gradients into synergistic optimization directions. Extensive simulated and real-world experiments demonstrate that JMOF outperforms state-of-the-art baselines against diverse black-box detectors. Crucially, JMOF exhibits substantial cross-
vision-task generalization, generating attacks capable of simultaneously deceiving object detection and semantic
segmentation or
monocular depth estimation models. This research advances the generalization limits of physical
adversarial attacks, providing a robust framework for evaluating visual AI vulnerabilities in real-world deployments.
Read the full abstract ▾
All authors Ziyang Liu, Hongyuan Wang, Zijian Wang, Yinxi Lu, Yunzhao Zang, Zhiqiang Yan, Qianhao Ning
Computer Vision
LatentUMM: Dual Latent Alignment for Unified Multimodal Models
Yinyi Luo, Wenwen Wang, Hayes Bai et al. (+2 more)
Unified
multimodal models (UMMs) achieve strong performance in both understanding and generation by learning a shared latent space, yet they often exhibit functional inconsistency between these two capabilities. We observe that this issue does not stem from a lack of shared representations, but from the absence of explicit
alignment between the transformations that map into and out of the latent space. As a result, generation and re-encoding can follow inconsistent trajectories, leading to semantic drift under modality transitions. In this work, we propose LatentUMM, a framework that constructs an enhanced shared latent space to explicitly align these transformations and improve cross-modal consistency. LatentUMM consists of two stages. First, dual latent
alignment enforces consistency at both the modality and capacity levels: cross-modal
alignment uses a stronger
embedding model to impose structured cross-modal semantics, while dual capacity
alignment enforces bidirectional consistency under generation and re-encoding. Second, latent dynamics stabilization improves robustness via
stochastic latent rollouts and preference optimization, favoring trajectories that better preserve semantic consistency. Experiments show that LatentUMM consistently improves
multimodal consistency across diverse architectures. Code is available at: https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/LatentUMM.
Read the full abstract ▾
All authors Yinyi Luo, Wenwen Wang, Hayes Bai, Marios Savvides, Jindong Wang
Computer Vision
FrequencyBooster: Full-Frequency Modeling for High-Fidelity Pixel Diffusion
Lichen Ma, Zipeng Guo, Yu He et al. (+5 more)
To circumvent the inherent fidelity bottlenecks and optimization misalignment of VAE-based latent
diffusion, pixel-space
diffusion models have emerged as a compelling end-to-end paradigm. However, existing pixel
diffusion models often struggle to balance computational efficiency with the preservation of high-frequency details. They frequently resort to patch-based compression or restricted local decoding, leading to a "spectral compromise" where high-frequency and fine-grained pixel information are suppressed. To address these challenges, we propose FrequencyBooster, a novel framework designed to empower pixel
diffusion with full-frequency modeling capabilities without prohibitive overhead. The core of our method is a high-capacity decoder that specializes in extracting exhaustive high-frequency details and low-frequency semantics, the latter of which is derived from a
Diffusion Transformer (DiT) backbone. Unlike prior works that sacrifice global context for local refinement, FrequencyBooster leverages high-dimensional feature representations to maintain global structural integrity while achieving superior pixel-level precision. Extensive experiments on ImageNet demonstrate the effectiveness of our approach: our model achieves a state-of-the-art FID of 1.60 at $256 \times 256$ resolution within only 320 epochs. Furthermore, at $512 \times 512$ resolution, FrequencyBooster attains an FID of 1.69, significantly outperforming existing pixel-space and latent-space generative models.
Read the full abstract ▾
All authors Lichen Ma, Zipeng Guo, Yu He, Xiaolong Fu, Luohang Liu, Jingling Fu, Junshi Huang, Yan Li
Computer Vision
Unleashing Vision Transformer Potential In Image Quality Assessment via Global-Local Adaptive Interaction
Yu Li, Puchao Zhou, Yachun Mi et al. (+3 more)
In the field of Blind Image Quality Assessment (BIQA), accurately predicting the perceptual quality of authentically distorted images remains highly challenging due to the diverse and complex distortions present in natural environments. Although existing methods have achieved notable accuracy, their scalability is often constrained by the high cost of subjective annotation and the limited size of available
datasets. Recent advances in large-scale pre-trained
vision models have introduced powerful semantic and representational capabilities, yet their application to IQA tasks is hindered by substantial computational demands and suboptimal fine-tuning efficiency. To overcome these limitations, we introduce the Global-Local Interaction Adapter (GLIA), a novel framework that effectively harnesses pre-trained
Vision Transformers through a dual-stream feature extraction mechanism coupled with interactive global-local fusion. By jointly retaining global semantic information and fine-grained local details, our approach delivers superior prediction accuracy and robustness while requiring significantly fewer trainable parameters. Extensive experiments on multiple
benchmarks validate the effectiveness and superiority of our approach.
Read the full abstract ▾
All authors Yu Li, Puchao Zhou, Yachun Mi, Yanfeng Wu, Xiaoming Wang, Shaohui Liu
Computer Vision
MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation
Ronyu Zhang, Aosong Cheng, Gaole Dai et al. (+8 more)
Continual test-time adaptation adapts a source-pretrained model to non-stationary, unlabeled target streams while retaining past competence, yet texture-biased backbones risk error accumulation and catastrophic forgetting. Drawing inspiration from the process of decoupling shape and texture in the human visual system, we introduce MoASE, a plug-in mixture-of-experts that disentangles domain-agnostic structure from domain-specific texture using Activation Sparsity Experts with Spatial Differentiable Dropout, forming complementary high- and low-activation pathways, while high- and low-rank bottlenecks diversify representations. The Activation Sparsity Gate produces input-adaptive SDD thresholds for precise
token selection, and the Domain-Aware Router assigns per-sample expert weights using texture-sensitive cues. To curb confirmation bias on unlabeled streams and stabilize supervision, we then introduce Domain-Adaptive On-Policy
Distillation to constitute MoASE++, with an EMA-anchored on-policy reverse KL
distillation and an augmentation policy conditioned on
entropy and confidence that aligns predictions across the same views and improves the robustness-plasticity balance. Extensive experiments on classification (CIFAR-10/100-C, ImageNet-C) and semantic
segmentation (Cityscapes->ACDC) demonstrate consistent state-of-the-art performance, offering a principled, controllable approach to continual adaptation in dynamic visual environments.
Read the full abstract ▾
All authors Ronyu Zhang, Aosong Cheng, Gaole Dai, Yulin Luo, Jiaming Liu, Li Du, Huanrui Yang, Dan Wang, Leyuan Fang, Yuan Du, Shanghang Zhang
Computer Vision
UST-Hand: An Uncertainty-aware Spatiotemporal Point Cloud Interaction Network for 3D Self-supervised Hand Pose Estimation
Tianhao Han, Haoyang Zhang, Liang Xie et al. (+5 more)
Manually annotating accurate 3D hand poses is extremely time-consuming and labor-intensive. Existing self-supervised hand pose estimation methods leverage the discrepancy between input images and rendered outputs, or multi-view consistency constraints, as the driving force to optimize networks and progressively refine pose accuracy. However, these methods are highly susceptible to noisy pseudo-labels and overlook the importance of fully exploiting fine-grained spatial correlations, which undermines the stability of model training. To address these issues, we propose UST-Hand, a self-supervised learning framework that estimates uncertainty distribution of hand pose and constructs a probabilistic point cloud feature space, which enables the complex spatiotemporal relationship modeling. UST-Hand employs a conditional normalizing flow model to capture hand pose distributions and samples diverse hypotheses, facilitating robust learning under noisy pseudo-labels supervision with enhanced stability. These multi-hypothesis are mapped to a unified probabilistic 3D point cloud space for multi-view and temporal feature interaction, comprehensively exploring hand motion patterns and fine-grained spatial correlations. Extensive experiments on three challenging
datasets demonstrate that UST-Hand achieves state-of-the-art performance, outperforming existing self-supervised methods by up to 37.8% in Mean Per Vertex Position Error (MPVPE).
Read the full abstract ▾
All authors Tianhao Han, Haoyang Zhang, Liang Xie, Haochen Chang, Kun Gao, Yuan Cheng, Pengfei Ren, Erwei Yin
Computer Vision
Domain Incremental Learning for Pandemic-Resilient Chest X-Ray Analysis
Danu Kim
Deep learning models achieved high accuracy in pneumonia detection from chest X-rays. However, their generalization across clinical domains remains limited due to variations in imaging devices, acquisition protocols, and institutional conditions. This study introduces a replay-based domain-incremental continual learning designed to enable continual adaptation to cross-domain variations without catastrophic forgetting. The proposed method incorporates a class-aware balanced replay to maintain balanced class representation within a constrained
memory and a class-aware loss to dynamically reweight class imbalance during training. Experiments conducted on a domain-shifted PneumoniaMNIST
dataset consisting of five simulated domains demonstrate that the proposed method achieves an average accuracy of 88.66%, outperforming Experience Replay, Fine-Tuning, and Joint Training baselines. These findings highlight the efficacy of the proposed approach in achieving robust and consistent pneumonia detection across clinical environment variations.
Read the full abstract ▾
Computer Vision
GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations
Zesheng Li, Chengchang Pan and Honggang Qi
Frozen
vision-
language embeddings contain signals at multiple semantic resolutions, from object identity to attributes, relations, and full-caption meaning, but they expose these signals through a fixed-length vector interface. We study whether
embedding length can be turned into a controllable semantic access interface. We propose GraSP-VL, which learns a shared near-orthogonal prefix transform over frozen VLM
embeddings. GraSP-VL instantiates a Semantic Matryoshka interface: short prefixes are assigned coarse semantic roles, while longer prefixes progressively expose finer
language-grounded distinctions. Because the transform is shared across image and text
embeddings and preserves full-dimensional geometry, prefix behavior changes without rewriting the original VLM space. On a 20,147-example COCO/Flickr30K annotation pool, GraSP-VL reaches a staircase score of 53.01 and hard-negative selectivity of 89.76, while keeping full-space drift below $10^{-6}$. It also transfers to SugarCrepe-clean with 86.03 object accuracy and 11.96 mean external emergence, and preserves full-dimensional zero-shot CIFAR-100 accuracy. These results show that frozen VLM
embeddings can be reorganized into a truncatable semantic prefix interface rather than merely compressed.
Read the full abstract ▾
Computer Vision
Patch-MoE Mamba: A Patch-Ordered Mixture-of-Experts State Space Architecture for Medical Image Segmentation
Diego Adame, Fabian Vazquez, Jose A. Nunez et al. (+7 more)
CNN- and
Transformer-based architectures have achieved strong performance in medical image
segmentation, but CNNs are limited in modeling long-range dependencies, while
Transformers often suffer from quadratic computational and
memory complexity. State space models, especially Mamba-based networks, offer an efficient alternative with linear sequence complexity. However, existing Mamba
segmentation models still face two limitations: pixel-wise directional scanning can disrupt local 2D spatial structure, and simple summation-based fusion of scan directions cannot adapt well to diverse object sizes, shapes, and boundaries. To address these issues, we propose Patch-MoE Mamba, a patch-ordered mixture-of-experts state space architecture for medical image
segmentation. It introduces a hierarchical patch-ordered scanning mechanism that preserves local spatial neighborhoods while capturing multi-scale context, and an MoE-based directional fusion module that adaptively combines multiple Mamba scanner outputs using four directional experts, a learnable concatenation expert, and residual directional aggregation. Experiments on five public polyp
segmentation benchmarks and the ISIC 2017/2018 skin lesion
segmentation datasets demonstrate the effectiveness and generality of Patch-MoE Mamba.
Read the full abstract ▾
All authors Diego Adame, Fabian Vazquez, Jose A. Nunez, Huimin Li, Jinghao Yang, Erik Enriquez, DongChul Kim, Haoteng Tang, Bin Fu, Pengfei Gu