Computer Vision
Surface Reconstruction from Range Scans Under Visibility Constraints
Gabriel Lenain, Edmond Boyer and Jean Ponce
Surface reconstruction from range scans is challenging in the presence of noise and outliers, as fidelity to observations must be balanced against sensitivity to measurement errors. Yet, most neural approaches discard the sensor positions and corresponding visibility information, compensating for this loss with
geometric priors or regularization terms. We instead formulate surface reconstruction as robust regression of the predicted first surface intersection along each acquisition ray against its measured depth, which penalizes predicted surface intersections within the segment between the sensor and the observed point. We represent the surface as the zero set of a 1-Lipschitz multi-layer perceptron, allowing the use of linearly convergent sphere tracing to compute the first hit, and use a robust Huber loss for regression. Our formulation requires neither oriented normals, sampled off-surface points, nor an eikonal loss. Comparisons with the state of the art on synthetic and real data with varying levels of noise and outlier contamination demonstrate competitive performance on clean data and much improved fidelity and robustness under even moderate corruption. Our analysis further confirms a trade-off between fidelity and robustness: greater fitting capacity improves data fidelity in the clean case but increases sensitivity to the measurement errors inherent in real acquisitions, with a sweet spot typically emerging at intermediate capacity.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Saliency-Guided Replay for Continual Learning in Deep Spiking Neural Networks
Francesco Panetta, Federico Di Valerio and Michela Proietti
Spiking Neural Networks (SNNs) are promising for energy-efficient neuromorphic computing, but they still suffer from catastrophic forgetting when learning tasks sequentially. This work investigates the integration of Experience Packing and Replay (EPR) as a
memory-efficient continual learning strategy for deep SNNs under low-latency constraints. The proposed framework combines Batch Normalization Through Time stabilized VGG9 SNNs with Spike Activation Maps (SAM), which provide forward-only explanations based on spike activity. These explanations are temporally aggregated into spatial
attention maps and used to select the most informative image patches for replay, instead of storing full images. Experiments on Split CIFAR10 and Split CIFAR100 show that SAM-guided EPR improves the accuracy-
memory trade-off under tight
memory budgets, and remains competitive while requiring substantially less storage.
Read the full abstract ▾
Computer Vision
Uncertainty-Aware Semi-Supervised Framework with Supervised Contrastive Learning for Sugarcane Leaf Disease Detection
Shalini Dangi, Divya Teja, Omkar Kabde et al. (+2 more)
Sugarcane leaf diseases severely impact crop yield and quality, making accurate and timely detection critical for sustainable agriculture. Conventional deep learning models for plant disease detection depend on large annotated
datasets, which are expensive to obtain and challenging to prepare due to challenges commonly observed across crop disease detection tasks, including (i) high intra-class variation caused by diverse symptom expressions across growth stages, (ii) subtle inter-class similarities between diseased and healthy tissues, and (iii) the risk of noisy labels degrading model reliability. To handle these challenges, we presented an Uncertainty-aware Mean-Teacher based Semi-Supervised Learning Disease Detection Network (U-SSDNet) that leverages unlabeled data while maintaining robustness. First, discrete wavelet transform preprocessing is applied to enhance lesion visibility and reduce intra- class variation. Subsequently, the KL divergence is employed to refine the pseudo-labels and reduce the impact of confirmation bias. At the same time, supervised contrastive learning is applied to make the features more distin- guishable and improve the separation among different classes. Experimental evaluations on sugarcane leaf disease
datasets, considered a representative case study, demonstrate that the proposed approach achieves performance close to that of fully supervised models while requiring far fewer annotated samples and consistently outperforms state-of-the-art supervised and semi-supervised approaches.
Read the full abstract ▾
All authors Shalini Dangi, Divya Teja, Omkar Kabde, Shahid Shafi Dar, Nagendra Kumar
Computer Vision
PolUR: Polarization Aware Multimodal Degradation Perception for Underwater Image Restoration
Wenshuo Zhang, Xiangyue Zhang, Junlin Li et al. (+5 more)
Recovering high-quality images from degraded underwater observations is crucial for underwater exploration. Polarization observations show promise as an effective source of complementary physical cues for recovering target radiance under clear-water conditions. However, existing methods have yet to jointly model raw intensity information and polarization cues to enable image restoration driven by an understanding of mixed degradations (color casts, haze, and low light). To overcome these limitations, we propose PolUR, a unified
multimodal language-driven framework that jointly uses polarization and RGB information to identify degradation types and severity levels through structured semantic instructions and enhances joint restoration representations under the guidance of degradation-aware semantic features to improve restoration robustness under mixed degradations. We further construct a new
benchmark named MixPolar, which contains underwater images exhibiting real-world degradations and their corresponding degradation descriptions. MixPolar supports systematic evaluation of degradation perception, image restoration, and performance on downstream tasks. Extensive experiments demonstrate that PolUR effectively integrates polarization and RGB information, achieving competitive performance in degradation perception and image restoration, while also demonstrating generalization capability and utility for downstream tasks.
Read the full abstract ▾
All authors Wenshuo Zhang, Xiangyue Zhang, Junlin Li, Wenjie Tian, Yubo Zheng, Hongli Xu, Jingyu Ru, Xinde Li
Computer Vision
Web Scale on a Budget: Tail-Aware Retrieval Evaluation
Boaz Meivar and Shai Avidan
A common practice in instance
retrieval is to grow the gallery size to improve a
benchmark's discrimination capability. One prominent example is ILIAS, which adds up to 100 million distractor images. The prevailing assumption is that faithful evaluation demands web scale, and that indexing all 100M images through the backbone is the unavoidable price of a comparable number. That price is steep: hundreds of GPU-hours and ~200 GB of descriptors per model, paid again for every method or ablation. We show the web-scale indexing price is largely avoidable: evaluation itself obeys a scaling law. The number of distractors that outrank a positive is Binomial in the gallery size. We define the per-query
retrieval hardness, a gallery-scale agnostic quantity that yields a
retrieval metric at any scale, and demonstrate it can be derived from a small random sample. The underlying sampled-rank machinery is long established in recommender-system and biometric evaluation, but to our knowledge has never been adapted to visual
retrieval. We make the law per-query and estimable, and validate it at true 100M scale on ILIAS and INSTRE, and on the full 759k-distractor Google Landmarks v2 pool. Concretely, within a 5% sample (a flat 20x less indexing), every backbone's tail-aware (EVT) estimate has converged onto the large-scale mAP, to within ~0.01 (mean absolute error 0.005), across five backbones, three
benchmarks, and two distractor pools (mAP@1000 at 100M on ILIAS/INSTRE; the official mAP@100 at the full 759k pool on GLDv2); the recommender-systems count baseline over-predicts everywhere (+0.037 to +0.068). This is an order of magnitude better than naively reporting the metric on the same subset (+0.08 to +0.13). We release code and the per-query hardness to facilitate faster, cheaper, and greener
retrieval evaluation.
Read the full abstract ▾
Computer Vision
dMRI-F: A Foundation Model for Generalizable Diffusion MRI Analysis from Sparse Acquisitions
Xi Zhu, Yijie Li, Xiaofan Wang et al. (+4 more)
Diffusion MRI (dMRI) probes water
diffusion within the
brain, providing unique information about tissue microstructure and white matter connectivity. High-quality dMRI enables advanced and accurate analyses but demands dense acquisitions, resulting in long scan times and hardware constraints that are often infeasible. Further, dMRI requires dedicated computational methods to support diverse analysis tasks (e.g., parcellation, registration, and synthesis) for both neuroscientific and clinical applications. Here, we present dMRI-F, a foundation model that unifies dMRI data enhancement and downstream tasks across diverse acquisitions and populations, while requiring only minimal acquisition input. dMRI-F is pre-trained with a q-space masked reconstruction task: predicting densely sampled multi-shell data from only six volumes. It learns generalizable latent feature representations of dMRI data, which serve as a core foundation for adapting to diverse downstream tasks, including
brain parcellation,
brain registration, and cross-modal synthesis. Across nine independently acquired
datasets, we demonstrate that dMRI-F outperforms state-of-the-art methods with robust generalization across acquisitions, populations, and tasks. Our results demonstrate dMRI-F's versatility as a general-purpose framework that enables diverse dMRI applications from low-cost, sparse acquisitions.
Read the full abstract ▾
All authors Xi Zhu, Yijie Li, Xiaofan Wang, Wei Zhang, Huafu Chen, Ce Zhu, Fan Zhang
Computer Vision
FETALFusion: Resolution-Aware State-Space Modeling for Multi-Domain Fetal Head Segmentation
Ammar Bhilwarawala and Mainak Bandyopadhyay
Head circumference (HC) measured from 2D ultrasound is a routine marker of fetal growth, and automating it needs segmentations whose boundaries hold up on individual scans, not only on average. We present FETALFusion, a dual-path CNN-Mamba encoder-decoder for fetal head segmentation across acquisition types. It applies four-directional state-space scans only at the two highest-resolution stages, routes bottleneck features through domain-specific channel gates that are averaged at test time, and adds a weak landmark shape prior. We train jointly on transabdominal (HC-18) and transperineal (PSFHS) images and compare ten methods on a held-out test set of 471 images, each trained under one schedule and scored by surface distances at native resolution. FETALFusion reaches a Dice score of 0.963 and an HD95 of 2.10 mm without test-time augmentation or a domain label. It improves on U-Mamba on all five reported metrics (HD95 reduced by 0.31 mm; Holm-corrected ($p < 10^{-4}$), a lead that holds under test-time augmentation and across two independent training runs, and in the mean on U-Net,
Attention U-Net and TransUNet. Residual encoders close the gap: a ResNet-34 UNet++ exceeds FETALFusion, and an ImageNet initialised ResNet-34 U-Net is best overall (HD95 1.75 mm). The ablation agrees. Calibrated against run-to-run variation, it attributes FETALFusion's measurable gains to the landmark shape prior and, on transperineal images, to the domain gates, while removing the state-space branch does not measurably change accuracy. HC from the predicted masks agrees with the clinical reference at an intraclass correlation of 0.9986.
Read the full abstract ▾
Computer Vision
AI-Enhanced Hyperspectral Imaging for Intraoperative Detection of Low-Grade Glioma: A Clinical Study
Siri Willems, Ishika Jain, Jens De Winne et al. (+9 more)
Current image-guided neuronavigation techniques, such as intraoperative MRI and ultrasound, are limited by high costs, limited sensitivity and operational complexity. There is a clear need for more efficient systems to enhance intra-operative tumor detection and improve patient outcomes for intrinsic
brain tumor resection. Whereas previous intraoperative
brain HSI studies have predominantly investigated high-grade gliomas, glioblastomas, or mixed-grade cohorts, this clinical study focuses specifically on low-grade gliomas (LGGs). LGGs strongly resemble normal
brain under white light conditions on the operative microscope. This study characterizes spectral differences between healthy and tumor tissue and provides a paired comparison of HSI- and RGB-based classification at matched tissue locations and under identical patient-level data splits. Tumor regions showed consistently lower reflectance, which may be associated with differences in hemoglobin concentration and oxygenation. A range of classification models was evaluated. The 3D CNN, which integrates spectral and spatial features, achieved the highest median ROC-AUC of 74.5%, compared with 67.9% for the spectral-only XGBoost model and 64.0% for the RGB-based model. These results support further investigation of HSI for low-grade glioma classification and quantify its performance relative to conventional RGB imaging in this cohort.
Read the full abstract ▾
All authors Siri Willems, Ishika Jain, Jens De Winne, Siri Luthman, Hamed Zivari Adab, Roeland Vandebriel, Anna Alperovich, Xiaohan Zhang, Ben Stoffelen, Tom De Schepper, Lien Solie, Steven De Vleeschouwer
Computer Vision · Cross-listed from cs.LG
Sea-Net: A Small and Inherently Interpretable Neural Network for Time Series Classification
Ubaid Ur Rehman and Tanmoy Mondal
Traditional time-series classification methods often operate as black boxes, providing a class prediction without indicating which time point or time interval within the series led to that prediction. To improve temporal interpretability, MILLET (Multiple Instance Learning with Temporal Explanation), which combines an InceptionTime encoder with Conjunctive Pooling, provides both classification and temporal explanations. However, MILLET contains approximately 424 K parameters, making it unsuitable for deployment on resource-constrained micro-controllers, and it achieves an overall time-series accuracy of 0.888 and an explanation score of 0.681 NDCG@n. We therefore propose SEA-Net, a lightweight architecture using small multi-scale depthwise-separable convolutional TSC encoders and improved Multiple Instance Learning (MIL) pooling heads. Our SEA-Net bottleneck encoder with Top-k pooling achieves 0.931 accuracy and 0.765 NDCG@n with only 41 K parameters. Another variant, using a gated encoder with Top-k pooling, achieves 0.955 accuracy and 0.750 NDCG@n with 62 K parameters. These results demonstrate that lightweight architectures can maintain strong classification performance and temporal explanations while substantially reducing model size. The proposed variants address specific limitations of the pooling strategy used by MILLET, providing a practical direction for interpretable time-series classification on resource-constrained edge devices.
Read the full abstract ▾
Computer Vision
Haoyang Jiang, Zhuotong Fan, Mingyang Yi et al. (+3 more)
Table
reasoning with
language models typically assumes access to clean textual tables. In practice, tables are often serialized into text, which can corrupt
cell contents, distort structural relationships, and discard visual cues, leaving the model to reason over incomplete evidence. In contrast, table images retain this information, but using it requires both accurate perception and
reasoning grounded in the relevant
cells. To address this, we propose MERIT (
Multimodal Evidence
Reasoning over Image Tables), a three-stage framework for learning to reason over table images, with serialized tables serving as optional auxiliary inputs (if provided). Our analysis identifies visual evidence acquisition as a bottleneck, motivating table-specific encoder adaptation through reconstruction, localization, and color-conditioned supervision. Building on the adapted representations, supervised fine-tuning teaches explicit evidence selection and
reasoning using verified trajectories. Finally, reinforcement learning refines the policy with separate rewards for evidence correctness, calculation, final answers, and output format. By providing feedback on intermediate steps, the framework targets
retrieval and computational errors that final-answer supervision alone does not distinguish. We evaluate the approach on table question answering and fact verification, analyze the contributions of visual inputs and training components, and assess generalization to additional
datasets with different table structures and visual presentations.
Read the full abstract ▾
All authors Haoyang Jiang, Zhuotong Fan, Mingyang Yi, Ju Fan, Lanqing Hu, Xiao Zan
Computer Vision
PUGS: Physics-Guided Uncertainty Modeling for Underwater Dynamic 3D Gaussian Splatting
Tingjun Li, Yuxi Wang, Guoqiang Zhong et al. (+1 more)
Dynamic 3D Gaussian Splatting (3DGS) provides an efficient framework for dynamic scene reconstruction. Most existing approaches model scene dynamics with deterministic deformation fields, assuming each Gaussian follows a distinct
geometric trajectory over time. This assumption breaks down underwater, where light scattering, refraction, marine snow and low visibility create substantial physical ambiguity in captured images. We present PUGS, a physics-guided uncertainty model for underwater dynamic 3DGS. PUGS learns inherent uncertainty induced by underwater image formation and leverages it as a physical signal for dynamic scene modeling. Rather than estimating a single deterministic displacement, it parameterizes each Gaussian's motion as a probability distribution with expected deformation and physical uncertainty. We adopt
stochastic deformation optimization to sample Gaussian trajectories via local reparameterization, exploring multiple physically plausible solutions during training. We further propose uncertainty
stochastic annealing and a dual-path self-consistency
distillation objective, converting
stochastic exploration to deterministic reconstruction while preserving temporal coherence. Extensive experiments on underwater dynamic
datasets and physics-aware
benchmarks demonstrate superior rendering quality, temporal consistency and reconstruction robustness. Our method achieves 1.16 dB PSNR gain over the strongest baseline on UVEB and 3.99 dB on DRUVA.
Read the full abstract ▾
All authors Tingjun Li, Yuxi Wang, Guoqiang Zhong, Zhaoxiang Zhang
Computer Vision
HierarchiRetina: A Diabetic Retinopathy Screening and Grading Cascade with Heterogeneous Mixture-of-Experts Lesion Segmentation
Md Asif Uddin, Niloy Farhan, Rafeed Rahman et al. (+1 more)
Clinical screening for diabetic retinopathy proceeds in stages, yet most automated systems grade fundus images with a single black-box network. Three gaps remain: (i) lesions differ widely in size and contrast, but segmenters typically apply the same blocks to all; (ii) ungradable images are usually excluded or treated as an extra class, and lesion-aware graders seldom test whether lesions drive the grade; (iii) staged systems often report each stage only on the images reaching it, leaving propagated errors unmeasured. HierarchiRetina, a screening, segmentation and grading cascade with inspectable stage outputs, addresses each gap. First, HSMoE-AUNet is a U-Net whose skip, bottleneck and decoder blocks are sparse mixtures of heterogeneous experts, with lesion-matched types, counts and losses; SwinHRUNetPP, a dual-encoder vessel segmenter, uses a field-of-view-restricted loss. Second, LG-DRG separates gradability from ordinal severity and lets lesion masks query the image through gated cross-
attention. Third, an end-to-end evaluation counts screening misses as errors and traces each error to its stage. Pooled Dice ranged from 0.184 (microaneurysms) to 0.579 (exudates); vessel Dice was 0.731. On 58,689 held-out images from six
datasets, end-to-end quadratic weighted kappa was 0.804, with referral sensitivity 0.832 at specificity 0.947, and the screening gate caused 66% of errors. In ablations, the grade ignored the masks, even with the cross-
attention gate started open. On all 4,105 official DDR test images, six-class accuracy was 85.6%, above the best published six-class result we found (83.8%). Staged screening systems should be evaluated end-to-end, with stage attribution and mask ablation, before clinical deployment.
Read the full abstract ▾
All authors Md Asif Uddin, Niloy Farhan, Rafeed Rahman, Md. Golam Rabiul Alam
Computer Vision
CineMem: Attention-Aligned Fine-grained Memory Retrieval for Multi-Shot Long Video Generation
Xiaoxue Wu, Xinyu Gao, Fang Yu et al. (+2 more)
Autoregressive
video generation relies on
memory retrieval to model long-range dependencies. This capability becomes particularly critical for multi-shot generation, where models must reconcile substantial visual changes across shot transitions with persistent semantic continuity. Existing methods tend to perform
memory retrieval at the frame level and share the same historical context across all query
tokens, overlooking the heterogeneous information needs and leading to redundant
retrieval and inefficient use of historical
memory. To bridge this gap, we introduce CineMem, an
attention-aligned fine-grained
memory retrieval framework that combines online
memory organization with query-adaptive
retrieval for autoregressive multi-shot
video generation. Specifically, guided by intrinsic
memory-access patterns revealed through
attention analysis, we propose an online-clustered KV cache as a
memory organization strategy over all historical
tokens, enabling semantic-aware
memory grouping and more precise
retrieval. In addition, we introduce a query-level
retrieval mechanism that provides broader access to historical information under a fixed
retrieval budget, thereby improving long-term
memory utilization. Together, these designs enable the multi-shot
video generator to preserve long-range consistency across shot transitions, particularly in revisitation scenarios, while approximating full-
memory access under a constrained
memory budget. To facilitate evaluation, we further introduce a suite of task-specific, VLM-assisted metrics for faithful inter-shot consistency assessment. Extensive experiments demonstrate the effectiveness of our framework.
Read the full abstract ▾
All authors Xiaoxue Wu, Xinyu Gao, Fang Yu, Yu Qiao, Xinyuan Chen
Computer Vision
LinMorph: Learning to Morph Weights for Efficient Video DiT Linearization
Lingyi Long, Congyue Zhao, Xiaoou Li et al. (+2 more)
Post-hoc linearization is an attractive way to accelerate pretrained
Video Diffusion Transformers (
Video DiTs), yet existing methods largely treat the quality loss induced by
attention replacement as a recovery problem. We show that a more fundamental bottleneck lies upstream: initialization. Random initialization discards pretrained knowledge, whereas manual weight reuse transfers it through rigid and incomplete source–target correspondences. We introduce LinMorph, which recasts
Video DiT linearization as a learnable softmax-to-linear weight transformation problem. Instead of copying predefined counterparts, LinMorph learns a cross-parameter morphism that lets each target Gated DeltaNet (GDN) parameter draw jointly from pretrained query, key, value, and output projections, including for parameters with no direct softmax counterpart. A single morphism is shared across layers and learned through local teacher
alignment, while producing layer-specific initializations from their respective pretrained weights. The resulting weights are then materialized as independent GDN modules for progressive
distillation and joint recovery.Since the morphism operates in weight space rather than
token space, it can be learned on short, low-resolution clips and directly reused at larger
video scales. Across pretrained
Video DiTs, LinMorph enables markedly faster recovery and better final generation quality than random initialization and heuristic weight reuse. Results show that efficient linearization depends not only on how a converted model is recovered, but critically on how pretrained knowledge is transferred before recovery. Project page: https://lonelyling.github.io/LinMorph.
Read the full abstract ▾
All authors Lingyi Long, Congyue Zhao, Xiaoou Li, Yanmin Qian, Tao Huang
Computer Vision · Cross-listed from cs.CL
Text-Only vs. Image-Aware VLM Judges for Manga Translation Evaluation
Subaru Kimura and Jun Suzuki
In manga translation, visual context outside
speech-bubble text, including speaker identities, referents, and scene tone, is depicted in page images, making it natural to refer to images during translation evaluation. However, most conventional manga translation evaluations rely solely on text-only automatic metrics, leaving the impact of image input insufficiently explored. We propose a practical validation framework that employs the same
vision-
language model (VLM) as both a text-only judge (TOJ) and an image-aware judge (IAJ) to investigate evaluation divergences associated with image-aware evaluation procedures. We use a Japanese-English manga
dataset of official translations aligned at the page and
speech-bubble levels. First, adding page images to TOJ improved the primary VLM's detection of deliberately injected, visually dependent degradations. Second, we analyzed evaluation divergences between TOJ and IAJ, finding different error counts and error-category distributions. Human verification did not reveal a consistent preference between TOJ and IAJ.
Read the full abstract ▾
Computer Vision
DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving
Shuzheng Wang, Haoxuan Li, Shengzhen Li et al. (+5 more)
Vision-
Language Models (VLMs) and world models have demonstrated strong capabilities in semantic understanding and future scene
evolution for autonomous driving. However, effectively integrating both capabilities for trajectory generation remains challenging. In this work, we propose DriveWLA, a unified World-
Language-Action model based on Mixture-of-
Transformers (MoT) that jointly models semantic understanding, future scene dynamics, and trajectory generation. DriveWLA consists of three experts: a World Expert for future prediction, a
Language Expert for semantic understanding, and an Action Expert for trajectory generation. We further introduce Cross-Expert
Attention, which enables controlled information flow across experts while aligning semantic and dynamic representations. To improve trajectory generation beyond imitation learning, we apply Flow-GRPO reinforcement learning post-training to refine the Action Expert. Extensive experiments show that DriveWLA achieves superior performance in both open-loop and closed-loop evaluation. DriveWLA achieves 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, demonstrating strong performance among VLM-based and world-model-based planners. Without specific fine-tuning, DriveWLA further demonstrates strong zero-shot generalization to HUGSIM, nuScenes, and Bench2Drive. These results validate the effectiveness of our unified World-
Language-Action modeling, establishing DriveWLA as a generalizable framework for autonomous driving.
Read the full abstract ▾
All authors Shuzheng Wang, Haoxuan Li, Shengzhen Li, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang
Computer Vision
Visual Dependence Does Not Imply Answerability: A Systematic Analysis of Answerability Signals in Frozen InstructBLIP
Minjun Kim, Hyunmin Song and Changwon Lim
Visual question answering models often produce plausible responses even when the available visual evidence is insufficient to support a valid answer. Although prior work has introduced
benchmarks and abstention mechanisms for unanswerable visual questions, it remains unclear which signals derived from a frozen
vision-
language model reliably indicate answerability on unseen image-question pairs. We systematically evaluate answerability signals on a stratified 10K subset of UNK-VQA using frozen InstructBLIP, covering predictive uncertainty, cross-modal
alignment, visual grounding, visual dependence, internal representations, and learned probes. Answerability discrimination remains weak, with test AUROC largely confined to 0.50-0.54. CLIP
alignment yields the highest point estimate but does not significantly outperform generation confidence under paired bootstrap analysis. Targeted removal of question-relevant visual evidence causes substantially larger changes in answer confidence than same-size random masking, yet intervention sensitivity provides essentially no answerability discrimination. Increasing probe capacity, combining multiple signals, and adding reason-aware supervision do not provide robust improvements, while post-freeze cross-validation yields similarly weak discrimination. Selective-answering analysis shows limited usable coverage under low-risk operating criteria. Failure-mode analysis shows that the relative utility of individual signals varies across perturbation mechanisms. Taken together, these results show that, on our UNK-VQA subset with frozen InstructBLIP, visual dependence, uncertainty, semantic
alignment, and static representations are not robust standalone proxies for answerability. Our findings emphasize the need to distinguish a model's reliance on visual evidence from whether that evidence is sufficient to support an answer.
Read the full abstract ▾
Computer Vision
Diagnose-to-Generate: Eliminating Structural Distortions in Flow Models via Patch-Level Grounded Alignment
Yifan Li, Lishuai Gao, Tong Zhang et al. (+8 more)
Flow-based image generators achieve strong semantic synthesis but still produce localized structural distortions, such as malformed hands and limbs. Existing post-training methods typically optimize global scalar rewards, which provide weak spatial credit assignment and may alter well-formed regions while correcting local defects. We present Diagnose-to-Generate (D2G), a grounded
alignment framework that separates visual diagnosis from generative optimization. We construct Achilles-600K, a region-annotated
dataset for structural anomalies, and train ARGUS through supervised cold-start and Region-Aware Group Relative Policy Optimization to predict anomaly categories and bounding boxes. These diagnoses are converted into confidence-weighted spatial rewards for our Patch-level Flow-GRPO. We further identify that global
stochastic exploration can shift semantic regions across group rollouts, making patch-wise relative advantages spatially inconsistent. Our Trajectory-Preserving
Stochastic Sampling (TPSS) anchors early denoising with deterministic ODE trajectories and introduces stochasticity only at later steps, thereby limiting intra-group spatial drift while retaining local exploration. Experiments on FLUX.1-dev and SD3.5 show improved structural fidelity with preserved FID and CLIP scores. Consistent gains under HumanRefiner, zero-shot GPT-4.1, and blinded human evaluation further demonstrate that the improvements extend beyond the training reward model.
Read the full abstract ▾
All authors Yifan Li, Lishuai Gao, Tong Zhang, Qilong Huang, Chen Zhao, Zhuoliang Kang, Jie Hu, Yujie Zhong, Fenghe Tang, S Kevin Zhou, Xiaoming Wei
Computer Vision
LoopREG: Representation Supervision Enables Test-Time Recurrent Depth in Diffusion Transformers
Zipeng Wang, Anping Xiong, Jing Zhang et al. (+17 more)
Reusing
Transformer blocks offers a way to increase
diffusion-model computation at test time, but naive recurrence can degrade generation beyond its training depth. We introduce LoopREG, a representation-supervised recurrent
diffusion Transformer that repeatedly applies a shared block at the state supervised by a pretrained visual encoder.
Alignment is applied after the final recurrent visit during training; inference extends the same learned operator to additional visits. On ImageNet 256 x 256, LoopREG reduces SiT-B/2 FID from the local REG baseline's 15.08 to 4.91 at 400K training steps, a 67.41% reduction. Within a checkpoint trained with two visits, one extra inference visit improves FID from 14.71 to 5.34 with unchanged weights. The benefit persists on SiT-L/2 and SiT-XL/2. Controlled interventions show that useful recurrence follows the location of representation supervision, while REPA
alignment produces a smaller extrapolation benefit. These results make representation-supervised recurrence a practical route to improving
diffusion generation through additional inference depth.
Read the full abstract ▾
All authors Zipeng Wang, Anping Xiong, Jing Zhang, Xiaoya Zhu, Yi Jiang, Yanbing Xiao, Zige Zeng, Zezhou Chen, Xiang Liu, Huan Hu, Ping Chen, Kai Wang, Zhaoxiang Liu, Shiguo Lian, Ye LV, Fang Zhao, Huanlin Gao, Zhengrui Huang, Bo Shen, Yuren You
Computer Vision
MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language–Image Pretraining
Yanran Ruan, Zhixiang Wei, Shiyao Hong et al. (+5 more)
Understanding fluorescence microscopy images is fundamental to automating biological research. However, existing methods typically learn predefined concepts from closed-set data, limiting transfer across tasks and biological contexts. In this work, we introduce MicroCLIP, a
vision-
language model that learns generalizable cellular and subcellular representations from natural
language supervision for open-set image classification and
retrieval. To pretrain MicroCLIP, we construct MicroPool-2M, a
dataset of approximately 2.5 million fluorescence microscopy image-text pairs spanning diverse cellular structures,
cell lines, and imaging conditions. Aligning microscopy images with biological concepts in natural
language is challenging because most images lack descriptive captions and biological knowledge annotations. We therefore progressively establish image-text associations at three levels to provide MicroCLIP with richer semantic supervision: (1) category-level
alignment links images to standardized labels defined by subcellular structure-
cell-line combinations; (2) knowledge-level
alignment incorporates biological priors derived from these labels to capture relationships across species and
cell types; and (3) detail-level
alignment uses a
multimodal large
language model to recaption and verify image-text pairs at scale, emphasizing fine-grained visual characteristics. Extensive experiments demonstrate that MicroCLIP achieves leading performance in zero-shot classification and text-image
retrieval on microscopy images. It retrieves images with specific cellular characteristics from thousands of candidates and generalizes to unseen biological concepts without additional supervision. These results highlight MicroCLIP's potential as a foundation model for fluorescence microscopy.
Read the full abstract ▾
All authors Yanran Ruan, Zhixiang Wei, Shiyao Hong, Xiaoxiao Ma, Yuxuan Gu, Liming Ye, Huaian Chen, Yi Jin
Computer Vision
Frequency Routing and Identity Conditioning for Thermal-to-Visible Face Translation
Haojie Li, Xinlong Sun, Xiaohaoyang Lei et al. (+4 more)
Thermal-to-visible (T2V) face translation has important applications in nighttime biometric recognition, low-light surveillance, and human interpretation of thermal observations. The task remains challenging because thermal images do not directly record visible appearance, while the cross-spectral cues useful for reconstruction vary across frequency components and semantic roles. Existing spatially conditioned models tend to process these cues through a homogeneous adaptation path and do not explicitly separate local structure from global identity. We propose FreqID-Control, a parameter-efficient framework that organizes thermal guidance according to spectral reliability and semantic role while preserving the pretrained generative prior. It combines frequency-routed low-rank adaptation, which allocates separate transformation subspaces to low-frequency structure and high-frequency detail, with shared cross-spectral identity conditioning, which supplies compact global identity
tokens independently of spatial thermal
tokens. A controlled analysis on MCXFace and SpeakingFaces provides the empirical basis for this design, revealing stable low-frequency correspondence and edge-localized detail correspondence. Experiments on both
datasets demonstrate strong image fidelity and biometric identity preservation, while controlled ablations on MCXFace confirm the complementary contributions of frequency routing and shared identity conditioning.
Read the full abstract ▾
All authors Haojie Li, Xinlong Sun, Xiaohaoyang Lei, Kai Cheng, Yiguo Qiao, Xu Yang, Cunjian Chen
Computer Vision
DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction
Haolan Wei, Jiahui Wang, Junli Zha et al. (+4 more)
Multimodal large
language models (MLLMs) have achieved strong performance on diverse
vision-
language tasks, but their computational demands limit deployment under constrained budgets, particularly on edge devices. Many works focus on Knowledge
Distillation (KD) and try to transfer knowledge from powerful teacher models to students. In addition to reducing model size, visual
token compression provides a complementary route to efficiency, but introduces mismatched visual
token counts that complicate direct
token-wise knowledge transfer. To mitigate this, we propose DVTR-KD, a knowledge
distillation framework based on Dense Visual
Token Reconstruction, which reconstructs the teacher's dense visual
tokens from the student's compressed
tokens. We reconstruct visual states after the final LLM layer and at the input to an intermediate layer identified as important for visual-information readout. At the intermediate layer, reconstruction is supervised through the teacher's response predictions rather than by direct
token-wise KL or MSE matching. An auxiliary
attention-readout loss complements this response-level supervision by aligning how the student and the original teacher integrate contextual information when predicting responses. Evaluations on diverse
benchmarks demonstrate that DVTR-KD improves the performance of
token-compressed students while retaining the inference efficiency afforded by visual
token compression.
Read the full abstract ▾
All authors Haolan Wei, Jiahui Wang, Junli Zha, Xinkai Lu, Yunxiao Zou, Chao Zhou, Xiang Song
Computer Vision
What Makes a Good Visual Representation for Spatial Understanding in MLLMs?
Xiaohu Huang, Yumin Guo and Kai Han
Spatial understanding is a necessary step in moving
multimodal large
language models (MLLMs) toward
reasoning about the 3D physical world. Their visual pretraining data, however, is almost entirely 2D, so the visual
tokens processed by MLLMs often lack inherent 3D structural information. A growing number of methods inject such structure by transferring representations from external
vision or geometry models, and collectively establish that such transfer helps. Behind these methods lies a largely underexplored question: what makes a visual representation good for spatial understanding? To answer it, we first separate what is transferred from how it is transferred, and then trace the resulting differences back to properties of the representations themselves. What we transfer comes from 3D geometry models, self-supervised image and
video models,
vision-
language models, and generative models, in the form of latent features or explicit depth and point maps. We transfer it either by
distillation, using it as a supervision target, or by fusion, feeding it as an additional input. We find that a representation's 3D structure correlates strongly with how much it helps, measured by cross-view consistency for latent features and by depth and reconstruction accuracy for explicit geometry. We further show that the form of a representation dictates the mechanism it needs. Latent features gain as much from fusion as from
distillation; explicit geometry gains little from fusion, and most of its
distillation gain depends on a geometry decoding head. Guided by these findings, we distill both latent features and explicit geometry into a single MLLM. Taking only RGB
video as input, it consistently improves over its base model on both static 3D and dynamic 4D spatial
reasoning benchmarks.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.IR
FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval
Tripti Shukla, Shwetha Somasundaram, Akshat Gupta et al. (+3 more)
Evaluating fashion
retrieval systems remains challenging due to the limited availability of
benchmarks that capture the fine-grained, open-ended queries encountered in real-world search and recommendation. Existing
benchmarks often rely on manual curation, predefined taxonomies, or synthetic
prompts, limiting scalability and semantic coverage. We introduce FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion
Retrieval, an automated framework that derives user-type queries and relevance sets directly from catalog structure. Our key insight is that hierarchical discrete representations learned by Residual Quantized Variational Autoencoders (RQ-VAE) organize products into semantically coherent clusters at progressively finer granularities. We leverage this structure to construct FASHION-RQBENCH, a scalable
benchmark without manual query annotation or predefined taxonomies. We propose four complementary tasks for evaluating ranking, relevance classification, fine-grained sibling discrimination, and paraphrase robustness, alongside query- and attribute-level relevance metrics. Experiments across recent
retrieval models demonstrate that FASHION-RQBENCH differentiates model capabilities across query types and attributes. A human study further shows that
benchmark queries achieve comparable naturalness and higher intent clarity than real user queries, supporting their validity for real-world evaluation. The code and FASHION-RQBENCH
dataset will be publicly released upon acceptance.
Read the full abstract ▾
All authors Tripti Shukla, Shwetha Somasundaram, Akshat Gupta, Rohit Garg, Paul Duff, Ian Anderson
Computer Vision
QK-Edit: Revisiting Attention-based Injection in MM-DiT for Image and Video Editing
Tiancheng Shen, Zilong Huang, Xiangtai Li et al. (+6 more)
Multimodal Diffusion Transformers (MM-DiTs) have recently emerged as a powerful framework for unified text-
vision synthesis, surpassing traditional U-Net architectures in generative tasks. One key innovation lies in its
Multimodal Self-
Attention (MM-SA) interaction, where image and text
tokens are concatenated and processed via self-
attention. However, this mechanism poses significant challenges for editing, rendering conventional U-Net-based
attention manipulation methods ineffective. To address this limitation, we propose QK-Edit, a training-free framework that exploits the unique
attention dynamics of MM-DiTs for precise text-guided image and
video editing. By introducing a novel query-key manipulation strategy, our method isolates and adjusts critical
attention components to achieve an optimal balance between
prompt fidelity and structural consistency. This enables seamless edits across various tasks, including object addition, object removal, object replacement, changing background, changing material, changing color, and style transformation. Notably, it can be easily implemented with feature replacement in inference. QK-Edit demonstrates superior editing performance on state-of-the-art models, such as FLUX and HunyuanVideo, effectively bridging the gap between generative power and editable flexibility in MM-DiTs, and paving the way for scalable
multimodal content creation.
Read the full abstract ▾
All authors Tiancheng Shen, Zilong Huang, Xiangtai Li, Zhijie Lin, Jiyang Liu, Yitong Wang, Jiashi Feng, Ming-Hsuan Yang, Jun Hao Liew
Computer Vision
SPARCL: Spatially Adaptive Regional Coverage for Active Learning in Chest X-ray Analysis
Juho Jung, Seongjae Kang, Mijoo Kim et al. (+4 more)
Pixel-level annotation for chest X-ray (CXR) lesion segmentation is expensive, making active learning (AL) essential. However, existing AL strategies fail to account for the spatial distribution of selected samples, resulting in inconsistent performance across focal (e.g., aortic calcification) and diffuse (e.g., nodules) pathologies. In this paper, we propose SPARCL (Spatially Adaptive Regional Coverage for Active Learning) to dynamically balance predictive uncertainty with cumulative spatial coverage. By employing a concave coverage function, SPARCL imposes diminishing returns on previously annotated regions, naturally directing the annotation budget toward under-labeled areas. Furthermore, an
entropy gate reverts the
algorithm to standard uncertainty sampling when spatial variation is absent. This coverage objective is monotone submodular, guaranteeing near-optimal greedy batch selection. To address how aggregate metrics like Dice obscure regional failures, we further introduce two metrics—Spatial Inconsistency (SI) and Performance Coverage (PC)—to quantify anatomical
fairness across AL rounds. Evaluated on an in-house calcification cohort and public CXR
benchmarks, SPARCL reaches 95% of fully supervised performance using only 24% of the data, compared to 29% for the best baseline. Ultimately, SPARCL ensures robust spatial reliability, surfacing regional gains that aggregate metrics miss.
Read the full abstract ▾
All authors Juho Jung, Seongjae Kang, Mijoo Kim, Sungduk Cho, Yunkyoung Jun, Sunghoon Joo, Yoona Hwang
Computer Vision · Cross-listed from cs.RO
Beyond the End-Effector: Learning Whole-Arm Contact Manipulation from Human Demonstrations
Bo-Ruei Huang, Daniel Seita and Erdem B{\i}y{\i}k
Robot manipulation has traditionally relied on end-effector-centric interactions. In contrast, humans routinely exploit whole-arm contacts to improve efficiency, robustness, and adaptability, especially when the hands are occupied or when precise grasping is unnecessary. Enabling similar capabilities in
robots remains challenging: the
robot must select where on its arm to make contact, generate effective non-prehensile interactions, and satisfy task-specific pose or safety constraints on the end-effector. We propose Whole-Arm Contact Learning (WACL), a hierarchical framework that describes each whole-arm contact task with three
robot-agnostic representations and hands them to a quasi-static surface-contact optimizer: semantic keypoints that summarize the scene, a reference contact trajectory that specifies where and how the arm presses on the object, and safety-aware path constraints that a
vision-
language model generates from the task description to protect the end-effector's task. The
robot's kinematic chain enters only in this optimizer, which grounds these representations onto the arm by jointly searching over which link makes contact, where on that link, and the joint configuration that realizes it. This decomposition lets WACL learn the contact trajectory from 10 bare-hand human demonstrations per task collected without teleoperation, while new end-effector constraints change only the optimizer's input rather than any trained component. Real-world experiments on a 7-DoF arm across four whole-arm tasks show that WACL sustains contact under end-effector constraints where a strong link-pose baseline fails (78% vs. 33% success) and generalizes to new constraints without retraining. Website: https://wacl-
robot.github.io
Read the full abstract ▾
Computer Vision
Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation
Hu Cao, Yifeng Cheng, Mu He et al. (+3 more)
Multi-modal semantic segmentation adds an auxiliary sensing stream to RGB images (RGB-X), such as thermal,
LiDAR, depth, event, polarization, or light field. This can improve robustness in poor lighting and under motion blur. But different sensors often produce features that do not match well, and fusion can become unstable. To alleviate these issues, we propose a bidirectional polarity-aware cross-modality fusion (BPCF) module that effectively captures complementary cues while enhancing feature
alignment. We evaluate the framework on six modality pairings: RGB-Thermal, RGB-
LiDAR, RGB-Depth, RGB-Event, RGB-Polarization, and RGB-Light Field. Our model achieve state-of-the-art (SOTA) results on eight public
datasets, including MFNet, KITTI-360, DELIVER, DDD17, DSEC, MCubeS, ZJU, and UrbanLF. The codes and trained models will be released upon acceptance.
Read the full abstract ▾
All authors Hu Cao, Yifeng Cheng, Mu He, Mengyu Li, Yinlong Liu, Alois Knoll
Computer Vision
MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search
Minghua Chen, Xingming Liao, Weiqiang Zhong et al. (+1 more)
GUI grounding enables web
agents to translate natural-
language instructions into actionable interface locations. However, dense layouts and small, visually similar controls can lead models toward plausible but incorrect elements, while refinement around a single prediction may leave alternative targets unexplored. We propose the Multi-Hypothesis Verification Framework (MHVF) for training-free GUI grounding. It combines multi-hypothesis exploration with local visual verification using a single pretrained
vision-
language model. To explore alternative targets, MHVF constructs spatial hypotheses from multiple full-screen predictions and examines the corresponding regions through zoomed re-localization. To resolve ambiguity among the resulting candidates, it verifies their agreement with the instruction in marked local views. Verification scores then guide a second round of local search and rank candidates retained from both search levels for final selection. Experiments with five general-purpose and GUI-specific models across eight
benchmark settings demonstrate improvements in grounding accuracy in most settings. MHVF improves UI-Venus-Ground-7B by 4.3 and 2.1 percentage points on Mind2Web and VisualWebBench, respectively, and raises Qwen2.5-VL-7B's accuracy on ScreenSpot-Pro from 23.6% to 53.6%.
Read the full abstract ▾
All authors Minghua Chen, Xingming Liao, Weiqiang Zhong, Zhuowei Wang
Computer Vision
When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs
Anubhav Kataria, Shreya Ghosh, Jingwen Ye et al. (+1 more)
Understanding facial behaviour in
video requires answering three fundamental questions: when a key facial expression occurs, where the model attends on the face at that moment, and why that evidence is necessary rather than merely correlated. Today's
multimodal large
language models (MLLMs) can label and describe
video, but they rarely provide faithful explanations, especially for small facial movements that change over time. We present VideoFace-X-LLM, a framework for explainable facial
video understanding comprising two key components. First, we introduce ExplainFace, a weakly supervised
video-text corpus for facial behaviour understanding, automatically constructed from CelebV-HQ
videos and validated through human evaluation and a two-model audit (at most 2.3% contradictions), demonstrating consistent gains across two backbone architectures. Second, H-TCC (Hierarchical Temporal Spatial Causal Concepts), which finds the key temporal moments (When), the facial regions (Where), and verifies its causal necessity (Why) through targeted occlusion. Fine-tuning on ExplainFace raises appearance Macro F1 from 19.48% to 32.87% (+13.39), with gains of +4.38 on actions, +6.77 on temporal emotion, and +11.84 F1 on deepfake detection. We also measure the explanations pointing lift, face/edge lift, stability, an Adebayo sanity check, and a necessity test rather than only showing examples. Our models, code, and corpus are available on request for research use.
Read the full abstract ▾
All authors Anubhav Kataria, Shreya Ghosh, Jingwen Ye, Abhinav Dhall