Computer Vision
LLaVA-Assessor: Building the Foundation LMM For Visual Quality Assessment
Ziheng Jia, Zicheng Zhang, Jiaying Qian et al. (+2 more)
Aligning with the human visual system~(HVS) in perceiving and evaluating the quality of visual signals is a central objective of machine-
vision-based visual quality assessment systems. With the rapid progress of large multi-modal models~(LMMs), visual question answering provides a promising paradigm for building unified foundation models for visual quality assessment under multi-modal and multi-task scenarios. Inspired by the classical “perception-decision" process in HVS-based quality evaluation, we formulate visual quality assessment for LMM-based machine
vision as two complementary tasks: “quality interpretation” and “quality scoring". Centered on these objectives, we propose LLaVA-Assessor, a unified data construction and model training system. To support multi-modal inputs, we design an adaptive model architecture that enables efficient processing of both images and
videos. For data construction, we develop rigorous human annotation protocols and a novel machine-synthesis-dominated data expansion pipeline to build a large-scale and high-quality
datasets. Furthermore, we introduce a simple yet effective
prompt disentanglement strategy to alleviate training-objective confusion in multi-task learning, thereby enabling stable and coherent joint training. The resulting all-in-one LMM LLaVA-Assessor-GIGA achieves superior performance on $11$ image/
video quality scoring test sets and 4 visual quality interpretation
benchmarks. Extensive results demonstrate the effectiveness of integrating structured data construction, adaptive model design, and multi-task joint training for automated visual quality assessment. Our work provides compelling insights for developing foundation LMMs for automatic visual quality assessment. Project page at https://github.com/jzhws/LLaVA-Assessor.
Read the full abstract ▾
All authors Ziheng Jia, Zicheng Zhang, Jiaying Qian, Guangtao Zhai, Xiongkuo Min
Computer Vision · Cross-listed from cs.HC
TRACE: Transparent Retrieval for Abstract Concept Evaluation
Joseph Bingham
Recent work reports that
vision–
language models (VLMs) struggle to establish and maintain stable reference in repeated reference games. Rather than ask which VLM does best, we ask a more basic question: do you need a large pretrained VLM for this at all? On grounding a single director utterance to one of twelve tangram silhouettes, we compare six off-the-shelf VLMs against a transparent baseline that uses no learned visual representation: classical SIFT keypoint matching and a signal-quality index over retrieved images. On identical trials, the transparent baseline matches the strongest VLM (SigLIP-large) and significantly outperforms the other five, including every CLIP and OpenCLIP variant. The baseline additionally retrieves external images, so this is not a matched-information comparison; what it shows is that a learned visual representation is not the bottleneck for this task: given retrieved images, a shape-appropriate classical similarity suffices. Along the way we find that abstract-grounding ability varies widely across VLMs (15–39% top-1; chance 8.33%, humans $\approx$77–80%), so the weakness is model-specific rather than intrinsic to contrastive pretraining; on the 1{,}013-shape KiloGram
benchmark the pattern generalizes for CLIP, with per-shape difficulty tracking human shape-nameability. The pipeline is a classical, inspectable alternative rather than a learned one. We close by sketching how an explicit, inspectable representation of listener-side pact state could carry this approach into interactive multi-turn reference, which we leave to future work. Code available in supplementary material.
Read the full abstract ▾
Computer Vision
Personalized and Explainable Blood Pressure Estimation from PPG via Hybrid CNN–Morphological Features
Myung-Kyu Yi, Jongshill Lee, Jeyeon Lee et al. (+1 more)
Continuous cuffless blood pressure (BP) monitoring using photoplethysmography (PPG) offers a promising solution for personalized healthcare. However, existing methods have two major limitations. Handcrafted feature-based approaches rely on precise fiducial point detection and are limited to short-term analysis, while deep learning models, despite their accuracy, often operate as black boxes with limited physiological interpretability. To address these challenges, we propose a physiology-guided hybrid framework for personalized BP estimation that couples a convolutional neural network (CNN) branch capturing global and local waveform dynamics with a morphology-prior branch that explicitly encodes person-specific vascular characteristics. By
embedding a morphology-based feature set that explicitly encodes individual vascular characteristics, the proposed framework enhances personalization and reduces dependence on large-scale training
datasets. Evaluated on a subset of the MIMIC-III database under a subject-specific (personalized) testing protocol, the proposed personalized physiology-guided hybrid approach achieved mean absolute errors (MAEs) of 3.77 mmHg for systolic BP and 2.36 mmHg for diastolic BP, corresponding to relative improvements of 43.7% and 32.4% over a subject-specific (personalized) CNN-only baseline. SHAP-based analysis confirmed that the introduced morphology-prior features align with individual vascular characteristics, reinforcing per-subject interpretability. These findings highlight the potential of personalized, physiology-guided hybrid learning with novel morphological descriptors for accurate and explainable BP monitoring in real-world settings.
Read the full abstract ▾
All authors Myung-Kyu Yi, Jongshill Lee, Jeyeon Lee, In Young Kim
Computer Vision · Cross-listed from cs.NE
Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer
Dongcheng Zhao, Sicheng Shen, Zhenyu Yang et al. (+6 more)
Spiking
Transformers model
token interactions primarily through spiking self-
attention (SSA). However, binary query and key representations map continuous similarities to sparse and discrete relation responses, which may suppress weak relations and limit the propagation of local spatial context. To address this limitation, we introduce Spiking Local Interaction (SLI) and Adaptive Complementary Fusion (ACF). SLI establishes an
attention-independent pathway for direct information exchange among neighboring spiking
tokens using lightweight depthwise–pointwise transformations. ACF integrates SSA and SLI through layer-specific, channel-wise coefficients that adaptively balance their contributions at different network depths. The proposed design preserves the original
attention formulation and can be incorporated into different Spiking
Transformer architectures with modest parameter overhead. Experiments on ImageNet-1K, CIFAR-10, CIFAR-100, CIFAR10-DVS, and ADE20K show consistent improvements across image classification, event-based recognition, and semantic
segmentation. In particular, QKFormer with SLI and ACF achieves $84.37\%$ Top-1 accuracy on ImageNet-1K and $37.5\%$ mIoU on ADE20K, where the
segmentation model is trained without ImageNet pretraining. Ablation studies and qualitative analyses further indicate that SSA and SLI capture complementary interaction patterns and that learnable fusion consistently outperforms fixed weighting.
Read the full abstract ▾
All authors Dongcheng Zhao, Sicheng Shen, Zhenyu Yang, Zhiyuan Li, Jinyan Yu, Yongjian Wang, Tiechui Yao, Wenli Zhang, Tielin Zhang
Computer Vision · Cross-listed from cs.RO
SpotlessGS: Relightable 3D Gaussian Splatting under Dynamic Illumination for Robotic Perception
Liang Hong, Jiaxin Wei, Simon Schaefer et al. (+2 more)
Robots operating in dark or poorly lit environments rely on onboard lights, which often produce uneven illumination that degrades downstream perception tasks. Prior approaches based on 2D image enhancement lack reliable supervision and fail to preserve multi-view
geometric consistency. To address these limitations, we extend Dark Gaussian Splatting (DarkGS) toward a more accurate and flexible relightable 3D reconstruction framework. First, we eliminate the need for explicit light parameter calibration by jointly optimizing lighting parameters within the Gaussian Splatting framework. Second, we introduce a low-frequency illumination model based on spherical harmonics (SH) to capture spatially varying residual and ambient lighting effects. Third, we incorporate an MLP-based Bidirectional Reflectance Distribution Function (BRDF) to model non-Lambertian reflectance. Experiments on synthetic and real-world
datasets demonstrate that our method effectively mitigates illumination artifacts while improving rendering quality and quantitative performance over prior approaches. We further validate its benefits for robotic perception through a downstream task.
Read the full abstract ▾
All authors Liang Hong, Jiaxin Wei, Simon Schaefer, Stefan Leutenegger, Jaehyung Jung
Computer Vision
Path2ST: Hierarchical Cell-Tissue Grounded Cross-Modal Translation for Spatial Transcriptomics
Ruochen Liu and Wei Lou
Predicting spatial gene expression from hematoxylin and eosin (H&E)-stained images offers a cost-effective alternative to spatial transcriptomics (ST). However, existing methods treat H&E images as generic visual inputs and ignore their intrinsic biological hierarchy, where spatially organized
cell types collectively form functional tissue microenvironments that govern local gene expression programs. To bridge this gap, we formulate H&E-to-ST prediction as a cross-modal semantic translation task and propose Path2ST, a hierarchically grounded autoregressive framework featuring three key components: (i) a Hierarchical
Cell-Tissue Conditioning mechanism that fuses explicit and implicit cellular features with tissue-level semantic representations to construct hierarchical conditioning signals; (ii) a Scale-Adaptive Autoregressive Generation process over a hierarchical semantic vocabulary, enabling coarse-to-fine, biologically consistent expression synthesis; and (iii) SpectraLoss, a full-spectrum objective that jointly enforces ordinal fidelity, models transcriptional bursts, and aligns semantic structures with
cell types. Extensive experiments on three
datasets demonstrate state-of-the-art performance, validating that Path2ST generates highly accurate and spatially coherent transcriptomic profiles. The related code is released at https://github.com/RuochenLiu23/Path2ST.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.SP
Hardware-in-the-Loop Phase-Aware CNN for Real-Time 5G Channel Estimation
Javad Zolfaghari-Bengar, Rakibul Rony, Elisa Gomez-de-Lope et al. (+3 more)
This demo presents real-time AI-based uplink channel-estimation inference using data collected from a hardware-in-the-loop 5G platform. The data-collection setup integrates commercial RF signal generation, programmable channel emulation, an O-RAN
Radio Unit, DU emulation, and a lightweight phase-aware convolutional neural network (CNN) that estimates the channel response directly from received DMRS signals. Unlike simulation-only evaluations, the hardware-derived
dataset exposes the
estimator to practical RF and system-level impairments, including calibration mismatches, synchronization imperfections, quantization effects, phase noise, and implementation-specific nonlinearities. During the demo, attendees will observe real-time CNN inference and channel reconstruction using captured hardware-generated DMRS observations and compare the proposed CNN against Least Squares (LS) and frequency-domain LMMSE baselines. The objective is to showcase a practical AI-native physical-layer inference pipeline that combines hardware-derived 5G data with real-time neural channel estimation for future 5G-Advanced and 6G systems.
Read the full abstract ▾
All authors Javad Zolfaghari-Bengar, Rakibul Rony, Elisa Gomez-de-Lope, Alejandro Villena-Rodriguez, Abhinav Mahadevan, Nicolas Kourtellis
Computer Vision
PE-CSNet: An equivariant network architecture with learnable patch-based sparse representation
Kai Li, Haitao Long, Bo Zhang et al. (+2 more)
Compressive sensing (CS) enables accurate signal reconstruction from sparse measurements and is widely applied in medical imaging, remote sensing, and image compression. However, designing an effective, task-specific sparse transform and the corresponding optimization procedure for high-quality CS remains challenging. This process typically requires expert domain knowledge and laborious parameter tuning. To address this issue, we present a Patch-based Equivariant deep unrolling architecture, termed PE-CSNet, for accurate CS recovery. While traditional CS methods generally use predefined patch-based transform sparsity, we generalize this idea by incorporating learnable transform sparsity that adapts to the specific CS task through an optimization-driven process. Specifically, we first establish a generalized patch-based CS model, which we solve via a block coordinate descent (BCD)
algorithm. The BCD solver is then unrolled into a deep neural network, where all parameters of both the CS model and solver are learned through end-to-end training. To improve data efficiency, we introduce a
stochastic equivariant training strategy that exploits the patch-wise structure of the network, enabling PE-CSNet to learn effectively even from limited data. We further provide a simpler, parameter-shared version of PE-CSNet and briefly discuss its convergence as an iterative solver. For practical applications, the network uses stage-specific (non-shared) parameters to enhance its expressive power and thereby improve its performance. On the tasks of CS
magnetic resonance imaging (CS-MRI) and CS coded diffraction patterns (CS-CDP), PE-CSNet achieves state-of-the-art accuracy with fast computational speed, outperforming traditional methods and existing deep unrolling methods.
Read the full abstract ▾
All authors Kai Li, Haitao Long, Bo Zhang, Haiwen Zhang, Zhi Zhou
Computer Vision
Equilibrium Forcing: Adaptive Video Generation Without Noise Conditioning
Hansen Jin Lillemark, Alex Rojas, Zachary Novack et al. (+5 more)
Standard autoregressive
video generation
algorithms based on
Diffusion and Flow Matching rely on rigid training objectives and static sampling schedules, limiting inference procedures from adapting to the data. We introduce
Equilibrium Forcing (EqF), a simplified framework for
video denoising generative models without noise level conditioning. EqF pioneers modular training- and inference-time designs for noise-unconditional generation that decouple learning the denoising field from sampling. This flexibility allows for inference-time
algorithms that operate in a closed loop by adapting to feedback from the sample, improving
video quality and consistency on challenging autoregressive
video generation
benchmarks. Extensive analysis elucidates exactly how removing the noise level conditioning enables EqF's data-dependent inference properties to surpass the performance of standard noise level-conditional denoising
video methods.
Read the full abstract ▾
All authors Hansen Jin Lillemark, Alex Rojas, Zachary Novack, Runqian Wang, Yilun Du, Yian Ma, Taylor Berg-Kirkpatrick, Rose Yu
Computer Vision
Language-Structured Relational Q-Learning for Threat-Aware Control in Safety-Critical Driving
Aditya Humnabadkar, Huaizhong Zhang and Ardhendu Behera
Natural-
language-based scenario generation offers an intuitive means of describing rare and complex driving interactions, yet it is still uncertain whether training with
language-structured data leads to truly adaptive control policies. We propose
Language-Structured Relational Q-Learning, instantiated through an Ego-Centric Relational Q-Network (ERQ-Net), which jointly learns inter-vehicle relevance and action values from dynamic traffic graphs.
Language descriptions define surrounding-vehicle behaviours during training, while
prompts and semantic actor roles are hidden from the policy. ERQ-Net must therefore infer threat relevance solely from observable kinematics and interactions. Across 2,500 safety-critical scenarios,
language-structured training improves test success from 49-52% to 55-58% and increases adversary-focused
attention from 1.2x to 2.1x, demonstrating emergent threat awareness. However, this representational gain does not consistently translate into adaptive control: trained policies perform similarly to the best constant action, while a portfolio of simple policies solves 76% of scenarios. We formalise this discrepancy as a recognition-control gap and show that reward reweighting and margin shaping do not eliminate the resulting policy collapse. Evaluations of realism, criticality, semantic accuracy, and transfer of state-interface representations to CARLA further highlight both the strengths and the constraints of
language-structured relational policy learning in safety-critical driving scenarios.
Read the full abstract ▾
Computer Vision
Mehran Tamjidi, Hamidreza Dastmalchi, Ali Cheraghian et al. (+3 more)
Object
Hallucination in large
vision-
language models (LVLMs), where models generate non-factual content about input images, remains a critical barrier to their reliability in real-world applications. Existing mitigation strategies can be categorized into training-based and training-free methods. Training-based methods often achieve strong performance but are costly, requiring extensive computational resources, large-scale data, and time-consuming fine-tuning. Training-free approaches are particularly appealing due to their efficiency. However, existing training-free methods either require multiple decoding rounds, which adds computational overhead, or modify internal states in a model-specific way that risks degrading pretrained knowledge. We propose Test-Time
Hallucination Mitigation (TTH) method, a novel training-free method that addresses both limitations. TTH introduces a
token-validator module, implemented as a zero-shot Multi-Modal Classifier (MMC), to generate auxiliary logits grounded in the input image. These logits are fused with the original LVLM outputs at the
token level for object
tokens selected from a candidate pool. An
entropy-based weighting scheme is then applied to enable robust and accurate predictions. Extensive experiments across multiple LVLM families and diverse
benchmarks demonstrate that TTH consistently improves accuracy and robustness, underscoring its generalizability and practical effectiveness. Code is released at https://github.com/Mehran-TAM/TTH
Read the full abstract ▾
All authors Mehran Tamjidi, Hamidreza Dastmalchi, Ali Cheraghian, Mohammadreza Alimoradijazi, Aijun An, Hossein Rahmani
Computer Vision
Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification
Max A. Nelson, Eminenur Sen Tasci, Zhixiang Wang et al. (+12 more)
Pancreatic cancer is among the most lethal malignancies; risk stratification of intraductal papillary mucinous neoplasms (IPMNs) offers a crucial opportunity for early intervention but typically requires invasive tissue biopsy. Dominant
vision-based approaches, including radiomics and deep learning, provide promising but initially separate discrimination opportunities. Similarly, multisequence MRI (T1W/T2W) and anatomically decomposed (head, body and tail) analysis of the pancreas provide additional and potentially complementary signals. Effective fusion of this information is crucial in ordinal IPMN dysplasia risk prediction and can be accomplished via a meticulously regularized and calibrated ensemble stacking combiner. We present cUPMI, a class-conditional Gaussian augmentation of a combiner's log-probability meta-features, and test it on various prediction paradigms. In our multi-center analysis, we find cUPMI adds limited value to properly regularized L2-logistic binary classification stacks, but consistently regularizes higher-capacity tree combiners in the binary and radiomics-only setting (RF +0.015 and XGBoost +0.024 binary AUC, positive in all seeds). Its cleanest ordinal benefit appears for XGBoost on an 8-stream radiomics task (3-class no < low < high, +0.022 QWK in all seeds). Separately, fold-locked fusion of radiomics and 2.5D CNN streams yields the strongest overall model, an RF stack reaching QWK 0.595 (95% CI [0.54, 0.64]) and binary AUC 0.839, surpassing radiomics, 2.5D ResNet, and 3D DenseNet-121 baselines.
Read the full abstract ▾
All authors Max A. Nelson, Eminenur Sen Tasci, Zhixiang Wang, Zongwei Zhou, Halil Ertugrul Aktas, Andrea M. Bejar, Elif Keles, Ziliang Hong, S{\i}tk{\i} Safa Taflan, Muhammed Enes Tasci, Frank H. Miller, Michael B. Wallace, Rajesh N. Keswani, Gorkem Durak, Ulas Bagci
Computer Vision
Multi-Agent Target-Existence Verification and Learned Mask Geometry Refinement: Winning Report of the MeViS-Text Track at the 8th LSVOS Challenge 2026
Jungyoon Lee, Gyuil Lim, Doeon Kim et al. (+1 more)
We present the first-place solution to the MeViS-Text track of the 8th Large-scale
Video Object
Segmentation (LSVOS) Challenge 2026: referring
video object
segmentation guided by written motion expressions, including deceptive no-target expressions that match no object in the
video and must yield empty masks in every frame. Our pipeline, SSUPER, resolves each expression into a visual concept, generates full-
video candidate masklets with SAM~3.1, and selects target IDs. At every
reasoning stage, three heterogeneous
multimodal large
language models independently execute the same stage-specific
prompt before a single synthesis pass commits one schema-validated verdict. Although this system rejects every no-target expression in validation, the leaderboard reveals that a substantial share of test no-target cases still slips through. The reason is that hard negatives name a plausible object and fail only under the complete temporal predicate, so when selection and existence are decided together, a category-plausible masklet anchors the verdict. Hence, we decouple existence verification into an independent multi-
agent audit of the full predicate (category, count, action, trajectory, event order, and semantic role) that distinguishes absence from temporary invisibility, discounts apparent motion caused by camera movement, and requires contradicting evidence rather than mere uncertainty for a no-target verdict. Without any new
segmentation call, this audit recovers most of the residual no-target errors. A training-data-only StyleRefiner then aligns mask geometry with the annotation style of MeViSv2 while preserving every presence decision by construction, showing that once the semantics are fixed, part of the remaining error is stylistic rather than semantic. The complete system reaches a Final score of 0.9081339614 on the official challenge leaderboard.
Read the full abstract ▾
All authors Jungyoon Lee, Gyuil Lim, Doeon Kim, Seong-heum Kim
Computer Vision
TangPoetryBench: A Multi-Dimensional Benchmark and Rubric-Conditioned Evaluator for Poetry-to-Image Generation
Haoqi Hu, Tongji Luo, Li Zhang et al. (+1 more)
Text-to-image (T2I) models are increasingly asked to illustrate literary and cultural content, yet we cannot measure how well an image renders the meaning of a poem. The task is many-sided: a good illustration must be visually sound, faithful to the poem's imagery and scene, culturally and stylistically apt, free of spurious text, and true to its
emotion, and its deepest requirements, imagery and especially implicit
emotion, are never stated in the words. Existing metrics (CLIPScore, BLIPScore, VQAScore) reward literal text-image correspondence and so cannot tell whether an illustration succeeds, let alone why, or even separate the best model from the worst. We introduce TangPoetryBench, a multi-dimensional
benchmark of 1,280 images (320 classical Chinese Tang poems x 4 state-of-the-art T2I models) with quality-controlled human annotations across ten dimensions. Analyzing this data, we reveal the shared and model-specific strengths and weaknesses of current T2I models, including their ability to evoke a poem's implicit
emotion. We further introduce PoemAutoEvaluator (PAE), an open, rubric-conditioned evaluator that reaches parity with a strong proprietary judge (Claude), generalizes to an unseen generator and a second poetic tradition (Song Ci), and lets the
benchmark scale to new images without fresh human annotation. We release the
benchmark, annotations, and evaluator.
Read the full abstract ▾
All authors Haoqi Hu, Tongji Luo, Li Zhang, Boning Zhou
Computer Vision · Cross-listed from cs.AI
Benchmarking LLM Judges for Mobile Agent Evaluation
Ziqiang Wang, Li Gu, Zhixiang Chi et al. (+4 more)
Mobile
agent benchmarks increasingly rely on LLM-based judges to evaluate task completion, yet the reliability of these judges on mobile
agent trajectories remains largely unexamined. We introduce MobileJudgeBench, a
benchmark for systematically evaluating LLM-as-judge methods on mobile
agent trajectories. Our
benchmark comprises 931 human-annotated trajectories spanning 6 mobile
agent benchmarks, 4
agent models, and 68 apps. We evaluate 6 judge methods (five adapted from SPA-Bench, A3 with two modes, AndroidArena, and AgentRewardBench, plus a simple baseline we design) across multiple LLM backends. Our experiments reveal three key findings. First, a simple baseline judge with sampled screenshots is competitive with, and often exceeds, purpose-built methods, indicating that more elaborate judge pipelines do not consistently improve judge quality; among competitive methods, the LLM backbone is the primary driver. Second,
benchmark quality metrics reliably predict real-world judge utility: they correlate with both
agent ranking fidelity for evaluation and downstream performance when judges serve as reward signals for on-policy reinforcement learning. Third, failure analysis across two LLM backends uncovers qualitatively opposite failure profiles, one conservative and the other permissive, linked to the backbone's precision-recall characteristics.
Read the full abstract ▾
All authors Ziqiang Wang, Li Gu, Zhixiang Chi, Zhi Liu, Seyed Mehdi Ayyoubzadeh, Yuanhao Yu, Yang Wang
Computer Vision
VLMs Win a Systematic Evaluation of Underwater Image Reconstruction
Sara Aghajanzadeh, Yingxue Wang, Ieva Bagdonaviciute et al. (+1 more)
Underwater image restoration consists of recovering an image which looks like there is no water present. To date, evaluation has not been systematic. This paper describes a systematic evaluation pipeline for
underwater reconstruction, which can be used to assess a method for accuracy; consistency of reconstruction over camera moves; and the effect of water parameters. We use this pipeline to evaluate a range of current procedures, from models constructed using explicit but approximate physical models of scattering to
Vision-
Language Models (VLMs which are not currently trained with explicit physical models). Overall, VLMs wholly and significantly outperform physically based models in our evaluation, likely because of the importance of a strong image prior. Results on images of real
underwater scenes strongly confirm the evaluation.
Read the full abstract ▾
All authors Sara Aghajanzadeh, Yingxue Wang, Ieva Bagdonaviciute, David Forsyth
Computer Vision · Cross-listed from cs.LG
Click2Poly: A VLM for vector mapping buildings and walls
Nicolas Girard, Jawher Ben Abdallah, Arno Gobbin et al. (+2 more)
Accurate vector mapping of buildings and walls is critical for geospatial applications but remains a labor-intensive process. While recent deep learning methods have improved automatic extraction, in order to meet cartographic standards they always require a human to perform quality control and fix complex cases in the extraction. We present Click2Poly, a human-in-the-loop AI assistant designed to speed up this manual step. Extending the Florence-2
Vision Language Model (VLM), Click2Poly responds to user clicks by editing the building or wall vector layer directly. Implemented as a QGIS plugin, Click2Poly speeds up the manual editing of building and wall vector layers in a real-world production environment.
Read the full abstract ▾
All authors Nicolas Girard, Jawher Ben Abdallah, Arno Gobbin, Liuyun Duan, Sacha Lepretre
Computer Vision · Cross-listed from cs.LG
Analysis of Federated Aggregation under Model Poisoning and Backdoor Attacks: A Reconstructed Cross-Dataset and Cross-Architecture Benchmark
Soumya Mazumdar, Vineet Kumar Rakesh and Tapas Samanta
Robust comparisons of
federated aggregation methods require joint consideration of predictive performance, threat definitions, metric semantics, and execution provenance. A 500-
cell seed-1 evaluation
matrix was reconstructed across five aggregation methods, five
datasets, five architectures, and four recorded conditions: clean, sign-flipping, Gaussian, and BadNets. Successful execution logs were identified for 454 original runs and 36 repaired or rerun executions, whereas 10 clean SVHN
cells were supported by summary-only provenance. Trimmed Mean achieved the highest clean macro-mean accuracy (76.02%) and the lowest mean within-task rank (1.70). Krum attained the highest recorded accuracy under both sign-flipping and Gaussian configurations. These relative rankings remained unchanged when analysis was restricted to 21 task pairs for which original successful logs were available for every method-condition combination. Audit of the supplied BadNets metric implementation established that every test input is triggered prior to target-label counting; consequently, the retained metric represents Triggered Target-Label Rate (TTLR) rather than a conventional target-excluding attack success rate. An audit of the supplied FedPARETO scaffold further identified a pathway in which predictive summaries may characterize an uncorrupted local model while the aggregation weight is applied to a separately corrupted update, introducing a potential discrepancy between reported predictive outcomes and the updates used for aggregation. The canonical
matrix contains a single identified seed for each
cell, and exact attack and configuration lineage is incomplete. Accordingly, the findings should be interpreted as descriptive comparisons within the recorded configurations and not as statistical estimates or universal claims regarding robustness.
Read the full abstract ▾
Computer Vision
COGENT: Counterfactual Gaussian Explanations for Volumetric Medical Images
Dorian Rząsa, Bartosz Zabdyr, Krzysztof Piekarz et al. (+7 more)
Explainability is essential for deploying deep learning models in high-stakes medical applications. Existing explainability methods for volumetric imaging predominantly operate in voxel space, overlooking the structured representations introduced by recent advances in 3D scene modeling. We present COGENT (Counterfactual Gaussian Explanations), a framework that generates counterfactual explanations directly in the parameter space of Gaussian-based volumetric representations. Built upon MedGS and the Sybil lung cancer risk prediction model, COGENT optimizes selected Gaussian primitives through a differentiable rendering pipeline, enabling gradients from the downstream predictor to identify representation components that most influence model decisions. Unlike conventional pixel- or voxel-level attribution methods, our approach formulates explainability as a counterfactual optimization problem over an explicit 3D scene representation, producing sparse and spatially localized explanations while preserving anatomical consistency. We evaluate COGENT on lung CT scans using quantitative comparisons with existing explainability methods together with qualitative analysis by medical experts. The results demonstrate that representation-space counterfactual optimization provides clinically meaningful explanations while offering a new perspective on interpreting volumetric deep learning models.
Read the full abstract ▾
All authors Dorian Rząsa, Bartosz Zabdyr, Krzysztof Piekarz, Jakub Grzywaczewski, Bartlomiej Sobieski, Przemyslaw Biecek, Żaneta Świderska-Chadaj, Olga Śliwicka, Przemysław Spurek, Joanna Świebocka-Więk
Computer Vision
Gaze Target Estimation Anywhere with Concepts
Xu Cao, Houze Yang, Vipin Gunda et al. (+5 more)
Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural
language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual
prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a
dataset and
benchmark of 120K high-quality,
prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a
transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE
benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical
dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere.
Read the full abstract ▾
All authors Xu Cao, Houze Yang, Vipin Gunda, Zhongyi Zhou, Tianyu Xu, Adarsh Kowdle, Inki Kim, James M. Rehg
Computer Vision · Cross-listed from cs.AI
Can Frontier LLMs Match Natively Multimodal Embeddings? A Comparison on Hard-Negative Text-to-Image Retrieval
Archan Dutta and Vyanktesh Kanungo
Multimodal retrieval and classification across different types of media, spanning text, images,
video and audio, has traditionally relied on dual-encoder models that align visual and textual representations through contrastive learning. The March 2026 release of Gemini
Embedding 2, Google's first natively
multimodal embedding model to map text, images,
video, audio, and documents into a single shared space, raises competition among
multimodal retrieval systems. Simultaneously, frontier Large
language models (LLMs) have also demonstrated strong visual understanding, raising the question of whether they can serve as effective zero-shot rankers. Our study provides the first direct comparison of native
multimodal embeddings against LLM-based visual ranking on Flickr30k. We observe that GPT-4.1 and Claude Sonnet 4.6 perform on par with Gemini
Embedding 2. Additionally, once
embeddings are precomputed,
multimodal embeddings are better suited for low-latency applications.
Read the full abstract ▾
Computer Vision
Dual-Domain Cross-Modal Decoding for Clinical Text-Guided Medical Image Segmentation
Md Maklachur Rahman and Tracy Hammond
Clinical text can narrow down what to segment, but recent text-guided designs emphasize spatial
alignment while overlooking frequency content that governs texture and boundaries. We propose Dual-Domain Cross-Modal Decoding (DD-CMD) for clinical text-guided pulmonary infection
segmentation, integrating two complementary forms of
language guidance during decoding. In the spatial domain, Text-Guided Spatial Cross-
Attention (TGSA) aligns multi-scale visual
tokens with text semantics and updates features through gated residual fusion. In the frequency domain, Spectral-Text Adaptive Modulation (STAM) applies a 2D DCT to compute learnable band-energy statistics and predicts text-conditioned FiLM parameters to recalibrate decoder channels for frequency-aware decoding. DD-CMD embeds TGSA and STAM into a coarse-to-fine decoder (7x7 to 56x56) and restores full-resolution masks using a lightweight two-stage refinement module. Experiments on QaTa-COV19 and MosMedData+ show that DD-CMD achieves 91.46% Dice / 84.26% mIoU and 81.95% Dice / 69.42% mIoU, respectively, with average gains of +1.96 Dice and +2.67 mIoU over the strongest prior baselines. Code: https://github.com/maklachur/DD-CMD.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Gloss-Free Representation Learning for Cross-Dataset Sign Spotting
Oğuz Akif Tüfekcioğlu, Ezgi Ekin, Mustafa Kaan Çevik et al. (+1 more)
Sign-
language research for resource-constrained
languages is often limited by the cost of dense linguistic labels such as glosses, temporal boundaries, and sign order. Broadcast news offers a practical alternative by pairing continuous signing with spoken-
language transcripts, but this supervision is weak since text and signing are loosely aligned. Morphologically rich
languages such as Turkish add further difficulty, as the same lexical meaning can appear in many inflected forms while some derived forms should remain distinct. We study whether weak transcript-based supervision can pretrain a reusable sign encoder in this setting, where poor text normalization can fragment pseudo-gloss targets and weaken representation learning. Unlike prior pseudo-gloss pipelines designed mainly to improve translation, we test whether the pretrained encoder transfers as a reusable representation for cross-
dataset sign spotting. We pretrain on TSL-News, a new Turkish broadcast corpus, using pseudo-gloss labels derived from transcripts rather than manual annotation, comparing rule-based morphological lemmatization with constrained LLM-assisted normalization over a fixed vocabulary. We evaluate the learned representations via cross-
dataset sign spotting on a new TSL Spotting
Benchmark built from the TSL Dictionary corpus. The LLM-assisted encoder raises top-5 temporal localization mean IoU from 0.235 to 0.465, with 56.2% of examples reaching an IoU of at least 0.50; a frequency analysis suggests this gain is not mainly driven by memorizing frequent pseudo-gloss labels. In a downstream translation check, the same pretraining improves BLEU-4 from 9.60 to 11.04 and ROUGE from 23.48 to 27.43. These results show that loosely aligned broadcast data can provide effective weak supervision for learning sign representations that capture both lexical content and temporal structure.
Read the full abstract ▾
All authors Oğuz Akif Tüfekcioğlu, Ezgi Ekin, Mustafa Kaan Çevik, Hacer Yalim Keles
Computer Vision · Cross-listed from cs.SD
Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA
Maryam Dehdashti
A common approach to adding audio to a
vision-
language model is to train or adapt a large omni-modal system. We show that a lightweight alternative can be highly effective for music audio-visual question answering (AVQA). Qwen-MusicAVQA-7B connects a frozen Whisper encoder to Qwen2-VL-7B-Instruct through learned linear projections. The same frozen encoder processes both the
video's music track and a TTS-spoken question through separate projectors, while the
language model fuses visual frames, music, and question audio through pretrained self-
attention, with no task-specific fusion network. On MUSIC-AVQA, our system reaches 96.0% +/- 3.9% accuracy across three independent training seeds on the 7,402-question available-
video test subset. Our central finding is that downstream accuracy tracks how much fine-grained local temporal information the audio representation preserves. In a matched 32-
token comparison, a stride-pooled Whisper frame sequence outperforms a globally pooled PANNs representation expanded to the same budget by 26 percentage points, even though PANNs sees at least as much audio and uses a far larger projector. The effect is not simply sequence versus vector: within Whisper alone, reducing temporal resolution at a fixed
token budget costs a comparable amount. Under matched data and inputs, fine-tuned Qwen2.5-Omni-7B reaches 80.9%, against 95.9% for our 30 s variant; because the systems differ in backbone and adaptation, this is a system-level comparison. Accuracy remains high on sampled head and tail splits of the rephrased MUSIC-AVQA-R
benchmark (96.5% and 95.6%). Because both encoders stay frozen and the music features are cached, the entire adaptation is cheap to train: the complete two-stage AVQA run takes approximately 5 hours on a single A100 80GB, and every run reported here fits on that one GPU.
Read the full abstract ▾
Computer Vision
Clinical Feasibility of Low-Magnification Fluorescence Imaging for Breast Cancer Margin Detection Using Texture Analysis and Deep Learning
Pouya Afshin, Tianling Niu, Tongtong Lu et al. (+6 more)
High-resolution images of unprocessed
surgical breast tissue can be obtained using microscopy with ultraviolet surface excitation (MUSE). This technique is considered a promising method for checking
surgical margins during breast cancer surgery. In this study, MUSE images at 4x and 10x magnifications were compared using patch-level classification methods. Texture analysis (TA) based on local binary patterns (LBP) and deep learning (DL) with a base
Vision Transformer (ViT) model were used. Both methods achieved similar performance at both magnifications. Using DL method, both 4x and 10x magnifications achieved 96.30% sensitivity, 100% specificity and 98.18% accuracy. Using TA method, 4x achieved better specificity (100% vs 93.33%) and 10x yielded higher sensitivity (100% vs 93.33%), but both had the same accuracy (96.67%). No clear improvement in performance was observed with 10x magnification. These results show that 4x imaging achieves the same diagnostic accuracy as 10x imaging. At the same time, 4x offers a larger field of view and faster image capture. Therefore, lower magnification can be effectively used in MUSE systems for accurate and efficient intraoperative margin assessment.
Read the full abstract ▾
All authors Pouya Afshin, Tianling Niu, Tongtong Lu, David Helminiak, Julie Jorns, Mollie Patton, Tina Yen, Donghye Ye, Bing Yu
Computer Vision
Self-Evolving Code-with-Image Reasoning
Tianze Yang, Liang Wu, Ruitong Sun et al. (+6 more)
Multimodal models increasingly reach for tools when solving visual tasks (crop, zoom, rotate, brighten), a paradigm known as thinking-with-images. The central challenge is one of perception: tools mostly serve to expose visual evidence,
reasoning over that evidence stays in
language, and most targets are ones a human could in principle determine by inspection. Some visual questions, however, are not bottlenecked by perception: recovering their answers requires executing a multi-step visual
algorithm over the pixels. On such questions a model often names the correct
algorithm at once yet still answers wrong, because
language can describe an
algorithm without being able to run one. Code-with-Image crosses that line: given nothing but a Python interpreter, the model must implement a genuine visual
algorithm in code to solve the task; the program itself becomes the
reasoning. The bottleneck then shifts from executing code to deciding which
algorithm to implement. So we let the model teach itself: a training-free reflection loop studies its own failed programs, tests repairs against constructive ground truth, and keeps what survives as portable skills. On our Code-with-Image Bench (CwI-Bench), thirty task families induced by hidden visual computations with disjoint learning and evaluation splits, even GPT-5.6-luna stays below 30% with tool-free chain of thought; given a bare interpreter it reaches 43%, and with skills evolved through its own executable reflection, 67%. The open 27B model climbs the same ladder (9% $\rightarrow$ 33% $\rightarrow$ 56%), and the skills are plain text, transferable across scales and families. When code carries the
reasoning, debugging code becomes debugging
reasoning.
Read the full abstract ▾
All authors Tianze Yang, Liang Wu, Ruitong Sun, Yucheng Shi, Yanqiao Wang, Mayank Darbari, Ninghao Liu, Jin Sun, Liangjie Hong
Computer Vision · Cross-listed from eess.IV
Uncertainty-Aware Compositional Localization and Placement Assessment of Catheters and Tubes in Chest X-Rays
Harshil Lodhiya
Assessing catheter and tube placement on chest X-rays is safety-critical yet tedious and error-prone. Current deep learning methods either classify placement globally – losing track of which device is where – or segment all devices into a single mask, making per-device assessment impossible when catheters overlap. We introduce UCompCXR, a compositional framework that detects local catheter fragments, associates them into device instances via graph-based clustering, fuses per-fragment tip predictions through precision-weighted Gaussian estimation, and classifies placement per device. On the RANZCR CLiP
dataset (30,083 images, 5-fold patient-level CV with bootstrap CIs), UCompCXR detects 26% more devices than a strong multi-task baseline sharing the same MobileNetV3 backbone, with 75% fewer false positives and well-calibrated tip uncertainty (95% coverage = 0.948). The aggregate tip error rises – but only because the model finds devices the baseline misses entirely, especially nasogastric tubes. On matched devices, catastrophic localization failures drop substantially. At 2.27M parameters in a single forward pass, the model is deployable on resource-constrained clinical hardware.
Read the full abstract ▾
Computer Vision
CLEAR: Class-wise Expert Aggregation with Structured Sampling for Long-Tailed Classification
Gawon Lim
Long-tailed classification poses a reliability challenge because models trained on imbalanced data are unevenly reliable across frequent and underrepresented classes. While existing methods address imbalance through re-balancing, adjustment, representation learning, or multi-expert modeling, they rarely estimate which expert should be trusted for each class. This paper proposes CLEAR (Class-wise reLiability-aware Expert Aggregation for long-tailed Recognition), a modular ensemble framework for long-tailed classification. CLEAR generates diverse experts through threshold-based structured sampling while preserving the full label space, then estimates a class-wise trust score for each expert using a smoothed class-wise precision formulation. During inference, expert predictions are combined through class-wise generalized product-of-experts aggregation, allowing different experts to be emphasized for different classes. Experiments on CIFAR-100-LT, ImageNet-LT, and Places-LT across multiple backbones show that CLEAR achieves competitive overall accuracy and particularly strong few-shot performance. These results support class-wise expert reliability as a useful design principle for long-tailed ensemble learning.
Read the full abstract ▾