Computer Vision
DCG-Net: Dual Cross-Attention with Concept-Value Graph Reasoning for Interpretable Medical Diagnosis
Getamesay Dagnaw, Xuefei Yin, Muhammad Hassan Maqsood et al. (+2 more)
Deep learning models have achieved strong performance in medical image analysis, but their internal decision processes remain difficult to interpret. Concept Bottleneck Models (CBMs) partially address this limitation by structuring predictions through human-interpretable clinical concepts. However, existing CBMs typically overlook the contextual dependencies among concepts. To address these issues, we propose an end-to-end interpretable framework DCG-Net that integrates
multimodal alignment with structured concept
reasoning. DCG-Net introduces a Dual Cross-
Attention module that replaces cosine similarity matching with bidirectional
attention between visual
tokens and canonicalized textual concept-value prototypes, enabling spatially localized evidence attribution. To capture the relational structure inherent to clinical concepts, we develop a Parametric Concept Graph initialized with Positive Pointwise Mutual Information priors and refined through sparsity-controlled message passing. This formulation models inter-concept dependencies in a manner consistent with clinical domain knowledge. Experiments on white blood
cell morphology and skin lesion diagnosis demonstrate that DCG-Net achieves state-of-the-art classification performance while producing clinically interpretable diagnostic explanations.
Read the full abstract ▾
All authors Getamesay Dagnaw, Xuefei Yin, Muhammad Hassan Maqsood, Yanming Zhu, Alan Wee-Chung Liew
Computer Vision
NCSTR: Node-Centric Decoupled Spatio-Temporal Reasoning for Video-based Human Pose Estimation
Quang Dang Huynh, Xuefei Yin, Andrew Busch et al. (+4 more)
Video-based human pose estimation remains challenged by motion blur, occlusion, and complex spatiotemporal dynamics. Existing methods often rely on heatmaps or implicit spatio-temporal feature aggregation, which limits joint
topology expressiveness and weakens cross-frame consistency. To address these problems, we propose a novel node-centric framework that explicitly integrates visual, temporal, and structural
reasoning for accurate pose estimation. First, we design a visuo-temporal velocity-based joint
embedding that fuses sub-pixel joint cues and inter-frame motion to build appearance- and motion-aware representations. Then, we introduce an
attention-driven pose-query encoder, which applies
attention over joint-wise heatmaps and frame-wise features to map the joint representations into a pose-aware node space, generating image-conditioned joint-aware node
embeddings. Building upon these node
embeddings, we propose a dual-branch decoupled spatio-temporal
attention graph that models temporal propagation and spatial constraint
reasoning in specialized local and global branches. Finally, a node-space expert fusion module is proposed to adaptively fuse the complementary outputs from both branches, integrating local and global cues for final joint predictions. Extensive experiments on three widely used
video pose
benchmarks demonstrate that our method outperforms state-of-the-art methods. The results highlight the value of explicit node-centric
reasoning, offering a new perspective for advancing
video-based human pose estimation.
Read the full abstract ▾
All authors Quang Dang Huynh, Xuefei Yin, Andrew Busch, Hugo G. Espinosa, Alan Wee-Chung Liew, Matthew T.O. Worsey, Yanming Zhu
Computer Vision
MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints
Yu Qi, Xinyi Xu, Ziyu Guo et al. (+10 more)
Video generative models show emerging
reasoning behaviors. It is essential to ensure that generated events remain causally consistent across frames for reliable deployment, a property we define as
reasoning coherence. To bridge the gap in literature for missing
reasoning coherence evaluation, we propose MME-CoF-Pro, a comprehensive
video reasoning benchmark to assess
reasoning coherence in
video models. Specifically, MME-CoF-Pro contains 303 samples across 16 categories, ranging from visual logical to scientific
reasoning. It introduces
Reasoning Score as evaluation metric for assessing process-level necessary intermediate
reasoning steps, and includes three evaluation settings, (a) no hint (b) text hint and (c) visual hint, enabling a controlled investigation into the underlying mechanisms of
reasoning hint guidance. Evaluation results in 7 open and closed-source
video models reveals insights including: (1)
Video generative models exhibit weak
reasoning coherence, decoupled from generation quality. (2) Text hints boost apparent correctness but often cause inconsistency and hallucinated
reasoning (3) Visual hints benefit structured perceptual tasks but struggle with fine-grained perception. Website: https://
video-
reasoning-coherence.github.io/
Read the full abstract ▾
All authors Yu Qi, Xinyi Xu, Ziyu Guo, Siyuan Ma, Renrui Zhang, Xinyan Chen, Ruichuan An, Ruofan Xing, Jiayi Zhang, Haojie Huang, Pheng-Ann Heng, Jonathan Tremblay, Lawson L.S. Wong
Computer Vision
From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering
Xinyi Shang, Yi Tang, Jiacheng Cui et al. (+9 more)
Existing tampering detection
benchmarks largely rely on object masks, which severely misalign with the true edit signal: many pixels inside a mask are untouched or only trivially modified, while subtle yet consequential edits outside the mask are treated as natural. We reformulate VLM image tampering from coarse region labels to a pixel-grounded, meaning and
language-aware task. First, we introduce a taxonomy spanning edit primitives (replace/remove/splice/inpaint/attribute/colorization, etc.) and their semantic class of tampered object, linking low-level changes to high-level understanding. Second, we release a new
benchmark with per-pixel tamper maps and paired category supervision to evaluate detection and classification within a unified protocol. Third, we propose a training framework and evaluation metrics that quantify pixel-level correctness with localization to assess confidence or prediction on true edit intensity, and further measure tamper meaning understanding via semantics-aware classification and natural
language descriptions for the predicted regions. We also re-evaluate the existing strong
segmentation/localization baselines on recent strong tamper detectors and reveal substantial over- and under-scoring using mask-only metrics, and expose failure modes on micro-edits and off-mask changes. Our framework advances the field from masks to pixels, meanings and
language descriptions, establishing a rigorous standard for tamper localization, semantic classification and description. Code and
benchmark data are available at https://github.com/VILA-Lab/PIXAR.
Read the full abstract ▾
All authors Xinyi Shang, Yi Tang, Jiacheng Cui, Ahmed Elhagry, Salwa K. Al Khatib, Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Jing-Hao Xue, Hao Li, Salman Khan, Zhiqiang Shen
Computer Vision
LumosX: Relate Any Identities with Their Attributes for Personalized Video Generation
Jiazheng Xing, Fei Du, Hangjie Yuan et al. (+7 more)
Recent advances in
diffusion models have significantly improved text-to-
video generation, enabling personalized content creation with fine-grained control over both foreground and background elements. However, precise face-attribute
alignment across subjects remains challenging, as existing methods lack explicit mechanisms to ensure intra-group consistency. Addressing this gap requires both explicit modeling strategies and face-attribute-aware data resources. We therefore propose LumosX, a framework that advances both data and model design. On the data side, a tailored collection pipeline orchestrates captions and visual cues from independent
videos, while
multimodal large
language models (MLLMs) infer and assign subject-specific dependencies. These extracted relational priors impose a finer-grained structure that amplifies the expressive control of personalized
video generation and enables the construction of a comprehensive
benchmark. On the modeling side, Relational Self-
Attention and Relational Cross-
Attention intertwine position-aware
embeddings with refined
attention dynamics to inscribe explicit subject-attribute dependencies, enforcing disciplined intra-group cohesion and amplifying the separation between distinct subject clusters. Comprehensive evaluations on our
benchmark demonstrate that LumosX achieves state-of-the-art performance in fine-grained, identity-consistent, and semantically aligned personalized multi-subject
video generation. Code and models are available at https://jiazheng-xing.github.io/lumosx-home/.
Read the full abstract ▾
All authors Jiazheng Xing, Fei Du, Hangjie Yuan, Pengwei Liu, Hongbin Xu, Hai Ci, Ruigang Niu, Weihua Chen, Fan Wang, Yong Liu
Computer Vision
Deterministic Mode Proposals: An Efficient Alternative to Generative Sampling for Ambiguous Segmentation
Sebastian Gerard and Josephine Sullivan
Many
segmentation tasks, such as medical image
segmentation or future state prediction, are inherently ambiguous, meaning that multiple predictions are equally correct. Current methods typically rely on generative models to capture this uncertainty. However, identifying the underlying modes of the distribution with these methods is computationally expensive, requiring large numbers of samples and post-hoc clustering. In this paper, we shift the focus from
stochastic sampling to the direct generation of likely outcomes. We introduce mode proposal models, a deterministic framework that efficiently produces a fixed-size set of proposal masks in a single forward pass. To handle superfluous proposals, we adapt a confidence mechanism, traditionally used in object detection, to the high-dimensional space of
segmentation masks. Our approach significantly reduces inference time while achieving higher ground-truth coverage than existing generative models. Furthermore, we demonstrate that our model can be trained without knowing the full distribution of outcomes, making it applicable to real-world
datasets. Finally, we show that by decomposing the velocity field of a pre-trained flow model, we can efficiently estimate prior mode probabilities for our proposals.
Read the full abstract ▾
Computer Vision
CoVR-R:Reason-Aware Composed Video Retrieval
Omkar Thawakar, Dmitry Demidov, Vaishnav Potlapalli et al. (+5 more)
Composed
Video Retrieval (CoVR) aims to find a target
video given a reference
video and a textual modification. Prior work assumes the modification text fully specifies the visual changes, overlooking after-effects and implicit consequences (e.g., motion, state transitions, viewpoint or duration cues) that emerge from the edit. We argue that successful CoVR requires
reasoning about these after-effects. We introduce a
reasoning-first, zero-shot approach that leverages large
multimodal models to (i) infer causal and temporal consequences implied by the edit, and (ii) align the resulting reasoned queries to candidate
videos without task-specific finetuning. To evaluate
reasoning in CoVR, we also propose CoVR-Reason, a
benchmark that pairs each (reference, edit, target) triplet with structured internal
reasoning traces and challenging distractors that require predicting after-effects rather than keyword matching. Experiments show that our zero-shot method outperforms strong
retrieval baselines on recall at K and particularly excels on implicit-effect subsets. Our automatic and human analysis confirm higher step consistency and effect factuality in our retrieved results. Our findings show that incorporating
reasoning into general-purpose
multimodal models enables effective CoVR by explicitly accounting for causal and temporal after-effects. This reduces dependence on task-specific supervision, improves generalization to challenging implicit-effect cases, and enhances interpretability of
retrieval outcomes. These results point toward a scalable and principled framework for explainable
video search. The model, code, and
benchmark are available at https://github.com/mbzuai-oryx/CoVR-R.
Read the full abstract ▾
All authors Omkar Thawakar, Dmitry Demidov, Vaishnav Potlapalli, Sai Prasanna Teja Reddy Bogireddy, Viswanatha Reddy Gajjala, Alaa Mostafa Lasheen, Rao Muhammad Anwer, Fahad Khan
Computer Vision
Wildfire Spread Scenarios: Increasing Sample Diversity of Segmentation Diffusion Models with Training-Free Methods
Sebastian Gerard and Josephine Sullivan
Predicting future states in uncertain environments, such as wildfire spread, medical diagnosis, or autonomous driving, requires models that can consider multiple plausible outcomes. While
diffusion models can effectively learn such multi-modal distributions, naively sampling from these models is computationally inefficient, potentially requiring hundreds of samples to find low-probability modes that may still be operationally relevant. In this work, we address the challenge of sample-efficient ambiguous
segmentation by evaluating several training-free sampling methods that encourage diverse predictions. We adapt two techniques, particle guidance and SPELL, originally designed for the generation of diverse natural images, to discrete
segmentation tasks, and additionally propose a simple clustering-based technique. We validate these approaches on the LIDC medical
dataset, a modified version of the Cityscapes
dataset, and MMFire, a new simulation-based wildfire spread
dataset introduced in this paper. Compared to naive sampling, these approaches increase the HM IoU* metric by up to 7.5% on MMFire and 16.4% on Cityscapes, demonstrating that training-free methods can be used to efficiently increase the sample diversity of
segmentation diffusion models with little cost to image quality and runtime. Code and
dataset: https://github.com/SebastianGer/wildfire-spread-scenarios
Read the full abstract ▾
Computer Vision
NOUS: Video-Driven 3D Human Reaction Generation via Observation-Reaction Mutual Steering
Yuan Zhou, Luanyuan Dai, Yongzhi Li et al. (+7 more)
Video-driven 3D human reaction generation aims to synthesize 3D human motion in response to the action observed in a
video, playing an important role in interactive multimedia systems and
embodied agents. Yet reaction motions generated by current methods often fail to match what the observed
video calls for. We observe that one factor behind this failure is relational distortion in the correspondence between visual observations and reactions:
videos lying close in the visual space may correspond to entirely different motions in the reaction space, which misleads the model into generating reactions inconsistent with the conditioning
video. This motivates us to propose a new observatioN-reactiOn mUtual Steering (NOUS) framework that enables mutual steering between the
video and motion modalities. It first performs Motion Feedback Steering (MFS), equipping the frozen pretrained
video encoder with a lightweight rectification modulator and training the modulator with a relational margin loss that pulls each
video embedding toward the motion prototype of its own category and away from those of other categories. In this way, the misaligned correspondence between visual observations and reactions can be calibrated. NOUS then applies Observation-Guided Refinement (OGR), which in turn exploits the rectified observations to further refine the generated reactions and enhance their quality. The results on the ViMo
dataset demonstrate that NOUS improves the quality of reaction motion while incurring negligible computational overhead at inference. Also, NOUS yields consistent gains across four pretrained
video encoders, showing its good compatibility.
Read the full abstract ▾
All authors Yuan Zhou, Luanyuan Dai, Yongzhi Li, Shijie Hao, Xingyu Zhu, Yi Tan, Qingshan Xu, Beier Zhu, Richang Hong, Hanwang Zhang
Computer Vision
Improving Image-to-Image Translation via a Rectified Flow Reformulation
Satoshi Iizuka, Shun Okamoto and Kazuhiro Fukui
In this work, we propose Image-to-Image Rectified Flow Reformulation (I2I-RFR), a practical plug-in reformulation that recasts standard I2I regression networks as continuous-time transport models. While pixel-wise I2I regression is simple, stable, and easy to adapt across tasks, it often over-smooths ill-posed and
multimodal targets, whereas generative alternatives often require additional components, task-specific tuning, and more complex training and inference pipelines. Our method augments the backbone input by channel-wise concatenation with a noise-corrupted version of the ground-truth target and optimizes a simple t-reweighted pixel loss. This objective admits a rectified-flow interpretation via an induced velocity field, enabling ODE-based progressive refinement at inference time while largely preserving the standard supervised training pipeline. In most cases, adopting I2I-RFR requires only expanding the input channels, and inference can be performed with a few explicit solver steps (e.g., 3 steps) without
distillation. Extensive experiments across multiple image-to-image translation and
video restoration tasks show that I2I-RFR generally improves performance across a wide range of tasks and backbones, with particularly clear gains in perceptual quality and detail preservation. Overall, I2I-RFR provides a lightweight way to incorporate continuous-time refinement into conventional I2I models without requiring a heavy generative pipeline.
Read the full abstract ▾
Computer Vision
VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking
Jingyang Lin, Jialian Wu, Jiang Liu et al. (+6 more)
Video agentic models have advanced challenging
video-
language tasks. However, most agentic approaches still heavily rely on greedy parsing over densely sampled
video frames, resulting in high computational cost. We present VideoSeek, a long-horizon
video agent that leverages
video logic flow to actively seek answer-critical evidence instead of exhaustively parsing the full
video. This insight allows the model to use far fewer frames while maintaining, or even improving, its
video understanding capability. VideoSeek operates in a think-act-observe loop with a well-designed toolkit for collecting multi-granular
video observations. This design enables query-aware exploration over accumulated observations and supports practical
video understanding and
reasoning. Experiments on four challenging
video understanding and
reasoning benchmarks demonstrate that VideoSeek achieves strong accuracy while using far fewer frames than prior
video agents and standalone LMMs. Notably, VideoSeek achieves a 10.2 absolute points improvement on LVBench over its base model, GPT-5, while using 93% fewer frames. Further analysis highlights the significance of leveraging
video logic flow, strong
reasoning capability, and the complementary roles of toolkit design.
Read the full abstract ▾
All authors Jingyang Lin, Jialian Wu, Jiang Liu, Ximeng Sun, Ze Wang, Xiaodong Yu, Jiebo Luo, Zicheng Liu, Emad Barsoum
Computer Vision
Adaptive Greedy Frame Selection for Long Video Understanding
Yuning Huang, Xiaoyu Ji, Joseph Huang et al. (+2 more)
Large
vision–
language models (VLMs) are increasingly applied to long-
video question answering, yet inference is often bottlenecked by the number of input frames and resulting visual
tokens. Naive sparse sampling can miss decisive moments, while purely relevance-driven selection frequently collapses onto near-duplicate frames and sacrifices coverage of temporally distant evidence. We propose a question-adaptive greedy frame selection method that jointly optimizes query relevance and semantic representativeness under a fixed frame budget. Our approach constructs a 1~FPS candidate pool (capped at 1000) with exact timestamp
alignment, embeds candidates in two complementary spaces (SigLIP for question relevance and DINOv2 for semantic similarity), and selects frames by greedily maximizing a weighted sum of a modular relevance term and a facility-location coverage term. This objective is normalized, monotone, and submodular, yielding a standard (1-1/e) greedy approximation guarantee. To account for question-dependent trade-offs between relevance and coverage, we introduce four preset strategies and a lightweight text-only question-type classifier that routes each query to its best-performing preset. Experiments on MLVU show consistent accuracy gains over uniform sampling and a strong recent baseline across frame budgets, with the largest improvements under tight budgets.
Read the full abstract ▾
All authors Yuning Huang, Xiaoyu Ji, Joseph Huang, Yichi Zhang, Fengqing Zhu
Computer Vision
LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis
Stanislaw Szymanowicz, Minghao Chen, Jianyuan Wang et al. (+2 more)
Recent work has shown that neural networks can perform 3D tasks such as Novel View Synthesis (NVS) without explicit 3D reconstruction. Even so, we argue that strong 3D inductive biases are still helpful in the design of such networks. We show this point by introducing LagerNVS, an encoder-decoder neural network for NVS that builds on `3D-aware' latent features. The encoder is initialized from a 3D reconstruction network pre-trained using explicit 3D supervision. This is paired with a lightweight decoder, and trained end-to-end with photometric losses. LagerNVS achieves state-of-the-art deterministic feed-forward Novel View Synthesis (including 31.4 PSNR on Re10k), with and without known cameras, renders in real time, generalizes to in-the-wild data, and can be paired with a
diffusion decoder for generative extrapolation.
Read the full abstract ▾
All authors Stanislaw Szymanowicz, Minghao Chen, Jianyuan Wang, Christian Rupprecht, Andrea Vedaldi
Computer Vision
TinyML Enhances CubeSat Mission Capabilities
Luigi Capogrosso and Michele Magno
Earth observation (EO) missions traditionally rely on transmitting raw or minimally processed imagery from satellites to ground stations for computationally intensive analysis. This paradigm is infeasible for CubeSat systems due to stringent constraints on the onboard embedded processors, energy availability, and communication bandwidth. To overcome these limitations, the paper presents a TinyML-based Convolutional Neural Networks (ConvNets) model optimization and deployment pipeline for onboard image classification, enabling accurate, energy-efficient, and hardware-aware inference under CubeSat-class constraints. Our pipeline integrates structured iterative
pruning, post-training INT8 quantization, and hardware-aware operator mapping to compress models and align them with the heterogeneous compute architecture of the STM32N6 microcontroller from STMicroelectronics. This Microcontroller Unit (MCU) integrates a novel Arm Cortex-M55 core and a Neural-ART Neural Processing Unit (NPU), providing a realistic proxy for CubeSat onboard computers. The paper evaluates the proposed approach on three EO
benchmark datasets (i.e., EuroSAT, RS_C11, MEDIC) and four models (i.e., SqueezeNet, MobileNetV3, EfficientNet, MCUNetV1). We demonstrate an average reduction in RAM usage of 89.55% and Flash
memory of 70.09% for the optimized models, significantly decreasing downlink bandwidth requirements while maintaining task-acceptable accuracy (with a drop ranging from 0.4 to 8.6 percentage points compared to the Float32 baseline). The energy consumption per inference ranges from 0.68 mJ to 6.45 mJ, with latency spanning from 3.22 ms to 30.38 ms. These results fully satisfy the stringent energy budgets and real-time constraints required for efficient onboard EO processing.
Read the full abstract ▾
Computer Vision
EgoForge: Goal-Directed Egocentric World Simulator
Yifan Shen, Jiateng Liu, Xinzhuo Li et al. (+9 more)
Generative world models have shown promise for simulating dynamic environments, yet
egocentric video remains challenging due to rapid viewpoint changes, frequent hand-object interactions, and goal-directed procedures whose
evolution depends on latent human intent. Existing approaches either focus on hand-centric instructional synthesis with limited scene
evolution, perform static view translation without modeling action dynamics, or rely on dense supervision, such as camera trajectories, long
video prefixes, and synchronized multi-camera capture. In this work, we introduce EgoForge, an
egocentric goal-directed world simulator that generates coherent, first-person
video rollouts from minimal static inputs: a single
egocentric image, a high-level instruction, and an optional auxiliary exocentric view. To improve intent
alignment and temporal coherence, we introduce GRAFT, a trajectory-level
diffusion refinement method that uses positive and negative rollout distributions, derived from goal, temporal, scene-consistency, and perceptual rewards, to steer the
diffusion velocity field toward coherent, goal-complete
egocentric simulations. Extensive experiments show EgoForge achieves consistent gains in semantic
alignment,
geometric stability, and motion fidelity over strong baselines, and performs robustly in real-world smart-glasses experiments.
Read the full abstract ▾
All authors Yifan Shen, Jiateng Liu, Xinzhuo Li, Yuanzhe Liu, Bingxuan Li, Houze Yang, Wenqi Jia, Yijiang Li, Tianjiao Yu, James Matthew Rehg, Xu Cao, Ismini Lourentzou
Computer Vision · Cross-listed from cs.LG
Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD
Emiel Hoogeboom, David Ruhe, Jonathan Heek et al. (+2 more)
It is currently difficult to distill discrete
diffusion models. In contrast, continuous
diffusion literature has many
distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching
Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete
distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image
datasets. Moreover, the newly distilled generators can outperform their teachers.
Read the full abstract ▾
All authors Emiel Hoogeboom, David Ruhe, Jonathan Heek, Thomas Mensink, Tim Salimans
Computer Vision
Can Large Multimodal Models Inspect Buildings? A Hierarchical Benchmark for Structural Pathology Reasoning
Hui Zhong, Yichun Gao, Luyan Liu et al. (+4 more)
Automated building facade inspection is a critical component of urban resilience and smart city maintenance. Traditionally, this field has relied on specialized discriminative models (e.g., YOLO, Mask R-CNN) that excel at pixel-level localization but are constrained to passive perception and worse generization without the visual understandng to interpret structural
topology. Large
Multimodal Models (LMMs) promise a paradigm shift toward active
reasoning, yet their application in such high-stakes engineering domains lacks rigorous evaluation standards. To bridge this gap, we introduce a human-in-the-loop semi-automated annotation framework, leveraging expert-proposal verification to unify 12 fragmented
datasets into a standardized, hierarchical ontology. Building on this foundation, we present DefectBench, the first multi-dimensional
benchmark designed to interrogate LMMs beyond basic semantic recognition. DefectBench evaluates 18 state-of-the-art (SOTA) LMMs across three escalating cognitive dimensions: Semantic Perception, Spatial Localization, and Generative Geometry
Segmentation. Extensive experiments reveal that while current LMMs demonstrate exceptional topological awareness and semantic understanding (effectively diagnosing "what" and "how"), they exhibit significant deficiencies in metric localization precision ("where"). Crucially, however, we validate the viability of zero-shot generative
segmentation, showing that general-purpose foundation models can rival specialized supervised networks without domain-specific training. This work provides both a rigorous benchmarking standard and a high-quality open-source database, establishing a new baseline for the advancement of autonomous AI
agents in civil engineering.
Read the full abstract ▾
All authors Hui Zhong, Yichun Gao, Luyan Liu, Hai Yang, Wang Wang, Haowei Zhang, Xinhu Zheng
Computer Vision
Synergistic Perception and Generative Recomposition: A Multi-Agent Orchestration for Expert-Level Building Inspection
Hui Zhong, Yichun Gao, Luyan Liu et al. (+4 more)
Building facade defect inspection is fundamental to structural health monitoring and sustainable urban maintenance, yet it remains a formidable challenge due to extreme
geometric variability, low contrast against complex backgrounds, and the inherent complexity of composite defects (e.g., cracks co-occurring with spalling). Such characteristics lead to severe pixel imbalance and feature ambiguity, which, coupled with the critical scarcity of high-quality pixel-level annotations, hinder the generalization of existing detection and
segmentation models. To address gaps, we propose FacadeFixer, a unified multi-
agent framework that treats defect perception as a collaborative
reasoning task rather than isolated recognition. Specifically,FacadeFixer orchestrates specialized
agents for detection and
segmentation to handle multi-type defect interference, working in tandem with a generative
agent to enable semantic recomposition. This process decouples intricate defects from noisy backgrounds and realistically synthesizes them onto diverse clean textures, generating high-fidelity augmented data with precise expert-level masks. To support this, we introduce a comprehensive multi-task
dataset covering six primary facade categories with pixel-level annotations. Extensive experiments demonstrate that FacadeFixer significantly outperforms state-of-the-art (SOTA) baselines. Specifically, it excels in capturing pixel-level structural anomalies and highlights generative synthesis as a robust solution to data scarcity in infrastructure inspection. Our code and
dataset will be made publicly available.
Read the full abstract ▾
All authors Hui Zhong, Yichun Gao, Luyan Liu, Xusen Guo, Zhaonian Kuang, Qiming Zhang, Xinhu Zheng
Computer Vision
Generalizable NGP-SR: Generalizable Neural Radiance Fields Super-Resolution via Neural Graph Primitives
Wanqi Yuan, Omkar Sharad Mayekar, Connor Pennington et al. (+1 more)
Neural Radiance Fields (NeRF) achieve photorealistic novel view synthesis but become costly when high-resolution (HR) rendering is required, as HR outputs demand dense sampling and higher-capacity models. Moreover, naively super-resolving per-view renderings in 2D often breaks multi-view consistency. We propose Generalizable NGP-SR, a 3D-aware super-resolution framework that reconstructs an HR radiance field directly from low-resolution (LR) posed images. Built on Neural Graphics Primitives (NGP), NGP-SR conditions radiance prediction on 3D coordinates and learned local texture
tokens, enabling recovery of high-frequency details within the radiance field and producing view-consistent HR novel views without external HR references or post-hoc 2D upsampling. Importantly, our model is generalizable: once trained, it can be applied to unseen scenes and rendered from novel viewpoints without per-scene optimization. Experiments on multiple
datasets show that NGP-SR consistently improves both reconstruction quality and runtime efficiency over prior NeRF-based super-resolution methods, offering a practical solution for scalable high-resolution novel view synthesis.
Read the full abstract ▾
All authors Wanqi Yuan, Omkar Sharad Mayekar, Connor Pennington, Nianyi Li
Computer Vision
Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning
Jiajie Li, Chenhui Xu, Meihuan Liu et al. (+1 more)
Conventional fine-tuning on domain-specific
datasets can inadvertently alter a model's pretrained
multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation framework designed to integrate domain knowledge while maintaining the model's inherent
reasoning and perceptual capabilities. CoA introduces a structured
reasoning format that enhances domain
alignment without sacrificing general
multimodal competence by reinforcement learning. Experiments on standard
surgical benchmarks, under both in-distribution and out-of-distribution settings, demonstrate that CoA achieves higher accuracy, stronger generalization, and more stable behavior than supervised fine-tuning. Furthermore, ablation studies confirm that CoA effectively preserves the model's core visual-
language abilities, providing a reliable pathway for domain specialization in VLMs.
Read the full abstract ▾
All authors Jiajie Li, Chenhui Xu, Meihuan Liu, Jinjun Xiong
Computer Vision
Preference-Guided Debiasing for No-Reference Enhancement Image Quality Assessment
Shiqi Gao, Kang Fu, Zitong Xu et al. (+4 more)
Current no-reference image quality assessment (NR-IQA) models for enhanced images often struggle to generalize, as they tend to overfit to the distinct patterns of specific enhancement
algorithms rather than evaluating genuine perceptual quality. To address this issue, we propose a preference-guided debiasing framework for no-reference enhancement image quality assessment (EIQA). Specifically, we first learn a continuous enhancement-preference
embedding space using supervised contrastive learning, where images generated by similar enhancement styles are encouraged to have closer representations. Based on this, we further estimate the enhancement-induced nuisance component contained in the raw quality representation and remove it before quality regression. In this way, the model is guided to focus on
algorithm-invariant perceptual quality cues instead of enhancement-specific visual fingerprints. To facilitate stable optimization, we adopt a two-stage training strategy that first learns the enhancement-preference space and then performs debiased quality prediction. Extensive experiments on public EIQA
benchmarks demonstrate that the proposed method effectively mitigates
algorithm-induced representation bias and achieves superior robustness and cross-
algorithm generalization compared with existing approaches.
Read the full abstract ▾
All authors Shiqi Gao, Kang Fu, Zitong Xu, Huiyu Duan, Xiongkuo Min, Jia Wang, Guangtao Zhai
Computer Vision
A Unified Platform and Quality Assurance Framework for 3D Ultrasound Reconstruction with Robotic, Optical, and Electromagnetic Tracking
Lewis Howell, Manisha Waterston, Tze Min Wah et al. (+2 more)
Three-dimensional (3D) Ultrasound (US) can facilitate diagnosis, treatment planning, and image-guided therapy. However, current studies rarely provide a comprehensive evaluation of volumetric accuracy and reproducibility, highlighting the need for robust Quality Assurance (QA) frameworks, particularly for tracked 3D US reconstruction using freehand or robotic acquisition. This study presents a QA framework for 3D US reconstruction and a flexible open source platform for tracked US research. A custom phantom containing
geometric inclusions with varying
symmetry properties enables straightforward evaluation of optical, electromagnetic, and robotic kinematic tracking for 3D US at different scanning speeds and insonation angles. A standardised pipeline performs real-time
segmentation and 3D reconstruction of
geometric targets (DSC = 0.97, FPS = 46) without GPU acceleration, followed by automated registration and comparison with ground-truth geometries. Applying this framework showed that our robotic 3D US achieves state-of-the-art reconstruction performance (DSC-3D = 0.94 +- 0.01, HD95 = 1.17 +- 0.12), approaching the spatial resolution limit imposed by the transducer. This work establishes a flexible experimental platform and a reproducible validation methodology for 3D US reconstruction. The proposed framework enables robust cross-platform comparisons and improved reporting practices, supporting the safe and effective clinical translation of 3D ultrasound in diagnostic and image-guided therapy applications.
Read the full abstract ▾
All authors Lewis Howell, Manisha Waterston, Tze Min Wah, James H. Chandler, James R. McLaughlan
Computer Vision
MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models
Puskal Khadka and KC Santosh
State Space Models (SSMs), especially recent Mamba architecture, have achieved remarkable success in sequence modeling tasks. However, extending SSMs to computer
vision remains challenging due to the non-sequential structure of visual data and its complex 2D spatial dependencies. Although several early studies have explored adapting selective SSMs for
vision applications, most approaches primarily depend on employing various traversal strategies over the same input. This introduces redundancy and distorts the intricate spatial relationships within images. To address these challenges, we propose MFil-Mamba, a novel visual state space architecture built on a multi-filter scanning backbone. Unlike fixed multi-directional traversal methods, our design enables each scan to capture unique and contextually relevant spatial information while minimizing redundancy. Furthermore, we incorporate an adaptive weighting mechanism to effectively fuse outputs from multiple scans in addition to architectural enhancements. MFil-Mamba achieves superior performance over existing state-of-the-art models across various
benchmarks that include image classification, object detection, instance
segmentation, and semantic
segmentation. For example, our tiny variant attains 83.2% top-1 accuracy on ImageNet-1K, 47.3% box AP and 42.7% mask AP on MS COCO, and 48.5% mIoU on the ADE20K
dataset. Code and models are available at https://github.com/puskal-khadka/MFil-Mamba.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Investigating a Policy-Based Formulation for Endoscopic Camera Pose Recovery
Jan Emily Mangulabnan, Akshat Chauhan, Laura Fleig et al. (+7 more)
In endoscopic surgery, surgeons continuously locate the endoscopic view relative to the anatomy by interpreting the evolving visual appearance of the intraoperative scene in the context of their prior knowledge.
Vision-based
navigation systems seek to replicate this capability by recovering camera pose directly from endoscopic
video, but most approaches do not embody the same principles of
reasoning about new frames that makes surgeons successful. Instead, they remain grounded in feature matching and
geometric optimization over keyframes, an approach that has been shown to degrade under the challenging conditions of endoscopic imaging like low texture and rapid illumination changes. Here, we pursue an alternative approach and investigate a policy-based formulation of endoscopic camera pose recovery that seeks to imitate experts in estimating trajectories conditioned on the previous camera state. Our approach directly predicts short-horizon relative motions without maintaining an explicit
geometric representation at inference time. It thus addresses, by design, some of the notorious challenges of geometry-based approaches, such as brittle correspondence matching, instability in texture-sparse regions, and limited pose coverage due to reconstruction failure. We evaluate the proposed formulation on cadaveric sinus endoscopy. Under oracle state conditioning, we compare short-horizon motion prediction quality to
geometric baselines achieving lowest mean translation error and competitive rotational accuracy. We analyze robustness by grouping prediction windows according to texture richness and illumination change indicating reduced sensitivity to low-texture conditions. These findings suggest that a learned motion policy offers a viable alternative formulation for endoscopic camera pose recovery.
Read the full abstract ▾
All authors Jan Emily Mangulabnan, Akshat Chauhan, Laura Fleig, Lalithkumar Seenivasan, Roger D. Soberanis-Mukul, S. Swaroop Vedula, Russell H. Taylor, Masaru Ishii, Gregory D. Hager, Mathias Unberath
Computer Vision · Cross-listed from quant-ph
Layered Quantum Architecture Search for 3D Point Cloud Classification
Natacha Kuete Meli, Jovita Lukasik, Vladislav Golyanik et al. (+1 more)
We introduce layered
Quantum Architecture Search (layered-QAS), a strategy inspired by classical network morphism that designs Parametrised
Quantum Circuit (PQC) architectures by progressively growing and adapting them. PQCs offer strong expressiveness with relatively few parameters, yet they lack standard architectural layers (e.g., convolution,
attention) that encode inductive biases for a given learning task. To assess the effectiveness of our method, we focus on 3D point cloud classification as a challenging yet highly structured problem. Whereas prior work on this task has used PQCs only as feature extractors for classical classifiers, our approach uses the PQC as the main building block of the classification model. Simulations show that our layered-QAS mitigates barren plateau, outperforms
quantum-adapted local and evolutionary QAS baselines, and achieves state-of-the-art results among PQC-based methods on the ModelNet
dataset.
Read the full abstract ▾
All authors Natacha Kuete Meli, Jovita Lukasik, Vladislav Golyanik, Michael Moeller
Computer Vision
Detached Skip-Links and $R$-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR
Ziye Yuan, Ruchang Yao, Chengxin Zheng et al. (+3 more)
Multimodal large
language models (MLLMs) excel at high-level
reasoning yet fail on OCR tasks where fine-grained visual details are compromised or misaligned. We identify an overlooked optimization issue in multi-layer feature fusion. Skip pathways introduce direct back-propagation paths from high-level semantic objectives to early visual layers. This mechanism overwrites low-level signals and destabilizes training. To mitigate this gradient interference, we propose Detached Skip-Links, a minimal modification that reuses shallow features in the forward pass while stopping gradients through the skip branch during joint training. This asymmetric design reduces gradient interference, improving stability and convergence without adding learnable parameters. To diagnose whether fine-grained information is preserved and usable by an LLM, we introduce $R$-Probe, which measures pixel-level reconstructability of projected visual
tokens using a shallow decoder initialized from the first quarter of the LLM layers. Across multiple ViT backbones and
multimodal benchmarks, and at scales up to 7M training samples, our approach consistently improves OCR-centric
benchmarks and delivers clear gains on general
multimodal tasks.
Read the full abstract ▾
All authors Ziye Yuan, Ruchang Yao, Chengxin Zheng, Yusheng Zhao, Daxiang Dong, Ming Zhang
Computer Vision
CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data
Tianling Liu, Hongying Liu, Fanhua Shang et al. (+3 more)
In clinical practice, crossmodal information including medical images and tabular data is essential for disease diagnosis. There exists a significant modality gap between these data types, which obstructs advancements in crossmodal diagnostic accuracy. Most existing crossmodal learning (CML) methods primarily focus on exploring relationships among high-level encoder outputs, leading to the neglect of local information in images. Additionally, these methods often overlook the extraction of task-relevant information. In this paper, we propose a novel coarse-to-fine crossmodal learning (CFCML) framework to progressively reduce the modality gap between
multimodal images and tabular data, by thoroughly exploring inter-modal relationships. At the coarse stage, we explore the relationships between multi-granularity features from various image encoder stages and tabular information, facilitating a preliminary reduction of the modality gap. At the fine stage, we generate unimodal and crossmodal prototypes that incorporate class-aware information, and establish hierarchical anchor-based relationship mining (HRM) strategy to further diminish the modality gap and extract discriminative crossmodal information. This strategy utilize modality samples, unimodal prototypes, and crossmodal prototypes as anchors to develop contrastive learning approaches, effectively enhancing inter-class disparity while reducing intra-class disparity from multiple perspectives. Experimental results indicate that our method outperforms the state-of-the-art (SOTA) methods, achieving improvements of 1.53% and 0.91% in AUC metrics on the MEN and Derm7pt
datasets, respectively. The code is available at https://github.com/IsDling/CFCML.
Read the full abstract ▾
All authors Tianling Liu, Hongying Liu, Fanhua Shang, Lequan Yu, Tong Han, Liang Wan
Computer Vision
Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features
Zheng Gao, Debin Meng, Yunqi Miao et al. (+4 more)
Current
diffusion-based makeup transfer methods commonly use the makeup information encoded by off-the-shelf foundation models (e.g., CLIP) as condition to preserve the makeup style of reference image in the generation. Although effective, these works mainly have two limitations: (1) foundation models pre-trained for generic tasks struggle to capture makeup styles; (2) the makeup features of reference image are injected to the
diffusion denoising model as a whole for global makeup transfer, overlooking the facial region-aware makeup features (i.e., eyes, mouth, etc) and limiting the regional controllability for region-specific makeup transfer. To address these, in this work, we propose Facial Region-Aware Makeup features (FRAM), which has two stages: (1) makeup CLIP fine-tuning; (2) identity and facial region-aware makeup injection. For makeup CLIP fine-tuning, unlike prior works using off-the-shelf CLIP, we synthesize annotated makeup style data using GPT-o3 and text-driven image editing model, and then use the data to train a makeup CLIP encoder through self-supervised and image-text contrastive learning. For identity and facial region-aware makeup injection, we construct before-and-after makeup image pairs from the edited images in stage 1 and then use them to learn to inject identity of source image and makeup of reference image to the
diffusion denoising model for makeup transfer. Specifically, we use learnable
tokens to query the makeup CLIP encoder to extract facial region-aware makeup features for makeup injection, which is learned via an
attention loss to enable regional control. As for identity injection, we use a ControlNet Union to encode source image and its 3D mesh simultaneously. The experimental results verify the superiority of our regional controllability and our makeup transfer performance. Code is available at https://github.com/zaczgao/Facial_Region-Aware_Makeup.
Read the full abstract ▾
All authors Zheng Gao, Debin Meng, Yunqi Miao, Zhensong Zhang, Songcen Xu, Ioannis Patras, Jifei Song
Computer Vision
NEC-Diff: Noise-Robust Event-RAW Complementary Diffusion for Seeing Motion in Extreme Darkness
Haoyue Liu, Jinghan Xu, Luxin Feng et al. (+4 more)
High-quality imaging of dynamic scenes in extremely low-light conditions is highly challenging.
Photon scarcity induces severe noise and texture loss, causing significant image degradation. Event cameras, featuring a high dynamic range (120 dB) and high sensitivity to motion, serve as powerful complements to conventional cameras by offering crucial cues for preserving subtle textures. However, most existing approaches emphasize texture recovery from events, while paying little
attention to image noise or the intrinsic noise of events themselves, which ultimately hinders accurate pixel reconstruction under
photon-starved conditions. In this work, we propose NEC-Diff, a novel
diffusion-based event-RAW hybrid imaging framework that extracts reliable information from heavily noisy signals to reconstruct fine scene structures. The framework is driven by two key insights: (1) combining the linear light-response property of RAW images with the brightness-change nature of events to establish a physics-driven constraint for robust dual-modal denoising; and (2) dynamically estimating the SNR of both modalities based on denoising results to guide adaptive feature fusion, thereby injecting reliable cues into the
diffusion process for high-fidelity visual reconstruction. Furthermore, we construct the REAL (Raw and Event Acquired in Low-light)
dataset which provides 47,800 pixel-aligned low-light RAW images, events, and high-quality references under 0.001-0.8 lux illumination. Extensive experiments demonstrate the superiority of NEC-Diff under extreme darkness. The project are available at: https://github.com/jinghan-xu/NEC-Diff.
Read the full abstract ▾
All authors Haoyue Liu, Jinghan Xu, Luxin Feng, Hanyu Zhou, Haozhi Zhao, Yi Chang, Luxin Yan
Computer Vision
Evaluating Test-Time Adaptation For Facial Expression Recognition Under Natural Cross-Dataset Distribution Shifts
John Turnbull, Shivam Grover, Amin Jalali et al. (+1 more)
Deep learning models often struggle under natural distribution shifts, a common challenge in real-world deployments. Test-Time Adaptation (TTA) addresses this by adapting models during inference without labeled source data. We present the first evaluation of TTA methods for FER under natural domain shifts, performing cross-
dataset experiments with widely used FER
datasets. This moves beyond synthetic corruptions to examine real-world shifts caused by differing collection protocols, annotation standards, and demographics. Results show TTA can boost FER performance under natural shifts by up to 11.34%.
Entropy minimization methods such as TENT and SAR perform best when the target distribution is clean. In contrast, prototype adjustment methods like T3A excel under larger distributional distance scenarios. Finally, feature
alignment methods such as SHOT deliver the largest gains when the target distribution is noisier than our source. Our cross-
dataset analysis shows that TTA effectiveness is governed by the distributional distance and the severity of the natural shift across domains.
Read the full abstract ▾
All authors John Turnbull, Shivam Grover, Amin Jalali, Ali Etemad