Computer Vision
COG: Confidence-aware Optimal Geometric Correspondence for Unsupervised Single-reference Novel Object Pose Estimation
Yuchen Che, Jingtu Wu, Hao Zheng et al. (+1 more)
Estimating the 6DoF pose of a novel object with a single reference view is challenging due to occlusions, view-point changes, and outliers. A core difficulty lies in finding robust cross-view correspondences, as existing methods often rely on discrete one-to-one matching that is non-differentiable and tends to collapse onto sparse key-points. We propose Confidence-aware Optimal
Geometric Correspondence (COG), an unsupervised framework that formulates correspondence estimation as a confidence-aware optimal transport problem. COG produces balanced soft correspondences by predicting point-wise confidences and injecting them as optimal transport marginals, suppressing non-overlapping regions. Semantic priors from
vision foundation models further regularize the correspondences, leading to stable pose estimation. This design integrates confidence into the correspondence finding and pose estimation pipeline, enabling unsupervised learning. Experiments show unsupervised COG achieves comparable performance to supervised methods, and supervised COG outperforms them.
Read the full abstract ▾
All authors Yuchen Che, Jingtu Wu, Hao Zheng, Asako Kanezaki
Computer Vision
ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models
Riccardo de Lutio, Tobias Fischer, Yen-Yu Chang et al. (+7 more)
Per-scene optimization methods such as 3D Gaussian Splatting provide state-of-the-art novel view synthesis quality but extrapolate poorly to under-observed areas. Methods that leverage generative priors to correct artifacts in these areas hold promise but currently suffer from two shortcomings. The first is scalability, as existing methods use image
diffusion models or bidirectional
video models that are limited in the number of views they can generate in a single pass (and thus require a costly iterative
distillation process for consistency). The second is quality itself, as generators used in prior work tend to produce outputs that are inconsistent with existing scene content and fail entirely in completely unobserved regions. To solve these, we propose a two-stage pipeline that leverages two key insights. First, we train a powerful bidirectional generative model with a novel opacity mixing strategy that encourages consistency with existing observations while retaining the model's ability to extrapolate novel content in unseen areas. Second, we distill it into a causal auto-regressive model that generates hundreds of frames in a single pass. This model can directly produce novel views or serve as pseudo-supervision to improve the underlying 3D representation in a simple and highly efficient manner. We evaluate our method extensively and demonstrate that it can generate plausible reconstructions in scenarios where existing approaches fail completely. When measured on commonly benchmarked
datasets, we outperform all existing baselines by a wide margin, exceeding prior state-of-the-art methods by 1-3 dB PSNR.
Read the full abstract ▾
All authors Riccardo de Lutio, Tobias Fischer, Yen-Yu Chang, Yuxuan Zhang, Jay Zhangjie Wu, Xuanchi Ren, Tianchang Shen, Katarina Tothova, Zan Gojcic, Haithem Turki
Computer Vision
Random Wins All: Rethinking Grouping Strategies for Vision Tokens
Qihang Fan, Yuang Ai, Huaibo Huang et al. (+1 more)
Since
Transformers are introduced into
vision architectures, their quadratic complexity has always been a significant issue that many research efforts aim to address. A representative approach involves grouping
tokens, performing self-
attention calculations within each group, or pooling the
tokens within each group into a single
token. To this end, various carefully designed grouping strategies have been proposed to enhance the performance of
Vision Transformers. Here, we pose the following questions: Are these carefully designed grouping methods truly necessary? Is there a simpler and more unified
token grouping method that can replace these diverse methods? Therefore, we propose the random grouping strategy, which involves a simple and fast random grouping strategy for
vision tokens. We validate this approach on multiple baselines, and experiments show that random grouping almost outperforms all other grouping methods. When transferred to downstream tasks, such as object detection, random grouping demonstrates even more pronounced advantages. In response to this phenomenon, we conduct a detailed analysis of the advantages of random grouping from multiple perspectives and identify several crucial elements for the design of grouping strategies: positional information, head feature diversity, global receptive field, and fixed grouping pattern. We demonstrate that as long as these four conditions are met,
vision tokens require only an extremely simple grouping strategy to efficiently and effectively handle various visual tasks. We also validate the effectiveness of our proposed random method across multiple modalities, including visual tasks, point cloud processing, and
vision-
language models. Code will be available at https://github.com/qhfan/random.
Read the full abstract ▾
All authors Qihang Fan, Yuang Ai, Huaibo Huang, Ran He
Computer Vision
RAISE: Requirement-Adaptive Evolutionary Refinement for Training-Free Text-to-Image Alignment
Liyao Jiang, Ruichen Chen, Chao Gao et al. (+1 more)
Recent text-to-image (T2I)
diffusion models achieve remarkable realism, yet faithful
prompt-image
alignment remains challenging, particularly for complex
prompts with multiple objects, relations, and fine-grained attributes. Existing training-free inference-time scaling methods rely on fixed iteration budgets that cannot adapt to
prompt difficulty, while reflection-tuned models require carefully curated reflection
datasets and extensive joint fine-tuning of
diffusion and
vision-
language models, often overfitting to reflection paths data and lacking transferability across models. We introduce RAISE (Requirement-Adaptive Self-Improving
Evolution), a training-free, requirement-driven evolutionary framework for adaptive T2I generation. RAISE formulates image generation as a requirement-driven adaptive scaling process, evolving a population of candidates at inference time through a diverse set of refinement actions-including
prompt rewriting, noise resampling, and instructional editing. Each generation is verified against a structured checklist of requirements, enabling the system to dynamically identify unsatisfied items and allocate further computation only where needed. This achieves adaptive test-time scaling that aligns computational effort with semantic query complexity. On GenEval and DrawBench, RAISE attains state-of-the-art
alignment (0.94 overall GenEval) while incurring fewer generated samples (reduced by 30-40%) and VLM calls (reduced by 80%) than prior scaling and reflection-tuned baselines, demonstrating efficient, generalizable, and model-agnostic multi-round self-improvement. Code is available at https://github.com/LiyaoJiang1998/RAISE.
Read the full abstract ▾
All authors Liyao Jiang, Ruichen Chen, Chao Gao, Di Niu
Computer Vision
Feibo Jiang, Siwei Tu, Li Dong et al. (+5 more)
Visual-
Language Models (VLMs), with their strong capabilities in image and text understanding, offer a solid foundation for intelligent communications. However, their effectiveness is constrained by limited
token granularity, overlong visual
token sequences, and inadequate cross-modal
alignment. To overcome these challenges, we propose TaiChi, a novel VLM framework designed for
token communications. TaiChi adopts a dual-visual tokenizer architecture that processes both high- and low-resolution images to collaboratively capture pixel-level details and global conceptual features. A Bilateral
Attention Network (BAN) is introduced to intelligently fuse multi-scale visual
tokens, thereby enhancing visual understanding and producing compact visual
tokens. In addition, a Kolmogorov Arnold Network (KAN)-based modality projector with learnable activation functions is employed to achieve precise nonlinear
alignment from visual features to the text semantic space, thus minimizing information loss. Finally, TaiChi is integrated into a
multimodal and multitask
token communication system equipped with a joint VLM-channel coding scheme. Experimental results validate the superior performance of TaiChi, as well as the feasibility and effectiveness of the TaiChi-driven
token communication system.
Read the full abstract ▾
All authors Feibo Jiang, Siwei Tu, Li Dong, Xiaolong Li, Kezhi Wang, Cunhua Pan, Zhu Han, Jiangzhou Wang
Computer Vision · Cross-listed from cs.LG
Analyzing Physical Adversarial Example Threats to Machine Learning in Election Systems
Khaleque Md Aashiq Kamal, Surya Eada, Aayushi Verma et al. (+4 more)
Developments in the machine learning voting domain have shown both promising results and risks. Trained models perform well on ballot classification tasks (> 99% accuracy) but are at risk from
adversarial example attacks that cause misclassifications. In this paper, we analyze an attacker who seeks to deploy
adversarial examples against machine learning ballot classifiers to compromise a U.S. election. We first derive a probabilistic framework for determining the number of
adversarial example ballots that must be printed to flip an election, in terms of the probability of each candidate winning and the total number of ballots cast. Second, it is an open question as to which type of
adversarial example is most effective when physically printed in the voting domain. We analyze six different types of
adversarial example attacks: l_infinity-APGD, l2-APGD, l1-APGD, l0 PGD, l0 + l_infinity PGD, and l0 + sigma-map PGD. Our experiments include physical realizations of 144,000
adversarial examples through printing and scanning with four different machine learning models. We empirically demonstrate an analysis gap between the physical and digital domains, wherein attacks most effective in the digital domain (l2 and l_infinity) differ from those most effective in the physical domain (l1 and l2, depending on the model). By unifying a probabilistic election framework with digital and physical
adversarial example evaluations, we move beyond prior close race analyses to explicitly quantify when and how
adversarial ballot manipulation could alter outcomes.
Read the full abstract ▾
All authors Khaleque Md Aashiq Kamal, Surya Eada, Aayushi Verma, Subek Acharya, Adrian Yemin, Benjamin Fuller, Kaleel Mahmood
Computer Vision
U-VLM: Hierarchical Vision Language Modeling for Report Generation
Pengcheng Shi, Minghui Zhang, Kehan Song et al. (+3 more)
Automated radiology report generation is key for reducing radiologist workload and improving diagnostic consistency, yet generating accurate reports for 3D medical imaging remains challenging. Existing
vision-
language models face two limitations: they do not leverage
segmentation-pretrained encoders, and they inject visual features only at the input layer of
language models, losing multi-scale information. We propose U-VLM, which enables hierarchical
vision-
language modeling in both training and architecture: (1) progressive training from
segmentation to classification to report generation, and (2) multi-layer visual injection that routes U-Net encoder features to corresponding
language model layers. Each training stage can leverage different
datasets without unified annotations. U-VLM achieves state-of-the-art performance on CT-RATE (F1: 0.414 vs 0.258, BLEU-mean: 0.349 vs 0.305) and AbdomenAtlas 3.0 (F1: 0.624 vs 0.518 for
segmentation-based detection) using only a 0.1B decoder trained from scratch, demonstrating that well-designed
vision encoder pretraining outweighs the benefits of 7B+ pre-trained
language models. Ablation studies show that progressive pretraining significantly improves F1, while multi-layer injection improves BLEU-mean. Code is available at https://github.com/yinghemedical/U-VLM.
Read the full abstract ▾
All authors Pengcheng Shi, Minghui Zhang, Kehan Song, Jiaqi Liu, Yun Gu, Xinglin Zhang
Computer Vision · Cross-listed from cs.LG
Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols
Xu Luo, Ji Zhang, Lianli Gao et al. (+2 more)
Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation
algorithms.However, there lacks a unified, rigorous evaluation protocol that is both challenging and realistic for real-world usage. In this work, we establish FEWTRANS, a comprehensive
benchmark containing 10 diverse
datasets, and propose the Hyperparameter Ensemble (HPE) protocol to overcome the "validation set illusion" in data-scarce regimes. Our empirical findings demonstrate that the choice of pre-trained model is the dominant factor for performance, while many sophisticated transfer methods offer negligible practical advantages over a simple full-parameter fine-tuning baseline. To explain this surprising effectiveness, we provide an in-depth mechanistic analysis showing that full fine-tuning succeeds via distributed micro-adjustments and more flexible reshaping of high-level semantic presentations without suffering from overfitting. Additionally, we quantify the performance collapse of
multimodal models in specialized domains as a result of linguistic rarity using adjusted Zipf frequency scores. By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research. We make the FEWTRANS
benchmark publicly available at https://github.com/Frankluox/FewTrans.
Read the full abstract ▾
All authors Xu Luo, Ji Zhang, Lianli Gao, Heng Tao Shen, Jingkuan Song
Computer Vision
High Dynamic Range Imaging Based on an Asymmetric Event-SVE Camera System
Pengju Sun, Banglei Guan, Jing Tao et al. (+4 more)
High dynamic range (HDR) imaging under extreme illumination remains challenging for conventional cameras due to overexposure. Event cameras provide microsecond temporal resolution and high dynamic range, while spatially varying exposure (SVE) sensors offer single-shot radiometric diversity.We present a hardware–
algorithm co-designed HDR imaging system that tightly integrates an SVE micro-attenuation camera with an event sensor in an asymmetric dual-modality configuration. To handle non-coaxial geometry and heterogeneous optics, we develop a two-stage cross-modal
alignment framework that combines feature-guided coarse homography estimation with a multi-scale refinement module based on spatial pooling and frequency-domain filtering. On top of aligned representations, we develop a cross-modal HDR reconstruction network with convolutional fusion, mutual-information regularization, and a learnable fusion loss that adaptively balances intensity cues and event-derived structural constraints. Comprehensive experiments on both synthetic
benchmarks and real captures demonstrate that the proposed system consistently improves highlight recovery, edge fidelity, and robustness compared with frame-only or event-only HDR pipelines. The results indicate that jointly optimizing optical design, cross-modal
alignment, and computational fusion provides an effective foundation for reliable HDR perception in highly dynamic and radiometrically challenging environments.
Read the full abstract ▾
All authors Pengju Sun, Banglei Guan, Jing Tao, Zhenbao Yu, Xuanyu Bai, Yang Shang, Qifeng Yu
Computer Vision
DreamWorld: Unified World Modeling in Video Generation
Boming Tan, Xiangdong Zhang, Ning Liao et al. (+7 more)
Despite impressive progress in
video generation, existing models remain limited to surface-level plausibility, lacking a coherent and unified understanding of the world. Prior approaches typically incorporate only a single form of world-related knowledge or rely on rigid
alignment strategies to introduce additional knowledge. However, aligning the single world knowledge is insufficient to constitute a world model that requires jointly modeling multiple heterogeneous dimensions (e.g., physical commonsense, 3D and temporal consistency). To address this limitation, we introduce DreamWorld, a unified framework that integrates complementary world knowledge into
video generators via a Joint World Modeling Paradigm, jointly predicting
video pixels and features from foundation models to capture temporal dynamics, spatial geometry, and semantic consistency. However, naively optimizing these heterogeneous objectives can lead to visual instability and temporal flickering. To mitigate this issue, we propose Consistent Constraint Annealing (CCA) to progressively regulate world-level constraints during training, and Multi-Source Inner-Guidance to enforce learned world priors at inference. Extensive evaluations show that DreamWorld improves world consistency, outperforming Wan2.1 by 2.26 points on VBench. Code will be made publicly available at \href{https://github.com/ABU121111/DreamWorld}{\textcolor{mypink}{Github}}.
Read the full abstract ▾
All authors Boming Tan, Xiangdong Zhang, Ning Liao, Jingtao Zhang, Yuqing Zhang, Xiaoqiu Zhong, Xiaosong Jia, Xue Yang, Yanyong Zhang, Shaofeng Zhang
Computer Vision
OPGAgent: An Agent for Auditable Dental Panoramic X-ray Interpretation
Zhaolin Yu, Litao Yang, Ben Babicka et al. (+7 more)
Orthopantomograms (OPGs) are the standard panoramic radiograph in dentistry, used for full-arch screening across multiple diagnostic tasks. While
Vision Language Models (VLMs) now allow multi-task OPG analysis through natural
language, they underperform task-specific models on most individual tasks. Agentic systems that orchestrate specialized tools offer a path to both versatility and accuracy, this approach remains unexplored in the field of dental imaging. To address this gap, we propose OPGAgent, a multi-tool agentic system for auditable OPG interpretation. OPGAgent coordinates specialized perception modules with a consensus mechanism through three components: (1) a Hierarchical Evidence Gathering module that decomposes OPG analysis into global, quadrant, and tooth-level phases with dynamically invoking tools, (2) a Specialized Toolbox encapsulating spatial, detection, utility, and expert zoos, and (3) a Consensus Subagent that resolves conflicts through anatomical constraints. We further propose OPG-Bench, a structured-report protocol based on (Location, Field, Value) triples derived from real clinical reports, which enables a comprehensive review of findings and
hallucinations, extending beyond the limitations of VQA indicators. On our OPG-Bench and the public MMOral-OPG
benchmark, OPGAgent outperforms current dental VLMs and medical
agent frameworks across both structured-report and VQA evaluation. Code will be released upon acceptance.
Read the full abstract ▾
All authors Zhaolin Yu, Litao Yang, Ben Babicka, Ming Hu, Jing Hao, Anthony Huang, James Huang, Yueming Jin, Jiasong Wu, Zongyuan Ge
Computer Vision
ReMoT: Reinforcement Learning with Motion Contrast Triplets
Cong Wan, Zeyu Guo, Jiangyang Li et al. (+5 more)
We present ReMoT, a unified training paradigm to systematically address the fundamental shortcomings of VLMs in spatio-temporal consistency – a critical failure point in
navigation, robotics, and autonomous driving. ReMoT integrates two core components: (1) A rule-based automatic framework that generates ReMoT-16K, a large-scale (16.5K triplets) motion-contrast
dataset derived from
video meta-annotations, surpassing costly manual or model-based generation. (2) Group Relative Policy Optimization, which we empirically validate yields optimal performance and data efficiency for learning this contrastive
reasoning, far exceeding standard Supervised Fine-Tuning. We also construct the first
benchmark for fine-grained motion contrast triplets to measure a VLM's discrimination of subtle motion attributes (e.g., opposing directions). The resulting model achieves state-of-the-art performance on our new
benchmark and multiple standard VLM
benchmarks, culminating in a remarkable 25.1% performance leap on spatio-temporal
reasoning tasks.
Read the full abstract ▾
All authors Cong Wan, Zeyu Guo, Jiangyang Li, SongLin Dong, Yifan Bai, Lin Peng, Zhiheng Ma, Yihong Gong
Computer Vision
Explainable Continuous-Time Mask Refinement with Local Self-Similarity Priors for Medical Image Segmentation
Rajdeep Chatterjee, Sudip Chakrabarty and Trishaani Acharjee
Accurate semantic
segmentation of foot ulcers is essential for automated wound monitoring, yet boundary delineation remains challenging due to tissue heterogeneity and poor contrast with surrounding skin. To overcome the limitations of standard intensity-based networks, we present LSS-LTCNet:an ante-hoc explainable framework synergizing deterministic structural priors with continuous-time neural dynamics. Our architecture departs from traditional black-box models by employing a Local Self-Similarity (LSS) mechanism that extracts dense, illumination-invariant texture descriptors to explicitly disentangle necrotic tissue from background artifacts. To enforce topological precision, we introduce a
Liquid Time-Constant (LTC) refinement module that treats boundary
evolution as an ODEgoverned dynamic system, iteratively refining masks over continuous time-steps. Comprehensive evaluation on the MICCAI FUSeg
dataset demonstrates that LSS-LTCNet achieves state-of-the-art boundary
alignment, securing a peak Dice score of 86.96% and an exceptional 95th percentile Hausdorff Distance (HD95) of 8.91 pixels. Requiring merely 25.70M parameters, the model significantly outperforms heavier U-Net and
transformer baselines in efficiency. By providing inherent visual audit trails alongside high-fidelity predictions, LSS-LTCNet offers a robust and transparent solution for computer-aided diagnosis in mobile healthcare (mHealth) settings.
Read the full abstract ▾
Computer Vision
Improved Adversarial Diffusion Compression for Real-World Video Super-Resolution
Bin Chen, Weiqi Li, Shijie Zhao et al. (+4 more)
While many
diffusion models have achieved impressive results in real-world
video super-resolution (Real-VSR) by generating rich and realistic details, their reliance on multi-step sampling leads to slow inference. One-step networks like SeedVR2, DOVE, and DLoRAL alleviate this through condensing generation into one single step, yet they remain heavy, with billions of parameters and multi-second latency. Recent
adversarial diffusion compression (ADC) offers a promising path via
pruning and distilling these models into a compact AdcSR network, but directly applying it to Real-VSR fails to balance spatial details and temporal consistency due to its lack of temporal awareness and the limitations of standard
adversarial learning. To address these challenges, we propose an improved ADC method for Real-VSR. Our approach distills a large
diffusion Transformer (DiT) teacher DOVE equipped with 3D spatio-temporal attentions, into a pruned 2D Stable
Diffusion (SD)-based AdcSR backbone, augmented with lightweight 1D temporal convolutions, achieving significantly higher efficiency. In addition, we introduce a dual-head
adversarial distillation scheme, in which discriminators in both pixel and feature domains explicitly disentangle the discrimination of details and consistency into two heads, enabling both objectives to be effectively optimized without sacrificing one for the other. Experiments demonstrate that the resulting compressed AdcVSR model reduces complexity by 95% in parameters and achieves an 8$\times$ acceleration over its DiT teacher DOVE, while maintaining competitive
video quality and efficiency.
Read the full abstract ▾
All authors Bin Chen, Weiqi Li, Shijie Zhao, Xuanyu Zhang, Junlin Li, Li Zhang, Jian Zhang
Computer Vision
SesaHand: Enhancing 3D Hand Reconstruction via Controllable Generation with Semantic and Structural Alignment
Zhuoran Zhao, Xianghao Kong, Linlin Yang et al. (+3 more)
Recent studies on 3D hand reconstruction have demonstrated the effectiveness of synthetic training data to improve estimation performance. However, most methods rely on game engines to synthesize hand images, which often lack diversity in textures and environments, and fail to include crucial components like arms or interacting objects. Generative models are promising alternatives to generate diverse hand images, but still suffer from misalignment issues. In this paper, we present SesaHand, which enhances controllable hand image generation from both semantic and structural
alignment perspectives for 3D hand reconstruction. Specifically, for semantic
alignment, we propose a pipeline with Chain-of-Thought inference to extract human behavior semantics from image captions generated by the
Vision-
Language Model. This semantics suppresses human-irrelevant environmental details and ensures sufficient human-centric contexts for hand image generation. For structural
alignment, we introduce hierarchical structural fusion to integrate structural information with different granularity for feature refinement to better align the hand and the overall human body in generated images. We further propose a hand structure
attention enhancement method to efficiently enhance the model's
attention on hand regions. Experiments demonstrate that our method not only outperforms prior work in generation performance but also improves 3D hand reconstruction with the generated hand images.
Read the full abstract ▾
All authors Zhuoran Zhao, Xianghao Kong, Linlin Yang, Zheng Wei, Pan Hui, Anyi Rao
Computer Vision
Mamba-CAD: State Space Model For 3D Computer-Aided Design Generative Modeling
Xueyang Li, Yunzhong Lou, Yu Song et al. (+1 more)
Computer-Aided Design (CAD) generative modeling has a strong and long-term application in the industry. Recently, the parametric CAD sequence as the design logic of an object has been widely mined by sequence models. However, the industrial CAD models, especially in component objects, are fine-grained and complex, requiring a longer parametric CAD sequence to define. To address the problem, we introduce Mamba-CAD, a self-supervised generative modeling for complex CAD models in the industry, which can model on a longer parametric CAD sequence. Specifically, we first design an encoder-decoder framework based on a Mamba architecture and pair it with a CAD reconstruction task for pre-training to model the latent representation of CAD models; and then we utilize the learned representation to guide a generative
adversarial network to produce the fake representation of CAD models, which would be finally recovered into parametric CAD sequences via the decoder of MambaCAD. To train Mamba-CAD, we further create a new
dataset consisting of 77,078 CAD models with longer parametric CAD sequences. Comprehensive experiments are conducted to demonstrate the effectiveness of our model under various evaluation metrics, especially in the generation length of valid parametric CAD sequences. The code and
dataset can be achieved from https://github.com/Sunny-Hack/Code-for-Mamba-CAD-AAAI-2025-.
Read the full abstract ▾
All authors Xueyang Li, Yunzhong Lou, Yu Song, Xiangdong Zhou
Computer Vision
Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
April Fu
Although Large
Vision-
Language Models (LVLMs) have made substantial progress,
hallucination, where generated text is not grounded in the visual input, remains a challenge. As LVLMs become stronger, previously reported
hallucination patterns, such as linguistic bias and overthinking phenomenon, become far less consistent, making the corresponding mitigation techniques substantially less effective. In this paper, we introduce an Internal self-Correction mechanism utilizing Layer
Attention (ICLA) that operates directly on hidden states during generation. Each layer selectively retrieves information from all preceding layers through a diagonal cross-layer
attention mechanism, enabling self-refinement without any external correction signals. With introducing and training only 0.2M and 0.1M additional parameters on LLaVA1.5-7B and Qwen2.5-VL-7B, \ours consistently improves visual grounding across multiple
hallucination benchmarks, demonstrating its effectiveness for more advanced LVLMs.
Read the full abstract ▾
Computer Vision
TAP-SLF: Parameter-Efficient Adaptation of Vision Foundation Models for Multi-Task Ultrasound Image Analysis
Hui Wan and Libin Lan
Executing multiple tasks simultaneously in medical image analysis, including
segmentation, classification, detection, and regression, often introduces significant challenges regarding model generalizability and the optimization of shared feature representations. While
Vision Foundation Models (VFMs) provide powerful general representations, full fine-tuning on limited medical data is prone to overfitting and incurs high computational costs. Moreover, existing parameter-efficient fine-tuning approaches typically adopt task-agnostic adaptation protocols, overlooking both task-specific mechanisms and the varying sensitivity of model layers during fine-tuning. In this work, we propose Task-Aware Prompting and Selective Layer Fine-Tuning (TAP-SLF), a unified framework for multi-task ultrasound image analysis. TAP-SLF incorporates task-aware soft
prompts to encode task-specific priors into the input
token sequence and applies LoRA to selected specific top layers of the encoder. This strategy updates only a small fraction of the VFM parameters while keeping the pre-trained backbone frozen. By combining task-aware
prompts with selective high-layer fine-tuning, TAP-SLF enables efficient VFM adaptation to diverse medical tasks within a shared backbone. Results on the FMC_UIA 2026 Challenge test set, where TAP-SLF wins fifth place, combined with evaluations on the officially released training
dataset using an 8:2 train-test split, demonstrate that task-aware prompting and selective layer tuning are effective strategies for efficient VFM adaptation.
Read the full abstract ▾
Computer Vision
Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models
Hulingxiao He, Zhi Tan and Yuxin Peng
A high-performing, general-purpose visual understanding model should map visual inputs to a taxonomic tree of labels, identify novel categories beyond the training set for which few or no publicly available images exist. Large
Multimodal Models (LMMs) have achieved remarkable progress in fine-grained visual recognition (FGVR) for known categories. However, they remain limited in hierarchical visual recognition (HVR) that aims at predicting consistent label paths from coarse to fine categories, especially for novel categories. To tackle these challenges, we propose Taxonomy-Aware Representation
Alignment (TARA), a simple yet effective strategy to inject taxonomic knowledge into LMMs. TARA leverages representations from biology foundation models (BFMs) that encode rich biological relationships through hierarchical contrastive learning. By aligning the intermediate representations of visual features with those of BFMs, LMMs are encouraged to extract discriminative visual cues well structured in the taxonomy tree. Additionally, we align the representations of the first answer
token with the ground-truth label, flexibly bridging the gap between contextualized visual features and categories of varying granularity according to user intent. Experiments demonstrate that TARA consistently enhances LMMs' hierarchical consistency and leaf node accuracy, enabling reliable recognition of both known and novel categories within complex biological taxonomies. Code is available at https://github.com/PKU-ICST-MIPL/TARA_CVPR2026.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
LLM-Bootstrapped Targeted Finding Guidance for Factual MLLM-based Medical Report Generation
Cunyuan Yang, Dejuan Song, Xiaotao Pang et al. (+7 more)
The automatic generation of medical reports utilizing
Multimodal Large
Language Models (MLLMs) frequently encounters challenges related to factual instability, which may manifest as the omission of findings or the incorporation of inaccurate information, thereby constraining their applicability in clinical settings. Current methodologies typically produce reports based directly on image features, which inherently lack a definitive factual basis. In response to this limitation, we introduce Fact-Flow, an innovative framework that separates the process of visual fact identification from the generation of reports. This is achieved by initially predicting clinical findings from the image, which subsequently directs the MLLM to produce a report that is factually precise. A pivotal advancement of our approach is a pipeline that leverages a Large
Language Model (LLM) to autonomously create a
dataset of labeled medical findings, effectively eliminating the need for expensive manual annotation. Extensive experimental evaluations conducted on two disease-focused medical
datasets validate the efficacy of our method, demonstrating a significant enhancement in factual accuracy compared to state-of-the-art models, while concurrently preserving high standards of text quality.
Read the full abstract ▾
All authors Cunyuan Yang, Dejuan Song, Xiaotao Pang, Qianqian Shen, Wenjie Nie, Yifan Huang, Lei Wu, Wei Han, Haishuai Wang, Jiajun Bu
Computer Vision
An Interpretable Local Editing Model for Counterfactual Medical Image Generation
Hyungi Min, Taeseung You, Hangyeul Lee et al. (+2 more)
Counterfactual medical image generation have emerged as a critical tool for enhancing AI-driven systems in medical domain by answering "what-if" questions. However, existing approaches face two fundamental limitations: First, they fail to prevent unintended modifications, resulting collateral changes in demographic attributes when only disease features should be affected. Second, they lack interpretability in their editing process, which significantly limits their utility in real-world medical applications. To address these limitations, we present InstructX2X, a novel interpretable local editing model for counterfactual medical image generation featuring Region-Specific Editing. This approach restricts modifications to specific regions, effectively preventing unintended changes while simultaneously providing a Guidance Map that offers inherently interpretable visual explanations of the editing process. Additionally, we introduce MIMIC-EDIT-INSTRUCTION, a
dataset for counterfactual medical image generation derived from expert-verified medical VQA pairs. Through extensive experiments, InstructX2X achieve state-of-the-art performance across all major evaluation metrics. Our model successfully generates high-quality counterfactual chest X-ray images along with interpretable explanations.
Read the full abstract ▾
All authors Hyungi Min, Taeseung You, Hangyeul Lee, Yeongjae Cho, Sungzoon Cho
Computer Vision
Station2Radar: query conditioned gaussian splatting for precipitation field
Doyi Kim, Minseok Seo and Changick Kim
Precipitation
forecasting relies on heterogeneous data. Weather radar is accurate, but coverage is geographically limited and costly to maintain. Weather stations provide accurate but sparse point measurements, while satellites offer dense, high-resolution coverage without direct rainfall
retrieval. To overcome these limitations, we propose Query-Conditioned Gaussian Splatting (QCGS), the first framework to fuse automatic weather station (AWS) observations with satellite imagery for generating precipitation fields. Unlike conventional 2D Gaussian splatting, which renders the entire image plane, QCGS selectively renders only queried precipitation regions, avoiding unnecessary computation in non-precipitating areas while preserving sharp precipitation structures. The framework combines a radar point proposal network that identifies rainfall-support locations with an implicit neural representation (INR) network that predicts Gaussian parameters for each point. QCGS enables efficient, resolution-flexible precipitation field generation in real time. Through extensive evaluation with
benchmark precipitation products, QCGS demonstrates over 50% improvement in RMSE compared to conventional gridded precipitation products, and consistently maintains high performance across multiple spatiotemporal scales.
Read the full abstract ▾
Computer Vision
DiffTrans: Differentiable Geometry-Materials Decomposition for Reconstructing Transparent Objects
Changpu Li, Shuang Wu, Songlin Tang et al. (+3 more)
Reconstructing transparent objects from a set of multi-view images is a challenging task due to the complicated nature and indeterminate behavior of light propagation. Typical methods are primarily tailored to specific scenarios, such as objects following a uniform
topology, exhibiting ideal transparency and surface specular reflections, or with only surface materials, which substantially constrains their practical applicability in real-world settings. In this work, we propose a differentiable rendering framework for transparent objects, dubbed DiffTrans, which allows for efficient decomposition and reconstruction of the geometry and materials of transparent objects, thereby reconstructing transparent objects accurately in intricate scenes with diverse
topology and complex texture. Specifically, we first utilize FlexiCubes with dilation and smoothness regularization as the iso-surface representation to reconstruct an initial geometry efficiently from the multi-view object silhouette. Meanwhile, we employ the environment light radiance field to recover the environment of the scene. Then we devise a recursive differentiable ray tracer to further optimize the geometry, index of refraction and absorption rate simultaneously in a unified and end-to-end manner, leading to high-quality reconstruction of transparent objects in intricate scenes. A prominent advantage of the designed ray tracer is that it can be implemented in CUDA, enabling a significantly reduced computational cost. Extensive experiments on multiple
benchmarks demonstrate the superior reconstruction performance of our DiffTrans compared with other methods, especially in intricate scenes involving transparent objects with diverse
topology and complex texture. The code is available at https://github.com/lcp29/DiffTrans.
Read the full abstract ▾
All authors Changpu Li, Shuang Wu, Songlin Tang, Guangming Lu, Jun Yu, Wenjie Pei
Computer Vision
PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models
Yuanhao Su, Shaofeng Zhang, Xiaosong Jia et al. (+1 more)
The development of 3D
Vision-
Language Models (VLMs), crucial for applications in robotics, autonomous driving, and augmented reality, is severely constrained by the scarcity of paired 3D-text data. Existing methods rely solely on next-
token prediction loss, using only
language tokens for supervision. This results in inefficient utilization of limited 3D data and leads to a significant degradation and loss of valuable
geometric information in intermediate representations. To address these limitations, we propose {\mname}, a novel feature-level
alignment regularization method. {\mname} explicitly supervises intermediate point cloud
tokens to preserve fine-grained 3D
geometric-semantic information throughout the
language modeling process. Specifically, we constrain the intermediate point cloud
tokens within the LLM to align with visual input
tokens via a consistency loss. By training only a lightweight
alignment projector and LoRA adapters, {\mname} achieves explicit feature-level supervision with minimal computational overhead, effectively preventing
geometric degradation. Extensive experiments on ModelNet40 and Objaverse
datasets demonstrate that our method achieves 2.08 pp improvement on average for classification tasks, with a substantial 7.50 pp gain on the challenging open-vocabulary Objaverse classification task and 4.88 pp improvement on 3D object captioning evaluated by Qwen2-72B-Instruct, validating the effectiveness of {\mname}. Code is publicly available at \href{https://github.com/yharoldsu0627/PointAlign}{https://github.com/yharoldsu0627/PointAlign}.
Read the full abstract ▾
All authors Yuanhao Su, Shaofeng Zhang, Xiaosong Jia, Qi Fan
Computer Vision
SSR: Pushing the Limit of Spatial Intelligence with Structured Scene Reasoning
Yi Zhang, Youya Xia, Yong Wang et al. (+7 more)
While
Multimodal Large
Language Models (MLLMs) excel in semantic tasks, they frequently lack the "spatial sense" essential for sophisticated
geometric reasoning. Current models typically suffer from exorbitant modality-
alignment costs and deficiency in fine-grained structural modeling precision.We introduce SSR, a framework designed for Structured Scene
Reasoning that seamlessly integrates 2D and 3D representations via a lightweight
alignment mechanism. To minimize training overhead, our framework anchors 3D
geometric features to the large
language model's pre-aligned 2D visual semantics through cross-modal addition and
token interleaving, effectively obviating the necessity for large-scale
alignment pre-training. To underpin complex spatial
reasoning, we propose a novel scene graph generation pipeline that represents global layouts as a chain of independent local triplets defined by relative coordinates. This is complemented by an incremental generation
algorithm, enabling the model to construct "
language-model-friendly" structural scaffolds for complex environments. Furthermore, we extend these capabilities to global-scale 3D global grounding task, achieving absolute metric precision across heterogeneous data sources. At a 7B parameter scale, SSR achieves state-of-the-art performance on multiple spatial intelligence
benchmarks, notably scoring 73.9 on VSI-Bench. Our approach significantly outperforms much larger models, demonstrating that efficient feature
alignment and structured scene
reasoning are the cornerstones of authentic spatial intelligence.
Read the full abstract ▾
All authors Yi Zhang, Youya Xia, Yong Wang, Meng Song, Xin Wu, Wenjun Wan, Bingbing Liu, AiXue Ye, Hongbo Zhang, Feng Wen