Computer Vision
DMAligner: Enhancing Image Alignment via Diffusion Model Based View Synthesis
Xinglong Luo, Ao Luo, Zhengning Wang et al. (+5 more)
Image
alignment is a fundamental task in computer
vision with broad applications. Existing methods predominantly employ optical flow-based image warping. However, this technique is susceptible to common challenges such as occlusions and illumination variations, leading to degraded
alignment visual quality and compromised accuracy in downstream tasks. In this paper, we present DMAligner, a
diffusion-based framework for image
alignment through
alignment-oriented view synthesis. DMAligner is crafted to tackle the challenges in image
alignment from a new perspective, employing a generation-based solution that showcases strong capabilities and avoids the problems associated with flow-based image warping. Specifically, we propose a Dynamics-aware
Diffusion Training approach for learning conditional image generation, synthesizing a novel view for image
alignment. This incorporates a Dynamics-aware Mask Producing (DMP) module to adaptively distinguish dynamic foreground regions from static backgrounds, enabling the
diffusion model to more effectively handle challenges that classical methods struggle to solve. Furthermore, we develop the Dynamic Scene Image
Alignment (DSIA)
dataset using Blender, which includes 1,033 indoor and outdoor scenes with over 30K image pairs tailored for image
alignment. Extensive experimental results demonstrate the superiority of the proposed approach on DSIA
benchmarks, as well as on a series of widely-used
video datasets for qualitative comparisons. Our code is available at https://github.com/boomluo02/DMAligner.
Read the full abstract ▾
All authors Xinglong Luo, Ao Luo, Zhengning Wang, Yueqi Yang, Chaoyu Feng, Lei Lei, Bing Zeng, Shuaicheng Liu
Computer Vision
SubspaceAD: Training-Free Few-Shot Anomaly Detection via Subspace Modeling
Camile Lendering, Erkut Akdag and Egor Bondarev
Detecting visual anomalies in industrial inspection often requires training with only a few normal images per category. Recent few-shot methods achieve strong results employing foundation-model features, but typically rely on
memory banks, auxiliary
datasets, or multi-modal tuning of
vision-
language models. We therefore question whether such complexity is necessary given the feature representations of
vision foundation models. To answer this question, we introduce SubspaceAD, a training-free method, that operates in two simple stages. First, patch-level features are extracted from a small set of normal images by a frozen DINOv2 backbone. Second, a Principal Component Analysis (PCA) model is fit to these features to estimate the low-dimensional subspace of normal variations. At inference, anomalies are detected via the reconstruction residual with respect to this subspace, producing interpretable and statistically grounded anomaly scores. Despite its simplicity, SubspaceAD achieves state-of-the-art performance across one-shot and few-shot settings without training,
prompt tuning, or
memory banks. In the one-shot anomaly detection setting, SubspaceAD achieves image-level and pixel-level AUROC of 97.1% and 97.5% on the MVTec-AD
dataset, and 93.2% and 98.2% on the VisA
dataset, respectively, surpassing prior state-of-the-art results. Code and demo are available at https://github.com/CLendering/SubspaceAD.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
Helmlab: A Two-Space Family of Analytical, Data-Driven Color Spaces for UI Design Systems
Gorkem Yildiz
We present Helmlab, a family of two purpose-built color spaces for UI design systems sharing a common 11-stage analytical structure: MetricSpace, a 72-parameter space optimized for color-difference prediction, and GenSpace, a 44-parameter space optimized for gradient and palette generation. The forward transform maps CIE XYZ to a perceptually-organized Lab representation through learned
matrices, per-channel power compression,
Fourier hue correction, and embedded Helmholtz-Kohlrausch lightness adjustment. A post-pipeline neutral correction holds gray-axis chroma below 1e-5 on a 21-step ramp, and a rigid rotation of the chromatic plane improves hue-angle
alignment without affecting the distance metric (which is invariant under isometries). On COMBVD (3,813 color pairs), MetricSpace v21 achieves STRESS 22.48, a 23 percent reduction from CIEDE2000 (29.20). On the held-out MacAdam 1974
dataset it scores 19.51 (CIEDE2000: 22.13; CAM16-UCS leads at 18.71). On a self-collected 3,552-judgement screen-condition set it scores 23.26 vs 62.54 for CIEDE2000. On academic He et al. 2022 (82 3D-printed pairs) MetricSpace scores 35.9 vs CIEDE2000 32.6, a regression we own. Averaging the three primary
datasets, MetricSpace scores 21.75 vs the next-best baseline CIECAM02-UCS at 35.98. GenSpace v0.11.1 trades distance accuracy for generation quality: on a 90-metric, 3,038-pair gradient/palette
benchmark across sRGB, P3, and Rec.2020, it wins 65 of 90 vs OKLab. The transform is invertible with round-trip errors below 1e-13. Production implementations ship on PyPI, npm, Color.js (PR 722, merged), and as a PostCSS plugin.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CE
An automatic counting algorithm for the quantification and uncertainty analysis of the number of microglial cells trainable in small and heterogeneous datasets
L. Martino, M. M. Garcia, P. S. Paradas et al. (+1 more)
Counting immunopositive
cells on biological tissues generally requires either manual annotation or (when available) automatic rough systems, for scanning signal surface and intensity in whole slide imaging. In this work, we tackle the problem of counting microglial
cells in lumbar spinal cord cross-sections of rats by omitting
cell detection and focusing only on the counting task. Manual
cell counting is, however, a time-consuming task and additionally entails extensive personnel training. The classic automatic color-based methods roughly inform about the total labeled area and intensity (
protein quantification) but do not specifically provide information on
cell number. Since the images to be analyzed have a high resolution but a huge amount of pixels contain just noise or artifacts, we first perform a pre-processing generating several filtered images {(providing a tailored, efficient feature extraction)}. Then, we design an automatic kernel counter that is a non-parametric and non-linear method. The proposed scheme can be easily trained in small
datasets since, in its basic version, it relies only on one hyper-parameter. However, being non-parametric and non-linear, the proposed
algorithm is flexible enough to express all the information contained in rich and heterogeneous
datasets as well (providing the maximum overfit if required). Furthermore, the proposed kernel counter also provides uncertainty estimation of the given prediction, and can directly tackle the case of receiving several expert opinions over the same image. Different numerical experiments with artificial and real
datasets show very promising results. Related Matlab code is also provided.
Read the full abstract ▾
All authors L. Martino, M. M. Garcia, P. S. Paradas, E. Curbelo
Computer Vision · Cross-listed from cs.AI
Certified Circuits: Stability Guarantees for Mechanistic Circuits
Alaa Anani, Tobias Lorenz, Bernt Schiele et al. (+2 more)
Understanding how neural networks arrive at their predictions is essential for debugging,
auditing, and deployment. Mechanistic interpretability pursues this goal by identifying
circuits–minimal subnetworks responsible for specific behaviors. However, existing
circuit discovery methods are brittle:
circuits depend strongly on the chosen concept
dataset and often fail to transfer out-of-distribution, raising doubts whether they capture the concept or merely
dataset-specific artifacts. We introduce Certified
Circuits, which provide provable stability guarantees for
circuit discovery. Our framework wraps any black-box discovery
algorithm with randomized data subsampling to certify that inclusion decisions over
circuit components–
neurons or edges of the model graph, depending on the base
algorithm–are invariant to bounded edit-distance perturbations of the concept
dataset. Unstable components are abstained from, yielding
circuits that are more compact and more accurate. We validate across three architectures (ResNet, ViT, GPT-2) on
vision (ImageNet and four OOD
datasets) and
language (IOI, IOI-Hard, Greater-Than) tasks. Certified
circuits achieve up to 56% higher accuracy and up to 80% fewer components, and remain reliable where baselines degrade. Certified
Circuits puts
circuit discovery on formal ground by producing mechanistic explanations that are provably stable and better aligned with the target concept. Code: https://github.com/AlaaAnani/certified-
circuits.
Read the full abstract ▾
All authors Alaa Anani, Tobias Lorenz, Bernt Schiele, Mario Fritz, Jonas Fischer
Computer Vision
UCM: Unified Modeling of Camera Control and Memory with Time-aware Positional Encoding Warping for World Models
Tianxing Xu, Zixuan Wang, Guangyuan Wang et al. (+5 more)
World models based on
video generation demonstrate remarkable potential for simulating interactive environments yet suffer from persistent difficulties in two key areas: maintaining long-term content consistency when scenes are revisited and enabling precise camera control from user-specified inputs. Existing methods based on explicit 3D reconstruction often compromise flexibility in unbounded scenarios and struggle to preserve fine-grained structures. Alternative methods rely directly on previously generated frames without establishing explicit spatial correspondence, thereby limiting controllability and consistency. To address these limitations, we present UCM, a novel framework for unified modeling of long-term
memory and precise camera control via a time-aware positional encoding warping mechanism. To reduce computational overhead, we design an efficient dual-stream
diffusion transformer for high-fidelity generation. Moreover, we introduce a scalable data curation strategy that utilizes point-cloud-based rendering to simulate scene revisiting, enabling training on over 500K
monocular videos. Extensive experiments on real-world and synthetic
benchmarks demonstrate that UCM significantly outperforms state-of-the-art methods on long-term scene consistency, while achieving precise camera controllability in high-fidelity
video generation.
Read the full abstract ▾
All authors Tianxing Xu, Zixuan Wang, Guangyuan Wang, Li Hu, Zhongyi Zhang, Peng Zhang, Bang Zhang, Songhai Zhang
Computer Vision
Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Study
Zihao Zhao, Frederik Hauke, Juliana De Castilhos et al. (+2 more)
The rapid progress of
multimodal large
language models (MLLMs) has led to increasing interest in
agent-based systems. While most prior work in medical imaging concentrates on automating routine clinical workflows, we study an underexplored yet clinically significant setting: distinguishing visually hard-to-separate diseases in a zero-shot setting. We
benchmark representative
agents on two imaging-only proxy diagnostic tasks, (1) melanoma vs. atypical nevus and (2) pulmonary edema vs. pneumonia, where visual features are highly confounded despite substantial differences in clinical management. We introduce a multi-
agent framework based on contrastive adjudication. Experimental results show improved diagnostic performance (an 11-percentage-point gain in accuracy on dermoscopy data) and reduced unsupported claims on qualitative samples, although overall performance remains insufficient for clinical deployment. We acknowledge the inherent uncertainty in human annotations and the absence of clinical context, which further limit the translation to real-world settings. Within this controlled setting, this pilot study provides preliminary insights into zero-shot
agent performance in visually confounded scenarios.
Read the full abstract ▾
All authors Zihao Zhao, Frederik Hauke, Juliana De Castilhos, Sven Nebelung, Daniel Truhn
Computer Vision
MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis
Feng Guo, Jiaxiang Liu, Yang Li et al. (+2 more)
Accurate
brain tumor diagnosis requires models to not only detect lesions but also generate clinically interpretable
reasoning grounded in imaging manifestations, yet existing public
datasets remain limited in annotation richness and diagnostic semantics. To bridge this gap, we introduce MM-NeuroOnco, a large-scale
multimodal benchmark and instruction-tuning
dataset for
brain tumor MRI understanding, consisting of 24,726 MRI slices from 20 data sources paired with approximately 200,000 semantically enriched
multimodal instructions spanning diverse tumor subtypes and imaging modalities. To mitigate the scarcity and high cost of diagnostic semantic annotations, we develop a multi-model collaborative pipeline for automated medical information completion and quality control, enabling the generation of diagnosis-related semantics beyond mask-only annotations. Building upon this
dataset, we further construct MM-NeuroOnco-Bench, a manually annotated evaluation
benchmark with a rejection-aware setting to reduce biases inherent in closed-ended question formats. Evaluation across ten representative models shows that even the strongest baseline, Gemini 3 Flash, achieves only 41.88% accuracy on diagnosis-related questions, highlighting the substantial challenges of
multimodal brain tumor diagnostic understanding. Leveraging MM-NeuroOnco, we further propose NeuroOnco-GPT, which achieves a 27% absolute accuracy improvement on diagnostic questions following fine-tuning. This result demonstrates the effectiveness of our
dataset and
benchmark in advancing clinically grounded
multimodal diagnostic
reasoning. Code and
dataset are publicly available at: https://github.com/gfnnnb/MM-NeuroOnco
Read the full abstract ▾
All authors Feng Guo, Jiaxiang Liu, Yang Li, Qianqian Shi, Mingkun Xu
Computer Vision
OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis
Junuk Cha, Jihyeon Kim and Han-Mu Park
Fingerspelling is a component of sign
languages in which words are spelled out letter by letter using specific hand poses. Automatic fingerspelling recognition plays a crucial role in bridging the communication gap between Deaf and hearing communities, yet it remains challenging due to the signing-hand ambiguity issue, the lack of appropriate training losses, and the out-of-vocabulary (OOV) problem. Prior fingerspelling recognition methods rely on explicit signing-hand detection, which often leads to recognition failures, and on a connectionist temporal classification (CTC) loss, which exhibits the peaky behavior problem. To address these issues, we develop OpenFS, an open-source approach for fingerspelling recognition and synthesis. We propose a multi-hand-capable fingerspelling recognizer that supports both single- and multi-hand inputs and performs implicit signing-hand detection by incorporating a dual-level positional encoding and a signing-hand focus (SF) loss. The SF loss encourages cross-
attention to focus on the signing hand, enabling implicit signing-hand detection during recognition. Furthermore, without relying on the CTC loss, we introduce a monotonic
alignment (MA) loss that enforces the output letter sequence to follow the temporal order of the input pose sequence through cross-
attention regularization. In addition, we propose a frame-wise letter-conditioned generator that synthesizes realistic fingerspelling pose sequences for OOV words. This generator enables the construction of a new synthetic
benchmark, called FSNeo. Through comprehensive experiments, we demonstrate that our approach achieves state-of-the-art performance in recognition and validate the effectiveness of the proposed recognizer and generator. Codes and data are available in: https://github.com/AIRC-KETI/OpenFS.
Read the full abstract ▾
Computer Vision
ToProVAR: Efficient Visual Autoregressive Modeling via Tri-Dimensional Entropy-Aware Semantic Analysis and Sparsity Optimization
Jiayu Chen, Ruoyu Lin, Zihao Zheng et al. (+4 more)
Visual Autoregressive(VAR) models enhance generation quality but face a critical efficiency bottleneck in later stages. In this paper, we present a novel optimization framework for VAR models that fundamentally differs from prior approaches such as FastVAR and SkipVAR. Instead of relying on heuristic skipping strategies, our method leverages
attention entropy to characterize the semantic projections across different dimensions of the model architecture. This enables precise identification of parameter dynamics under varying
token granularity levels, semantic scopes, and generation scales. Building on this analysis, we further uncover sparsity patterns along three critical dimensions-
token, layer, and scale-and propose a set of fine-grained optimization strategies tailored to these patterns. Extensive evaluation demonstrates that our approach achieves aggressive acceleration of the generation process while significantly preserving semantic fidelity and fine details, outperforming traditional methods in both efficiency and quality. Experiments on Infinity-2B and Infinity-8B models demonstrate that ToProVAR achieves up to 3.4x acceleration with minimal quality loss, effectively mitigating the issues found in prior work. Our code will be made publicly available.
Read the full abstract ▾
All authors Jiayu Chen, Ruoyu Lin, Zihao Zheng, Jingxin Li, Maoliang Li, Guojie Luo, Xiang Chen
Computer Vision
Velocity and stroke rate reconstruction of canoe sprint team boats based on panned and zoomed video recordings
Julian Ziegler, Daniel Matthes, Finn Gerdts et al. (+5 more)
Pacing strategies, defined by velocity and stroke rate profiles, are essential for peak performance in canoe sprint. While GPS is the gold standard for analysis, its limited availability necessitates automated
video-based solutions. This paper presents an extended framework for reconstructing performance metrics from panned and zoomed
video recordings across all sprint disciplines (K1-K4, C1-C2) and distances (200m-500m). Our method utilizes YOLOv8 for buoy and athlete detection, leveraging the known buoy grid to estimate homographies. We generalized the estimation of the boat position by means of learning a boat-specific athlete offset using a U-net based boat tip calibration. Further, we implement a robust tracking scheme using optical flow to adapt to multi-athlete boat types. Finally, we introduce methods to extract stroke rate information from either pose estimations or the athlete bounding boxes themselves. Evaluation against GPS data from elite competitions yields a velocity MAPE of 0.011 [0.008 0.014] (Spearman rho=0.974) and a stroke rate MAPE of 0.009 [0.006 0.013] (Spearman rho = 0.975). The methods provide coaches with highly accurate, automated feedback with minimal manual initialization work required, and without requiring sensors.
Read the full abstract ▾
All authors Julian Ziegler, Daniel Matthes, Finn Gerdts, Patrick Frenzel, Torsten Warnke, Matthias Englert, Tina Koevari, Mirco Fuchs
Computer Vision
pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation
Shentong Mo, Xufang Luo and Dongsheng Li
Parameter-efficient fine-tuning has demonstrated promising results across various visual adaptation tasks, such as classification and
segmentation. Typically,
prompt tuning techniques have harnessed knowledge from a single pre-trained model, whether from a general or a specialized medical domain. However, this approach typically overlooks the potential synergies that could arise from integrating diverse domain knowledge within the same tuning process. In this work, we propose a novel Mixture-of-Experts
prompt tuning method called pMoE, which leverages the strengths of multiple expert domains through expert-specialized
prompt tokens and the learnable dispatcher, effectively combining their expertise in a unified model framework. Our pMoE introduces expert-specific
prompt tokens and utilizes a dynamic
token dispatching mechanism at various
prompt layers to optimize the contribution of each domain expert during the adaptation phase. By incorporating both domain knowledge from diverse experts, the proposed pMoE significantly enhances the model's versatility and applicability to a broad spectrum of tasks. We conduct extensive experiments across 47 adaptation tasks, including both classification and
segmentation in general and medical domains. The results demonstrate that our pMoE not only achieves superior performance with a large margin of improvements but also offers an optimal trade-off between computational efficiency and adaptation effectiveness compared to existing methods.
Read the full abstract ▾
Computer Vision
MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding
Wenhui Tan, Xiaoyi Yu, Jiaze Li et al. (+5 more)
Efficiently understanding long-form
videos remains a fundamental challenge for
multimodal large
language models (MLLMs). In this paper, we present MLLM-Sampler Joint
Evolution (MSJoE), a novel framework that jointly evolves the MLLM and a lightweight key-frame sampler for efficient long-form
video understanding. MSJoE builds upon a key assumption that only a small subset of key-frames is truly informative for answering each question to a
video. Specifically, MSJoE first reasons out several queries, which describe diverse visual perspectives relevant to the question. Then, these queries interact with a frozen CLIP model to produce a query-frame similarity
matrix. Finally, a lightweight sampler predicts key-frame sampling weights from this
matrix, selecting a compact set of informative frames, which are then fed into the MLLM for answer generation. Both the MLLM and sampler are jointly optimized through reinforcement learning, enabling co-adaptation of query-
reasoning, frame-sampling, and key-frame understanding. A new long-
video QA
dataset containing 2.8K
videos with 7K question-answer pairs is collected to support the training process. Extensive experiments on VideoMME, LongVideoBench, LVBench, and MLVU show that MSJoE achieves 8.0% accuracy gain upon the base MLLM, and 1.1% higher accuracy than strongest baseline method.
Read the full abstract ▾
All authors Wenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen, Jianzhong Ju, Zhenbo Luo, Ruihua Song, Jian Luan
Computer Vision
WaterVideoQA: ASV-Centric Perception and Rule-Compliant Reasoning via Multi-Modal Agents
Runwei Guan, Shaofeng Liang, Ningwei Ouyang et al. (+9 more)
While autonomous
navigation has achieved remarkable success in passive perception (e.g., object detection and
segmentation), it remains fundamentally constrained by a void in knowledge-driven, interactive environmental cognition. In the high-stakes domain of maritime
navigation, the ability to bridge the gap between raw visual perception and complex cognitive
reasoning is not merely an enhancement but a critical prerequisite for Autonomous Surface Vessels to execute safe and precise maneuvers. To this end, we present WaterVideoQA, the first large-scale, comprehensive
Video Question Answering
benchmark specifically engineered for all-waterway environments. This
benchmark encompasses 3,029
video clips across six distinct waterway categories, integrating multifaceted variables such as volatile lighting and dynamic weather to rigorously stress-test ASV capabilities across a five-tier hierarchical cognitive framework. Furthermore, we introduce NaviMind, a pioneering multi-
agent neuro-symbolic system designed for open-ended maritime
reasoning. By synergizing Adaptive Semantic Routing, Situation-Aware Hierarchical
Reasoning, and Autonomous Self-Reflective Verification, NaviMind transitions ASVs from superficial pattern matching to regulation-compliant, interpretable decision-making. Experimental results demonstrate that our framework significantly transcends existing baselines, establishing a new paradigm for intelligent, trustworthy interaction in dynamic maritime environments.
Read the full abstract ▾
All authors Runwei Guan, Shaofeng Liang, Ningwei Ouyang, Weichen Fei, Shanliang Yao, Wei Dai, Chenhao Ge, Penglei Sun, Xiaohui Zhu, Tao Huang, Ryan Wen Liu, Hui Xiong
Computer Vision
OSDaR-AR: Enhancing Railway Perception Datasets via Multi-modal Augmented Reality
Federico Nesti, Gianluca D'Amico, Mauro Marinoni et al. (+1 more)
Although deep learning has significantly advanced the perception capabilities of intelligent transportation systems, railway applications continue to suffer from a scarcity of high-quality, annotated data for safety-critical tasks like obstacle detection. While photorealistic simulators offer a solution, they often struggle with the “sim-to-real" gap; conversely, simple image-masking techniques lack the spatio-temporal coherence required to obtain augmented single- and multi-frame scenes with the correct appearance and dimensions. This paper introduces a multi-modal augmented reality framework designed to bridge this gap by integrating photorealistic virtual objects into real-world railway sequences from the OSDaR23
dataset. Utilizing Unreal Engine 5 features, our pipeline leverages
LiDAR point-clouds and INS/GNSS data to ensure accurate object placement and temporal stability across RGB frames. This paper also proposes a
segmentation-based refinement strategy for INS/GNSS data to significantly improve the realism of the augmented sequences, as confirmed by the comparative study presented in the paper. Carefully designed augmented sequences are collected to produce OSDaR-AR, a public
dataset designed to support the development of next-generation railway perception systems. The
dataset is available at the following page: https://syndra.retis.santannapisa.it/osdarar.html
Read the full abstract ▾
All authors Federico Nesti, Gianluca D'Amico, Mauro Marinoni, Giorgio Buttazzo
Computer Vision
Chain of Flow: ECG-Conditioned 4D Cardiac Cine Generation from Patient-Specific Anatomical Anchor
Haofan Wu, Nay Aung, Theodoros N. Arvanitis et al. (+3 more)
Cardiac cine
magnetic resonance imaging (MRI) is central to functional cardiac assessment, yet a full current cine sequence may not always be directly available at the point of analysis. We introduce Chain of Flow (COF), an electrocardiography (ECG)-conditioned framework that combines patient-specific MRI and current ECG for subject-specific 4D cardiac cine generation. On the UK Biobank
dataset, COF achieves strong image-level fidelity and downstream function-oriented performance on a shared same-visit evaluable
benchmark. Multi-slice and multi-resolution analyses indicate stable structural generation quality across the short-axis stack and heterogeneous acquisition resolutions. Controlled phase-robustness analyses across resampled input MRI phases further provide same-visit proxy support for patient-specific MRI plus current ECG when a target MRI phase is not directly observed. A cross-visit route provides exploratory serial evidence, with the clearest gains in current-facing region-of-interest readout. Disease-category functional audits, case-level volume-trajectory evidence review further delineate where the current patient-specific MRI plus ECG formulation remains stable for anatomy-aware downstream cardiac analysis. Code is available at https://anonymous.4open.science/r/COF-paper-release-C88B.
Read the full abstract ▾
All authors Haofan Wu, Nay Aung, Theodoros N. Arvanitis, Joao A. C. Lima, Steffen E. Petersen, Le Zhang
Computer Vision
Towards Multimodal Domain Generalization with Few Labels
Hongzhao Li, Hao Dong, Hualei Wan et al. (+3 more)
Multimodal models ideally should generalize to unseen domains while remaining data-efficient to reduce annotation costs. To this end, we introduce and study a new problem, Semi-Supervised
Multimodal Domain Generalization (SSMDG), which aims to learn robust
multimodal models from multi-source data with few labeled samples. We observe that existing approaches fail to address this setting effectively:
multimodal domain generalization methods cannot exploit unlabeled data, semi-supervised
multimodal learning methods ignore domain shifts, and semi-supervised domain generalization methods are confined to single-modality inputs. To overcome these limitations, we propose a unified framework featuring three key components: Consensus-Driven Consistency Regularization, which obtains reliable pseudo-labels through confident fused-unimodal consensus; Disagreement-Aware Regularization, which effectively utilizes ambiguous non-consensus samples; and Cross-Modal Prototype
Alignment, which enforces domain- and modality-invariant representations while promoting robustness under missing modalities via cross-modal translation. We further establish the first SSMDG
benchmarks, on which our method consistently outperforms strong baselines in both standard and missing-modality scenarios. Our
benchmarks and code are available at https://github.com/lihongzhao99/SSMDG.
Read the full abstract ▾
All authors Hongzhao Li, Hao Dong, Hualei Wan, Shupan Li, Mingliang Xu, Muhammad Haris Khan
Computer Vision · Cross-listed from cs.AI
OmniGAIA: Towards Native Omni-Modal AI Agents
Xiaoxi Li, Wenxiang Jiao, Jiarui Jin et al. (+10 more)
Human intelligence naturally intertwines omni-modal perception – spanning
vision, audio, and
language – with complex
reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g.,
vision-
language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive
benchmark designed to evaluate omni-modal
agents on tasks necessitating deep
reasoning and multi-turn tool execution across
video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal
reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation
agent under tool-integrated
reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.
Read the full abstract ▾
All authors Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Haoxuan Li, Hao Wang, Shijian Wang, Guanting Dong, Jiajie Jin, Yinuo Wang, Yuan Lu, Ji-Rong Wen, Zhicheng Dou, Zhouchen Lin
Computer Vision
SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction
Qinfeng Zhu, Yunxi Jiang and Lei Fan
Panoramic dense-prediction models, spanning semantic
segmentation and depth estimation, are typically trained under a strict
gravity-aligned assumption. Real-world captures, however, routinely violate it: handheld devices jitter and aerial platforms change attitude, so the camera is rarely upright. Under such 3D reorientation, standard spherical
Transformers overfit global latitude cues and collapse. We introduce SO3UFormer, an architecture that learns intrinsic spherical features largely decoupled from the underlying coordinate frame, through three
geometric components: (1) removing absolute latitude encoding, which breaks the dependence on the
gravity axis; (2) quadrature-consistent spherical
attention, which corrects for non-uniform sampling density; and (3) a gauge-aware relative positional bias built from local tangent-plane angles rather than global axes. A logit-space SO(3)-consistency regularizer, used only during training, further suppresses residual discretization effects. To
benchmark robustness, we introduce Pose35, a variant of Stanford2D3D perturbed by random rotations within $\pm 35^\circ$, and evaluate under a full, arbitrary SO(3) stress test. There, the baseline SphereUFormer collapses from 67.53 mIoU on Pose35 to 25.26 under the full SO(3) test, whereas SO3UFormer reaches 72.03 on Pose35 and retains 70.67 under the same test. Similarly, on a second real-world
dataset (Matterport3D) for
segmentation and on panoramic depth estimation, SO3UFormer remains essentially rotation-invariant while the
gravity-anchored baseline again loses most of its accuracy. Code and models are available at https://github.com/zhuqinfeng1999/SO3UFormer.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
GraspLDP: Towards Generalizable Grasping Policy via Latent Diffusion
Enda Xiang, Haoxiang Ma, Xinzhu Ma et al. (+2 more)
This paper focuses on enhancing the grasping precision and generalization of manipulation policies learned via imitation learning.
Diffusion-based policy learning methods have recently become the mainstream approach for robotic manipulation tasks. As grasping is a critical subtask in manipulation, the ability of imitation-learned policies to execute precise and generalizable grasps merits particular
attention. Existing imitation learning techniques for grasping often suffer from imprecise grasp executions, limited spatial generalization, and poor object generalization. To address these challenges, we incorporate grasp prior knowledge into the
diffusion policy framework. In particular, we employ a latent
diffusion policy to guide action chunk decoding with grasp pose prior, ensuring that generated motion trajectories adhere closely to feasible grasp configurations. Furthermore, we introduce a self-supervised reconstruction objective during
diffusion to embed the graspness prior: at each reverse
diffusion step, we reconstruct wrist-camera images back-projected the graspness from the intermediate representations. Both simulation and real
robot experiments demonstrate that our approach significantly outperforms baseline methods and exhibits strong dynamic grasping capabilities.
Read the full abstract ▾
All authors Enda Xiang, Haoxiang Ma, Xinzhu Ma, Zicheng Liu, Di Huang
Computer Vision
From Blind Spots to Gains: Diagnostic-Driven Iterative Training for Large Multimodal Models
Hongrui Jia, Chaoya Jiang, Yongrui Heng et al. (+2 more)
As Large
Multimodal Models (LMMs) scale up and reinforcement learning (RL) methods mature, LMMs have made notable progress in complex
reasoning and decision making. Yet training still relies on static data and fixed recipes, making it difficult to diagnose capability blind spots or provide dynamic, targeted reinforcement. Motivated by findings that test driven error exposure and feedback based correction outperform repetitive practice, we propose Diagnostic-driven Progressive
Evolution (DPE), a spiral loop where diagnosis steers data generation and reinforcement, and each iteration re-diagnoses the updated model to drive the next round of targeted improvement. DPE has two key components. First, multiple
agents annotate and quality control massive unlabeled
multimodal data, using tools such as web search and image editing to produce diverse, realistic samples. Second, DPE attributes failures to specific weaknesses, dynamically adjusts the data mixture, and guides
agents to generate weakness focused data for targeted reinforcement. Experiments on Qwen3-VL-8B-Instruct and Qwen2.5-VL-7B-Instruct show stable, continual gains across eleven
benchmarks, indicating DPE as a scalable paradigm for continual LMM training under open task distributions. Our code, models, and data are publicly available at https://github.com/hongruijia/DPE.
Read the full abstract ▾
All authors Hongrui Jia, Chaoya Jiang, Yongrui Heng, Shikun Zhang, Wei Ye
Computer Vision
A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling
Chong Wang, Yabin Zhang, Yunhe Gao et al. (+9 more)
Foundation models for medical imaging are typically pretrained on increasingly large
datasets, following a "scale-at-all-costs" paradigm. However, this strategy faces two critical challenges: large-scale medical
datasets often contain substantial redundancy and severe class imbalance that bias representation learning toward over-represented patterns, and indiscriminate training regardless of heterogeneity in data quality incurs considerable computational inefficiency. Here we demonstrate that active, principled data curation during pretraining can serve as a viable, cost-effective alternative to brute-force
dataset enlargement. We introduce CheXficient, a chest X-ray (CXR) foundation model that selectively prioritizes informative training samples. CheXficient is pretrained on only 22.7% of 1,235,004 paired CXR images and reports while consuming under 27.3% of the total compute budget, yet achieving comparable or superior performance to its full-data counterpart and other large-scale pretrained models. We assess CheXficient across 20 individual
benchmarks spanning 5 task types, including non-adapted off-the-shelf evaluations (zero-shot findings classification and crossmodal
retrieval) and adapted downstream tasks (disease prediction, semantic
segmentation, and radiology report generation). Further analyses show that CheXficient systematically prioritizes under-represented training samples, improving generalizability on long-tailed or rare conditions. Overall, our work offers practical insights into the data and computation demands for efficient pretraining and downstream adaptation of medical
vision-
language foundation models.
Read the full abstract ▾
All authors Chong Wang, Yabin Zhang, Yunhe Gao, Maya Varma, Clemence Mottez, Faidra Patsatzi, Jiaming Liu, Jin Long, Jean-Benoit Delbrouck, Sergios Gatidis, Akshay S. Chaudhari, Curtis P. Langlotz
Computer Vision · Cross-listed from cs.LG
Direction-Flipped Influence Audits Reveal Hidden Structure in Moral Choices of LLMs
Phil Blandfort, Tushar Karayil, Alex McKenzie et al. (+3 more)
Moral
benchmarks for LLMs typically score models on context-free
prompts, implicitly treating the measured choice rate as stable. We test this assumption with a direction-flipped influence audit: for each scenario, we compare a baseline
prompt with matched cues steering toward option A or option B. Across a trolley-problem-style moral triage task, BBQ, and DailyDilemmas, and across five LLM families with and without
reasoning, short contextual cues shift per-condition choice rates by 12-18 percentage points on average. These shifts reveal structure that baseline scores miss: roughly 40% of baseline-neutral triage and BBQ conditions exhibit directional asymmetry under influence, and a meaningful share of significant effects backfire, moving opposite the cue's intended direction. In follow-up probes, models often recognize the cue while denying that it affected their choice. Among significant backfire trials, this stated-vs.-revealed inconsistency appears in 78% of cases.
Reasoning does not eliminate contextual sensitivity but reshapes it: social-pressure cues such as user preference and emotional appeal weaken across
benchmarks, while few-shot demonstrations strengthen sharply on both triage and BBQ. We recommend direction-flipped influence pairs as a standard complement to context-free moral-bias evaluation, and release the harness and data to make such audits routine.
Read the full abstract ▾
All authors Phil Blandfort, Tushar Karayil, Alex McKenzie, Urja Pawar, Robert Graham, Dmitrii Krasheninnikov
Computer Vision
Reflectance Multispectral Imaging for Soil Composition Estimation and USDA Texture Classification
G.A.S.L Ranasinghe, J.A.S.T. Jayakody, M.C.L. De Silva et al. (+5 more)
Soil texture is a foundational attribute that governs water availability and erosion in agriculture, as well as load bearing capacity, deformation response, and shrink-swell risk in geotechnical engineering. Yet texture is still typically determined by slow and labour intensive laboratory particle size tests, while many sensing alternatives are either costly or too coarse to support routine field scale deployment. This paper proposes a robust and field deployable multispectral imaging (MSI) system and machine learning framework for predicting soil composition and the United States Department of Agriculture (USDA) texture classes. The proposed system uses a cost effective in-house MSI device operating from 365 nm to 940 nm to capture thirteen spectral bands, which effectively capture the spectral properties of soil texture. Regression models use the captured spectral properties to estimate clay, silt, and sand percentages, while a direct classifier predicts one of the twelve USDA textural classes. Indirect classification is obtained by mapping the regressed compositions to texture classes via the USDA soil texture triangle. The framework is evaluated on mixture data by mixing clay, silt, and sand in varying proportions, using the USDA classification triangle as a basis. Experimental results show that the proposed approach achieves a coefficient of determination R^2 up to 0.99 for composition prediction and over 99% accuracy for texture classification. These findings indicate that MSI combined with data-driven modeling can provide accurate, non-destructive, and field deployable soil texture characterization suitable for geotechnical screening and precision agriculture.
Read the full abstract ▾
All authors G.A.S.L Ranasinghe, J.A.S.T. Jayakody, M.C.L. De Silva, G. Thilakarathne, G.M.R.I. Godaliyadda, H.M.V.R. Herath, M.P.B. Ekanayake, S.K. Navaratnarajah
Computer Vision
CMSA-Net: Causal Multi-scale Aggregation with Adaptive Multi-source Reference for Video Polyp Segmentation
Tong Wang, Yaolei Qi, Siwen Wang et al. (+3 more)
Video polyp
segmentation (VPS) is an important task in computer-aided colonoscopy, as it helps doctors accurately locate and track polyps during examinations. However, VPS remains challenging because polyps often look similar to surrounding mucosa, leading to weak semantic discrimination. In addition, large changes in polyp position and scale across
video frames make stable and accurate
segmentation difficult. To address these challenges, we propose a robust VPS framework named CMSA-Net. The proposed network introduces a Causal Multi-scale Aggregation (CMA) module to effectively gather semantic information from multiple historical frames at different scales. By using causal
attention, CMA ensures that temporal feature propagation follows strict time order, which helps reduce noise and improve feature reliability. Furthermore, we design a Dynamic Multi-source Reference (DMR) strategy that adaptively selects informative and reliable reference frames based on semantic separability and prediction confidence. This strategy provides strong multi-frame guidance while keeping the model efficient for real-time inference. Extensive experiments on the SUN-SEG
dataset demonstrate that CMSA-Net achieves state-of-the-art performance, offering a favorable balance between
segmentation accuracy and real-time clinical applicability.
Read the full abstract ▾
All authors Tong Wang, Yaolei Qi, Siwen Wang, Imran Razzak, Guanyu Yang, Yutong Xie
Computer Vision
Face Time Traveller : Travel Through Ages Without Losing Identity
Purbayan Kar, Ayush Ghadiya, Vishal Chudasama et al. (+2 more)
Face aging, an ill-posed problem shaped by environmental and genetic factors, is vital in entertainment, forensics, and digital archiving, where realistic age transformations must preserve both identity and visual realism. However, existing works relying on numerical age representations overlook the interplay of biological and contextual cues. Despite progress in recent face aging models, they struggle with identity preservation in wide age transformations, also static
attention and optimization-heavy inversion in
diffusion limit adaptability, fine-grained control and background consistency. To address these challenges, we propose Face Time Traveller (FaceTT), a
diffusion-based framework that achieves high-fidelity, identity-consistent age transformation. Here, we introduce a Face-Attribute-Aware
Prompt Refinement strategy that encodes intrinsic (biological) and extrinsic (environmental) aging cues for context-aware conditioning. A tuning-free Angular Inversion method is proposed that efficiently maps real faces into the
diffusion latent space for fast and accurate reconstruction. Moreover, an Adaptive
Attention Control mechanism is introduced that dynamically balances cross-
attention for semantic aging cues and self-
attention for structural and identity preservation. Extensive experiments on
benchmark datasets and in-the-wild testset demonstrate that FaceTT achieves superior identity retention, background preservation and aging realism over state-of-the-art (SOTA) methods.
Read the full abstract ▾
All authors Purbayan Kar, Ayush Ghadiya, Vishal Chudasama, Pankaj Wasnik, C.V. Jawahar
Computer Vision
PhotoAgent: Exploratory Visual Aesthetic Planning with Large Vision Models
Mingde Yao, Zhiyuan You, King-Man Tam et al. (+2 more)
With the recent fast development of generative models, instruction-based image editing has shown great potential in generating high-quality images. However, the quality of editing highly depends on carefully designed instructions, placing the burden of task decomposition and sequencing entirely on the user. To achieve autonomous image editing, we present PhotoAgent, a system that advances image editing through explicit aesthetic planning. Specifically, PhotoAgent formulates autonomous image editing as a long-horizon decision-making problem. It reasons over user aesthetic intent, plans multi-step editing actions via tree search, and iteratively refines results through closed-loop execution with
memory and visual feedback, without requiring step-by-step user
prompts. To support reliable evaluation in real-world scenarios, we introduce UGC-Edit, an aesthetic evaluation
benchmark consisting of 7,000 photos and a learned aesthetic reward model. We also construct a test set containing 1,017 photos to systematically assess autonomous photo editing performance. Extensive experiments demonstrate that PhotoAgent consistently improves both instruction adherence and visual quality compared with baseline methods. The project page is https://mdyao.github.io/PhotoAgent/.
Read the full abstract ▾
All authors Mingde Yao, Zhiyuan You, King-Man Tam, Menglu Wang, Tianfan Xue
Computer Vision
GSTurb: Gaussian Splatting for Atmospheric Turbulence Mitigation
Hanliang Du, Zhangji Lu, Zewei Cai et al. (+3 more)
Atmospheric
turbulence causes significant image degradation due to pixel displacement (tilt) and blur, particularly in long-range imaging applications. In this paper, we propose a novel framework for atmospheric
turbulence mitigation, GSTurb, which integrates optical flow-guided tilt correction and Gaussian splatting for modeling non-isoplanatic blur. The framework employs Gaussian parameters to represent tilt and blur, and optimizes them across multiple frames to enhance restoration. Experimental results on the ATSyn-static
dataset demonstrate the effectiveness of our method, achieving a peak PSNR of 27.67 dB and SSIM of 0.8735. Compared to the state-of-the-art method, GSTurb improves PSNR by 1.3 dB (a 4.5% increase) and SSIM by 0.048 (a 5.8% increase). Additionally, on real
datasets, including the TSRWGAN Real-World and CLEAR
datasets, GSTurb outperforms existing methods, showing significant improvements in both qualitative and quantitative performance. These results highlight that combining optical flow-guided tilt correction with Gaussian splatting effectively enhances image restoration under both synthetic and real-world
turbulence conditions. The code for this method will be available at https://github.com/DuhlLiamz/3DGS_
turbulence/tree/main.
Read the full abstract ▾
All authors Hanliang Du, Zhangji Lu, Zewei Cai, Qijian Tang, Qifeng Yu, Xiaoli Liu
Computer Vision
Robust Human Trajectory Prediction via Self-Supervised Skeleton Representation Learning
Taishu Arashima, Hiroshi Kera and Kazuhiko Kawamoto
Human trajectory prediction plays a crucial role in applications such as autonomous
navigation and
video surveillance. While recent works have explored the integration of human skeleton sequences to complement trajectory information, skeleton data in real-world environments often suffer from missing joints caused by occlusions. These disturbances significantly degrade prediction accuracy, indicating the need for more robust skeleton representations. We propose a robust trajectory prediction method that incorporates a self-supervised skeleton representation model pretrained with masked autoencoding. Experimental results in occlusion-prone scenarios show that our method improves robustness to missing skeletal data without sacrificing prediction accuracy, and consistently outperforms baseline models in clean-to-moderate missingness regimes.
Read the full abstract ▾
Computer Vision
SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation
Ling Wang, Hao-Xiang Guo, Xinzhou Wang et al. (+9 more)
We introduce SceneTransporter, an end-to-end framework for structured 3D scene generation from a single image. While existing methods generate part-level 3D objects, they often fail to organize these parts into distinct instances in open-world scenes. Through a debiased clustering probe, we reveal a critical insight: this failure stems from the lack of structural constraints within the model's internal assignment mechanism. Based on this finding, we reframe the task of structured 3D scene generation as a global correlation assignment problem. To solve this, SceneTransporter formulates and solves an entropic Optimal Transport (OT) objective within the denoising loop of the compositional DiT model. This formulation imposes two powerful structural constraints. First, the resulting transport plan gates cross-
attention to enforce an exclusive, one-to-one routing of image patches to part-level 3D latents, preventing
entanglement. Second, the competitive nature of the transport encourages the grouping of similar patches, a process that is further regularized by an edge-based cost, to form coherent objects and prevent fragmentation. Extensive experiments show that SceneTransporter outperforms existing methods on open-world scene generation, significantly improving instance-level coherence and
geometric fidelity. Code and models will be publicly available at https://2019epwl.github.io/SceneTransporter/.
Read the full abstract ▾
All authors Ling Wang, Hao-Xiang Guo, Xinzhou Wang, Fuchun Sun, Kai Sun, Pengkun Liu, Hang Xiao, Zhong Wang, Guangyuan Fu, Eric Li, Yang Liu, Yikai Wang