Computer Vision · Cross-listed from eess.IV
Exploring Polarimetric Properties Preservation during Reconstruction of PolSAR images using Complex-valued Convolutional Neural Networks
Quentin Gabot, Joana Frontera-Pons, Jérémy Fix et al. (+2 more)
The inherently complex-valued nature of Polarimetric SAR data necessitates using specialized
algorithms capable of directly processing complex-valued representations. However, this aspect remains underexplored in the deep learning community, with many studies opting to convert complex signals into the real domain before applying conventional real-valued models. In this work, we leverage complex-valued neural networks and investigate the performance of complex-valued Convolutional AutoEncoders. We show that these networks can effectively compress and reconstruct fully polarimetric SAR data while preserving essential physical characteristics, as demonstrated through Pauli, Krogager, and Cameron coherent decompositions, as well as the non-coherent $H-\alpha$ decomposition. Finally, we highlight the advantages of complex-valued neural networks over their real-valued counterparts. These insights pave the way for developing robust, physics-informed, complex-valued generative models for SAR data processing.
Read the full abstract ▾
All authors Quentin Gabot, Joana Frontera-Pons, Jérémy Fix, Chengfang Ren, Jean-Philippe Ovarlez
Computer Vision
MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representation
Haoming Wang, Qiyao Xue, Weichen Liu et al. (+1 more)
When
embodied AI is expanding from traditional object detection and recognition to more advanced tasks of
robot manipulation and actuation planning, visual spatial
reasoning from the
video inputs is necessary to perceive the spatial relationships of objects and guide device actions. However, existing visual
language models (VLMs) have very weak capabilities in spatial
reasoning due to the lack of knowledge about 3D spatial information, especially when the
reasoning task involve complex spatial relations across multiple
video frames. In this paper, we present a new inference-time computing technique for on-device
embodied AI, namely MosaicThinker, which enhances the on-device small VLM's spatial
reasoning capabilities on difficult cross-frame
reasoning tasks. Our basic idea is to integrate fragmented spatial information from multiple frames into a unified space representation of global semantic map, and further guide the VLM's spatial
reasoning over the semantic map via a visual
prompt. Experiment results show that our technique can greatly enhance the accuracy of cross-frame spatial
reasoning on resource-constrained
embodied AI devices, over
reasoning tasks with diverse types and complexities.
Read the full abstract ▾
All authors Haoming Wang, Qiyao Xue, Weichen Liu, Wei Gao
Computer Vision · Cross-listed from cs.MM
Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data
Thu Hang Phung, Duong M. Nguyen, Thanh Trung Huynh et al. (+3 more)
This paper introduces a generalized
federated prompt-tuning framework for practical scenarios where local
datasets are multi-modal and exhibit different distributional patterns of missing features at the input level. The proposed framework bridges the gap between
federated learning and multi-modal
prompt-tuning which have traditionally focused on either uni-modal or centralized data. A key challenge in this setting arises from the lack of semantic
alignment between
prompt instructions that encode similar distributional patterns of missing data across different clients. To address this, our framework introduces specialized client-tuning and server-aggregation designs that simultaneously optimize, align, and aggregate
prompt-tuning instructions across clients and data modalities. This allows
prompt instructions to complement one another and be combined effectively. Extensive evaluations on diverse
multimodal benchmark datasets demonstrate that our work consistently outperforms state-of-the-art (SOTA) baselines.
Read the full abstract ▾
All authors Thu Hang Phung, Duong M. Nguyen, Thanh Trung Huynh, Quoc Viet Hung Nguyen, Trong Nghia Hoang, Phi Le Nguyen
Computer Vision
MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images
Ankan Deria, Komal Kumar, Adinath Madhavrao Dukre et al. (+3 more)
Multimodal large
language models have advanced rapidly, but their adoption in medicine is constrained by limited domain coverage, imperfect modality
alignment, and insufficient grounded
reasoning. We introduce MedMO, a medical
multimodal foundation model built on a general MLLM architecture and trained exclusively on large-scale domain-specific data. MedMO uses a multi-stage training recipe that includes cross-modal pretraining to align heterogeneous visual encoders with a medical
language backbone, instruction tuning with multi-task supervision spanning captioning, VQA, report generation,
retrieval, and bounding-box disease localization, and reinforcement learning with verifiable rewards that combine factuality checks with a box-level GIoU signal to improve spatial grounding and step-by-step
reasoning in challenging clinical settings. Across modalities and tasks, MedMO surpasses strong open-source medical baselines. MedMO-8B-Next achieves consistent gains on VQA
benchmarks, improving by 6.6% on average over Fleming-VL-8B, including gains of 6.0% on MMMU-Med, 9.8% on PMC-VQA, and 21.3% on MedXpertQA. On text-based QA, it improves by 14.4% over Fleming-VL-8B, driven by gains of 8.4% on MMLU-Med and 30.1% on MedQA. For medical report generation, it improves by 6.7% on MIMIC-CXR. MedMO-8B-Next also demonstrates strong grounding performance, reaching 56.1 IoU on Bacteria, which is a 47.8 IoU gain over Fleming-VL-8B. At smaller scale, MedMO-4B-Next remains competitive and exceeds Fleming-VL-8B across VQA, QA, and report generation. Evaluations spanning radiology, ophthalmology, and pathology microscopy further confirm broad cross-modality generalization. Project is available at https://genmilab.github.io/MedMO-Page
Read the full abstract ▾
All authors Ankan Deria, Komal Kumar, Adinath Madhavrao Dukre, Eran Segal, Salman Khan, Imran Razzak
Computer Vision
CineScene: Implicit 3D as Effective Scene Representation for Cinematic Video Generation
Kaiyi Huang, Yukun Huang, Yu Li et al. (+8 more)
Cinematic
video production requires control over scene-subject composition and camera movement, but live-action shooting remains costly due to the need for constructing physical sets. To address this, we introduce the task of cinematic
video generation with decoupled scene context: given multiple images of a static environment, the goal is to synthesize high-quality
videos featuring dynamic subject while preserving the underlying scene consistency and following a user-specified camera trajectory. We present CineScene, a framework that leverages implicit 3D-aware scene representation for cinematic
video generation. Our key innovation is a novel context conditioning mechanism that injects 3D-aware features in an implicit way: By encoding scene images into visual representations through VGGT, CineScene injects spatial priors into a pretrained text-to-
video generation model by additional context concatenation, enabling camera-controlled
video synthesis with consistent scenes and dynamic subjects. To further enhance the model's robustness, we introduce a simple yet effective random-shuffling strategy for the input scene images during training. To address the lack of training data, we construct a scene-decoupled
dataset with Unreal Engine 5, containing paired
videos of scenes with and without dynamic subjects, panoramic images representing the underlying static scene, along with their camera trajectories. Experiments show that CineScene achieves state-of-the-art performance in scene-consistent cinematic
video generation, handling large camera movements and demonstrating generalization across diverse environments.
Read the full abstract ▾
All authors Kaiyi Huang, Yukun Huang, Yu Li, Jianhong Bai, Xintao Wang, Zinan Lin, Xuefei Ning, Jiwen Yu, Pengfei Wan, Yu Wang, Xihui Liu
Computer Vision · Cross-listed from cs.RO
DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
Shenyuan Gao, William Liang, Kaiyuan Zheng et al. (+27 more)
Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist
agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of
egocentric human
videos. Our data mixture represents the largest
video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled
videos. After post-training on small-scale target
robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a
distillation pipeline that accelerates DreamDojo to a real-time speed of 10.81 FPS and further improves context consistency. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD)
benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose
robot world models.
Read the full abstract ▾
All authors Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Qianli Ma, Seungjun Nah, Loic Magne, Jiannan Xiang, Yuqi Xie, Ruijie Zheng, Dantong Niu, You Liang Tan, K.R. Zentner, George Kurian, Suneel Indupuru, Pooya Jannaty, Jinwei Gu, Jun Zhang, Jitendra Malik, Pieter Abbeel, Ming-Yu Liu, Yuke Zhu, Joel Jang, Linxi "Jim" Fan
Computer Vision
Reliable Mislabel Detection for Video Capsule Endoscopy Data
Julia Werner, Julius Oexle, Oliver Bause et al. (+5 more)
The classification performance of deep neural networks relies strongly on access to large, accurately annotated
datasets. In medical imaging, however, obtaining such
datasets is particularly challenging since annotations must be provided by specialized physicians, which severely limits the pool of annotators. Furthermore, class boundaries can often be ambiguous or difficult to define which further complicates machine learning-based classification. In this paper, we want to address this problem and introduce a framework for mislabel detection in medical
datasets. This is validated on the two largest, publicly available
datasets for
Video Capsule Endoscopy, an important imaging procedure for examining the gastrointestinal tract based on a
video stream of lowresolution images. In addition, potentially mislabeled samples identified by our pipeline were reviewed and re-annotated by three experienced gastroenterologists. Our results show that the proposed framework successfully detects incorrectly labeled data and results in an improved anomaly detection performance after cleaning the
datasets compared to current baselines.
Read the full abstract ▾
All authors Julia Werner, Julius Oexle, Oliver Bause, Maxime Le Floch, Franz Brinkmann, Hannah Tolle, Jochen Hampe, Oliver Bringmann
Computer Vision
Seeing Beyond Redundancy: Task Complexity's Role in Vision Token Specialization in VLLMs
Darryl Hannan, John Cooper, Dylan White et al. (+1 more)
Vision capabilities in
vision large
language models (VLLMs) have consistently lagged behind their linguistic capabilities. In particular, numerous
benchmark studies have demonstrated that VLLMs struggle when fine-grained visual information or spatial
reasoning is required. However, we do not yet understand exactly why VLLMs struggle so much with these tasks relative to others. Some works have focused on visual redundancy as an explanation, where high-level visual information is uniformly spread across numerous
tokens and specific, fine-grained visual information is discarded. In this work, we investigate this premise in greater detail, seeking to better understand exactly how various types of visual information are processed by the model and what types of visual information are discarded. To do so, we introduce a simple synthetic
benchmark dataset that is specifically constructed to probe various visual features, along with a set of metrics for measuring visual redundancy, allowing us to better understand the nuances of their relationship. Then, we explore fine-tuning VLLMs on a number of complex visual tasks to better understand how redundancy and compression change based upon the complexity of the data that a model is trained on. We find that there is a connection between task complexity and visual compression, implying that having a sufficient ratio of high complexity visual data is crucial for altering the way that VLLMs distribute their visual representation and consequently improving their performance on complex visual tasks. We hope that this work will provide valuable insights for training the next generation of VLLMs.
Read the full abstract ▾
All authors Darryl Hannan, John Cooper, Dylan White, Yijing Watkins
Computer Vision
PANC: Prior-Aware Normalized Cut via Anchor-Augmented Token Graphs
Juan Gutiérrez, Victor Gutiérrez-García and José Luis Blanco-Murillo
Unsupervised
segmentation from self-supervised ViT patches holds promise but lacks robustness: multi-object scenes confound saliency cues, and low-semantic images weaken patch relevance, both leading to erratic masks. To address this, we present Prior-Aware Normalized Cut (PANC), a training-free method that data-efficiently produces consistent, user-steerable
segmentations. PANC extends the Normalized Cut
algorithm by connecting labeled prior
tokens to foreground/background anchors, forming an anchor-augmented generalized eigenproblem that steers low-frequency partitions toward the target class while preserving global spectral structure. With prior-aware eigenvector orientation and thresholding, our approach yields stable masks. Spectral diagnostics confirm that injected priors widen eigengaps and stabilize partitions, consistent with our analytical hypotheses. PANC outperforms strong unsupervised and weakly supervised baselines, achieving mIoU improvements of +2.3% on DUTS-TE, +2.8% on DUT-OMRON, and +8.7% on low-semantic CrackForest
datasets.
Read the full abstract ▾
Computer Vision
Yuxuan Yao, Yuxuan Chen, Hui Li et al. (+6 more)
Multimodal Diffusion Transformers (MMDiTs) for text-to-image generation maintain separate text and image branches, with bidirectional information flow between text
tokens and visual latents throughout denoising. In this setting, we observe a
prompt forgetting phenomenon: the semantics of the
prompt representation in the text branch is progressively forgotten as depth increases. We further verify this effect on three representative MMDiTs–SD3, SD3.5, and FLUX.1 by probing linguistic attributes of the representations over the layers in the text branch. Motivated by these findings, we introduce a training-free approach,
prompt reinjection, which reinjects
prompt representations from early layers into later layers to alleviate this forgetting. Experiments on GenEval, DPG, and T2I-CompBench++ show consistent gains in instruction-following capability, along with improvements on metrics capturing preference, aesthetics, and overall text–image generation quality.
Read the full abstract ▾
All authors Yuxuan Yao, Yuxuan Chen, Hui Li, Kaihui Cheng, Qipeng Guo, Yuwei Sun, Zilong Dong, Jingdong Wang, Siyu Zhu
Computer Vision · Cross-listed from cs.LG
Vision Transformer Finetuning Benefits from Non-Smooth Components
Ambroise Odonnat, Laetitia Chapel, Romain Tavenard et al. (+1 more)
The smoothness of the
transformer architecture has been extensively studied in the context of generalization, training stability, and
adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of
vision transformer components to adapt their outputs to changes in inputs, or, in other words, their plasticity. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies a low smoothness. Our theoretical analysis and extensive experiments – over $1,000$ finetuning runs on large-scale
vision transformers – showcase that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the
attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on
transformers' functional properties. The code is available at https://github.com/ambroiseodt/vit-plasticity.
Read the full abstract ▾
All authors Ambroise Odonnat, Laetitia Chapel, Romain Tavenard, Ievgen Redko
Computer Vision
NanoFLUX: Distillation-Driven Compression of Large Text-to-Image Generation Models for Mobile Devices
Ruchika Chavhan, Malcolm Chadwick, Alberto Gil Couto Pimentel Ramos et al. (+3 more)
While large-scale text-to-image
diffusion models continue to improve in visual quality, their increasing scale has widened the gap between state-of-the-art models and on-device solutions. To address this gap, we introduce NanoFLUX, a 2.4B text-to-image flow-matching model distilled from 17B FLUX.1-Schnell using a progressive compression pipeline designed to preserve generation quality. Our contributions include: (1) A model compression strategy driven by
pruning redundant components in the
diffusion transformer, reducing its size from 12B to 2B; (2) A ResNet-based
token downsampling mechanism that reduces latency by allowing intermediate blocks to operate on lower-resolution
tokens while preserving high-resolution processing elsewhere; (3) A novel text encoder
distillation approach that leverages visual signals from early layers of the denoiser during sampling. Empirically, NanoFLUX generates 512 x 512 images in approximately 2.5 seconds on mobile devices, demonstrating the feasibility of high-quality on-device text-to-image generation.
Read the full abstract ▾
All authors Ruchika Chavhan, Malcolm Chadwick, Alberto Gil Couto Pimentel Ramos, Luca Morreale, Mehdi Noroozi, Abhinav Mehrotra
Computer Vision
RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editing
Mohammadreza Salehi, Mehdi Noroozi, Luca Morreale et al. (+4 more)
Instructional
video editing applies edits to an input
video using only text
prompts, enabling intuitive natural-
language control. Despite rapid progress, most methods still require fixed-length inputs and substantial compute. Meanwhile, autoregressive
video generation enables efficient variable-length synthesis, yet remains under-explored for
video editing. We introduce a causal, efficient
video editing model that edits variable-length
videos frame by frame. For efficiency, we start from a 2D image-to-image (I2I)
diffusion model and adapt it to
video-to-
video (V2V) editing by conditioning the edit at time step t on the model's prediction at t-1. To leverage
videos' temporal redundancy, we propose a new I2I
diffusion forward process formulation that encourages the model to predict the residual between the target output and the previous prediction. We call this Residual Flow
Diffusion Model (RFDM), which focuses the denoising process on changes between consecutive frames. Moreover, we propose a new
benchmark that better ranks state-of-the-art methods for editing tasks. Trained on paired
video data for global/local style transfer and object removal, RFDM surpasses I2I-based methods and competes with fully spatiotemporal (3D) V2V models, while matching the compute of image models and scaling independently of input
video length. More content can be found in: https://smsd75.github.io/RFDM_page/
Read the full abstract ▾
All authors Mohammadreza Salehi, Mehdi Noroozi, Luca Morreale, Ruchika Chavhan, Malcolm Chadwick, Alberto Gil Ramos, Abhinav Mehrotra
Computer Vision
Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing
Meng Lou, Stanley Yu and Yizhou Yu
Adapting pre-trained
vision models using parameter-efficient fine-tuning (PEFT) remains challenging, as it aims to achieve performance comparable to full fine-tuning using a minimal number of trainable parameters. When applied to complex dense prediction tasks, existing methods exhibit limitations, including input-agnostic modeling and redundant cross-layer representations. To this end, we propose ParaX, a new adapter-style method featuring a simple mixture-of-experts (MoE) architecture. Specifically, we introduce shared expert centers, where each expert is a trainable parameter
matrix. During a feedforward pass, each ParaX module in the network dynamically generates weight
matrices tailored for the current module via a simple dynamic parameter routing mechanism, which selectively aggregates parameter
matrices in the corresponding expert center. Dynamic weight
matrices in ParaX modules facilitate low-rank adaptation in an input-dependent manner, thus generating more customized and powerful feature representations. Moreover, since ParaX modules across multiple network layers share the same expert center, they improve feature diversity by promoting implicit cross-layer feature interaction. Extensive experimental results demonstrate the superiority of ParaX across diverse visual recognition tasks. Code is publicly released at: https://github.com/LMMMEng/ParaX.
Read the full abstract ▾
Computer Vision
Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping
Chao Zhou, Tianyi Wei, Yiling Chen et al. (+2 more)
While modern text-to-image models excel at
prompt-based generation, they often lack the fine-grained control necessary for specific user requirements like spatial layouts or subject appearances. Multi-condition control addresses this, yet its integration into
Diffusion Transformers (DiTs) is bottlenecked by the conventional “concatenate-and-attend” strategy, which suffers from quadratic computational and
memory overhead as the number of conditions scales. Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant. To this end, we propose Position-aligned and Keyword-scoped
Attention (PKA), a highly efficient framework designed to eliminate these redundancies. Specifically, Position-Aligned
Attention (PAA) linearizes spatial control by enforcing localized patch
alignment, while Keyword-Scoped
Attention (KSA) prunes irrelevant subject-driven interactions via semantic-aware masking. To facilitate efficient learning, we further introduce a Conditional Sensitivity-Aware Sampling (CSAS) strategy that reweights the training objective towards critical denoising phases, drastically accelerating convergence and enhancing conditional fidelity. Empirically, PKA delivers a 10.0$\times$ inference speedup and a 5.1$\times$ VRAM saving, providing a scalable and resource-friendly solution for high-fidelity multi-conditioned generation.
Read the full abstract ▾
All authors Chao Zhou, Tianyi Wei, Yiling Chen, Wenbo Zhou, Nenghai Yu
Computer Vision
GaussianPOP: Principled Simplification Framework for Compact 3D Gaussian Splatting via Error Quantification
Soonbin Lee, Yeong-Gyu Kim, Simon Sasse et al. (+5 more)
Existing 3D Gaussian Splatting simplification methods commonly use importance scores, such as blending weights or sensitivity, to identify redundant Gaussians. However, these scores are not driven by visual error metrics, often leading to suboptimal trade-offs between compactness and rendering fidelity. We present GaussianPOP, a principled simplification framework based on analytical Gaussian error quantification. Our key contribution is a novel error criterion, derived directly from the 3DGS rendering equation, that precisely measures each Gaussian's contribution to the rendered image. By introducing a highly efficient
algorithm, our framework enables practical error calculation in a single forward pass. The framework is both accurate and flexible, supporting on-training
pruning as well as post-training simplification via iterative error re-quantification for improved stability. Experimental results show that our method consistently outperforms existing state-of-the-art
pruning methods across both application scenarios, achieving a superior trade-off between model compactness and high rendering quality.
Read the full abstract ▾
All authors Soonbin Lee, Yeong-Gyu Kim, Simon Sasse, Tomas M. Borges, Yago Sanchez, Eun-Seok Ryu, Thomas Schierl, Cornelius Hellge
Computer Vision · Cross-listed from cs.LG
AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Models
Yuming Li, Qingyu Li, Chengyu Bai et al. (+6 more)
Reinforcement learning from human feedback (RLHF) shows promise for aligning
diffusion and flow models, yet policy optimization methods such as GRPO suffer from inefficient and static sampling strategies. These methods treat all
prompts and denoising steps uniformly, ignoring substantial variations in sample learning value as well as the dynamic nature of critical exploration moments. To address this issue, we conduct a detailed analysis of the internal
attention dynamics during GRPO training and uncover a key insight:
attention entropy can serve as a powerful dual-signal proxy. First, across different samples, the relative change in
attention entropy ({\Delta}
Entropy), which reflects the divergence between the current policy and the base policy, acts as a robust indicator of sample learning value. Second, during the denoising process, the peaks of absolute
attention entropy (
Entropy(t)), which quantify
attention dispersion, effectively identify critical timesteps where high-value exploration occurs. Building on this observation, we propose Adaptive
Entropy-Guided Policy Optimization (AEGPO), a novel dual-signal, dual-level adaptive optimization strategy. At the global level, AEGPO uses {\Delta}
Entropy to dynamically allocate rollout budgets, prioritizing
prompts with higher learning value. At the local level, it exploits the peaks of
Entropy(t) to guide exploration selectively at critical high-dispersion timesteps rather than uniformly across all denoising steps. By focusing computation on the most informative samples and the most critical moments, AEGPO enables more efficient and effective policy optimization. Experiments on text-to-image generation tasks demonstrate that AEGPO significantly accelerates convergence and achieves superior
alignment performance compared to standard GRPO variants.
Read the full abstract ▾
All authors Yuming Li, Qingyu Li, Chengyu Bai, Xiangyang Luo, Zeyue Xue, Wenyu Qin, Meng Wang, Yikai Wang, Shanghang Zhang
Computer Vision
RAIGen: Rare Attribute Identification in Text-to-Image Generative Models
Silpa Vadakkeeveetil Sreelatha, Dan Wang, Serge Belongie et al. (+2 more)
Text-to-image
diffusion models achieve impressive generation quality but inherit and amplify training-data biases, skewing coverage of semantic attributes. Prior work addresses this in two ways. Closed-set approaches mitigate biases in predefined
fairness categories (e.g., gender, race), assuming socially salient minority attributes are known a priori. Open-set approaches frame the task as bias identification, highlighting majority attributes that dominate outputs. Both overlook a complementary task: uncovering rare or minority features underrepresented in the data distribution (social, cultural, or stylistic) yet still encoded in model representations. We introduce RAIGen, the first framework, to our knowledge, for label-free rare-attribute discovery in
diffusion models, requiring no predefined minority categories. RAIGen leverages Matryoshka Sparse Autoencoders and a novel minority metric combining
neuron activation frequency with semantic distinctiveness to identify interpretable
neurons whose top-activating images reveal underrepresented attributes. Experiments show RAIGen discovers attributes beyond fixed
fairness categories in Stable
Diffusion, scales to larger models such as SDXL, supports systematic
auditing across architectures, and enables targeted amplification of rare attributes during generation. The project page is available at https://vssilpa.github.io/RAIGen_webpage/ .
Read the full abstract ▾
All authors Silpa Vadakkeeveetil Sreelatha, Dan Wang, Serge Belongie, Muhammad Awais, Anjan Dutta
Computer Vision
A Unified Formula for Affine Transformations between Calibrated Cameras
Levente Hajder
In this technical note, we derive a closed-form expression for the affine transformation mapping local image patches between two calibrated views. We show that the transformation is a function of the relative camera pose, the image coordinates, and the local surface normal.
Computer Vision
Machine Learning for Detection and Severity Estimation of Sweetpotato Weevil Damage in Field and Lab Conditions
Doreen M. Chelangat, Sudi Murindanyi, Bruce Mugizi et al. (+6 more)
Sweetpotato weevils (Cylas spp.) are considered among the most destructive pests impacting sweetpotato production, particularly in sub-Saharan Africa. Traditional methods for assessing weevil damage, predominantly relying on manual scoring, are labour-intensive, subjective, and often yield inconsistent results. These challenges significantly hinder breeding programs aimed at developing resilient sweetpotato varieties. This study introduces a computer
vision-based approach for the automated evaluation of weevil damage in both field and laboratory contexts. In the field settings, we collected data to train classification models to predict root-damage severity levels, achieving a test accuracy of 71.43%. Additionally, we established a laboratory
dataset and designed an object detection pipeline employing YOLO12, a leading real-time detection model. This methodology incorporated a two-stage laboratory pipeline that combined root
segmentation with a tiling strategy to improve the detectability of small objects. The resulting model demonstrated a mean average precision of 77.7% in identifying minute weevil feeding holes. Our findings indicate that computer
vision technologies can provide efficient, objective, and scalable assessment tools that align seamlessly with contemporary breeding workflows. These advancements represent a significant improvement in enhancing phenotyping efficiency within sweetpotato breeding programs and play a crucial role in mitigating the detrimental effects of weevils on food security.
Read the full abstract ▾
All authors Doreen M. Chelangat, Sudi Murindanyi, Bruce Mugizi, Paul Musana, Benard Yada, Milton A. Otema, Florence Osaru, Andrew Katumba, Joyce Nakatumba-Nabende
Computer Vision
Revisiting Emotions Representation for Recognition in the Wild
Joao Baptista Cardia Neto, Claudio Ferrari and Stefano Berretti
Facial
emotion recognition has been typically cast as a single-label classification problem of one out of six prototypical
emotions. However, that is an oversimplification that is unsuitable for representing the multifaceted spectrum of spontaneous emotional states, which are most often the result of a combination of multiple
emotions contributing at different intensities. Building on this, a promising direction that was explored recently is to cast
emotion recognition as a distribution learning problem. Still, such approaches are limited in that research
datasets are typically annotated with a single
emotion class. In this paper, we contribute a novel approach to describe complex emotional states as probability distributions over a set of
emotion classes. To do so, we propose a solution to automatically re-label existing
datasets by exploiting the result of a study in which a large set of both basic and compound
emotions is mapped to probability distributions in the Valence-Arousal-Dominance (VAD) space. In this way, given a face image annotated with VAD values, we can estimate the likelihood of it belonging to each of the distributions, so that emotional states can be described as a mixture of
emotions, enriching their description, while also accounting for the ambiguous nature of their perception. In a preliminary set of experiments, we illustrate the advantages of this solution and a new possible direction of investigation. Data annotations are available at https://github.com/jbcnrlz/affectnet-b-annotation.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Orientation-Robust Latent Motion Trajectory Learning for Annotation-free Cardiac Phase Detection in Fetal Echocardiography
Yingyu Yang, Qianye Yang, Can Peng et al. (+4 more)
Fetal echocardiography is essential for detecting congenital heart disease (CHD), facilitating pregnancy management, optimized delivery planning, and timely postnatal interventions. Among standard imaging planes, the four-chamber view (4CV) provides important information for CHD diagnosis, where clinicians carefully inspect the end-diastolic (ED) and end-systolic (ES) phases to evaluate cardiac structure and motion. Automated detection of these cardiac phases is thus a critical component towards fully automated CHD analysis. However, existing approaches typically rely on manual annotation of ED/ES frames, which is labour-intensive and time-consuming. We present ORBIT (Orientation-Robust Beat Inference from Trajectories), a self-supervised framework that identifies cardiac phases without manual annotations under various fetal heart orientation. ORBIT employs registration as self-supervision task and learns a latent motion trajectory of cardiac deformation, whose turning points capture transitions between cardiac relaxation and contraction, enabling accurate and orientation-robust localization of ED and ES frames across diverse fetal positions. Trained exclusively on normal fetal echocardiography
videos, ORBIT achieves consistent performance on both normal (mean absolute error 1.9 frames for ED and 1.6 for ES) and CHD cases (mean absolute error 2.4 frames for ED and 2.1 for ES), outperforming existing annotation-free approaches constrained by fixed orientation assumptions. These results highlight the potential of ORBIT to facilitate robust cardiac phase detection directly from 4CV fetal echocardiography.
Read the full abstract ▾
All authors Yingyu Yang, Qianye Yang, Can Peng, Elena D'Alberti, Olga Patey, Aris T. Papageorghiou, J.Alison Noble
Computer Vision
Gold Exploration using Representations from a Multispectral Autoencoder
Argyro Tsandalidou, Konstantinos Dogeas, Eleftheria Tetoula Tsonga et al. (+3 more)
Satellite imagery is employed for large-scale prospectivity mapping due to the high cost and typically limited availability of on-site mineral exploration data. In this work, we present a proof-of-concept framework that leverages generative representations learned from multispectral Sentinel-2 imagery to identify gold-bearing regions from space. An autoencoder foundation model, called Isometric, which is pretrained on the large-scale FalconSpace-S2 v1.0
dataset, produces information-dense spectral-spatial representations that serve as inputs to a lightweight XGBoost classifier. We compare this representation-based approach with a raw spectral input baseline using a
dataset of 63 Sentinel-2 images from known gold and non-gold locations. The proposed method improves patch-level accuracy from 0.51 to 0.68 and image-level accuracy from 0.55 to 0.73, demonstrating that generative
embeddings capture transferable mineralogical patterns even with limited labeled data. These results highlight the potential of foundation-model representations to make mineral exploration more efficient, scalable, and globally applicable.
Read the full abstract ▾
All authors Argyro Tsandalidou, Konstantinos Dogeas, Eleftheria Tetoula Tsonga, Elisavet Parselia, Georgios Tsimiklis, George Arvanitakis
Computer Vision
Clinical-Prior Guided Multi-Modal Learning with Latent Attention Pooling for Gait-Based Scoliosis Screening
Dong Chen, Zizhuang Wei, Jialei Xu et al. (+6 more)
Adolescent Idiopathic Scoliosis (AIS) is a prevalent spinal deformity whose progression can be mitigated through early detection. Conventional screening methods are often subjective, difficult to scale, and reliant on specialized clinical expertise.
Video-based gait analysis offers a promising alternative, but current
datasets and methods frequently suffer from data leakage, where performance is inflated by repeated clips from the same individual, or employ oversimplified models that lack clinical interpretability. To address these limitations, we introduce ScoliGait, a new
benchmark dataset comprising 1,572 gait
video clips for training and 300 fully independent clips for testing. Each clip is annotated with radiographic Cobb angles and descriptive text based on clinical kinematic priors. We propose a multi-modal framework that integrates a clinical-prior-guided kinematic knowledge map for interpretable feature representation, alongside a latent
attention pooling mechanism to fuse
video, text, and knowledge map modalities. Our method establishes a new state-of-the-art, demonstrating a significant performance gap on a realistic, non-repeating subject
benchmark. Our approach establishes a new state of the art, showing a significant performance gain on a realistic, subject-independent
benchmark. This work provides a robust, interpretable, and clinically grounded foundation for scalable, non-invasive AIS assessment.
Read the full abstract ▾
All authors Dong Chen, Zizhuang Wei, Jialei Xu, Xinyang Sun, Zonglin He, Meiru An, Huili Peng, Yong Hu, Kenneth MC Cheung
Computer Vision · Cross-listed from cs.LG
Diffeomorphism-Equivariant Neural Networks
Josephine Elisabeth Oettinger, Zakhar Shumaylov, Johannes Bostelmann et al. (+2 more)
Incorporating group
symmetries via equivariance into neural networks has emerged as a robust approach for overcoming the efficiency and data demands of modern deep learning. While most existing approaches, such as group convolutions and averaging-based methods, focus on compact, finite, or low-dimensional groups with linear actions, this work explores how equivariance can be extended to infinite-dimensional groups. We propose a strategy designed to induce diffeomorphism equivariance in pre-trained neural networks via energy-based canonicalisation. Formulating equivariance as an optimisation problem allows us to access the rich toolbox of already established differentiable image registration methods. Empirical results on
segmentation and classification tasks confirm that our approach achieves approximate equivariance and generalises to unseen transformations without relying on extensive data augmentation or retraining.
Read the full abstract ▾
All authors Josephine Elisabeth Oettinger, Zakhar Shumaylov, Johannes Bostelmann, Jan Lellmann, Carola-Bibiane Schönlieb
Computer Vision
Can We Build a Monolithic Model for Fake Image Detection? SICA: Semantic-Induced Constrained Adaptation for Unified-Yet-Discriminative Artifact Feature Space Reconstruction
Bo Du, Xiaochen Ma, Xuekang Zhu et al. (+8 more)
Fake Image Detection (FID), aiming at unified detection across four image forensic subdomains, is critical in real-world forensic scenarios. Compared with ensemble approaches, monolithic FID models are theoretically more promising, but to date, consistently yield inferior performance in practice. In this work, we identify the intrinsic distinctness of artifacts across subdomains, a critical barrier we term the “Ji-Zhe phenomenon". Driven by this phenomenon, we diagnose the cause of this underperformance for the first time: the collapse of the artifact feature space. The core challenge for developing a practical monolithic FID model thus boils down to the “unified-yet-discriminative" reconstruction of the artifact feature space. To address this paradoxical challenge, we hypothesize that high-level semantics can serve as a structural prior for the reconstruction, and further propose Semantic-Induced Constrained Adaptation (SICA), the first monolithic FID paradigm. Extensive experiments on our OpenMMSec
dataset demonstrate that SICA outperforms 15 state-of-the-art methods and reconstructs the target unified-yet-discriminative artifact feature space in a near-orthogonal manner, thus firmly validating our hypothesis. The code and
dataset are available at: https://github.com/venus-guangjian/SICA_OpenMMSec.
Read the full abstract ▾
All authors Bo Du, Xiaochen Ma, Xuekang Zhu, Zhe Yang, Chaogun Niu, Chenfan Qu, Mingqi Fang, Zhenming Wang, Jingjing Liu, Jian Liu, Ji-Zhe Zhou
Computer Vision
CytoCrowd: A Multi-Annotator Benchmark Dataset for Cytology Image Analysis
Yonghao Si, Xingyuan Zeng, Zhao Chen et al. (+4 more)
High-quality annotated
datasets are crucial for advancing machine learning in medical image analysis. However, a critical gap exists: most
datasets either offer a single, clean ground truth, which hides real-world expert disagreement, or they provide multiple annotations without a separate gold standard for objective evaluation. To bridge this gap, we introduce CytoCrowd, a new public
benchmark for cytology analysis. The
dataset features 446 high-resolution images, each with two key components: (1) raw, conflicting annotations from four independent pathologists, and (2) a separate, high-quality gold-standard ground truth established by a senior expert. This dual structure makes CytoCrowd a versatile resource. It serves as a
benchmark for standard computer
vision tasks, such as object detection and classification, using the ground truth. Simultaneously, it provides a realistic testbed for evaluating annotation aggregation
algorithms that must resolve expert disagreements. We provide comprehensive baseline results for both tasks. Our experiments demonstrate the challenges presented by CytoCrowd and establish its value as a resource for developing the next generation of models for medical image analysis. The
dataset is publicly available at https://github.com/YHSI5358/CytoCrowd-
Datasets.
Read the full abstract ▾
All authors Yonghao Si, Xingyuan Zeng, Zhao Chen, Libin Zheng, Caleb Chen Cao, Lei Chen, Jian Yin
Computer Vision
PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks
Junxian Li, Kai Liu, Leyang Chen et al. (+7 more)
Unified
multimodal models (UMMs) have shown impressive capabilities in complex natural image generation. However, their potential in computer-use planning tasks, which are closely related to our lives, remains rarely explored. Image generation and editing in computer-use tasks require capabilities like fine-grained perception and procedural understanding, and it is still unknown whether UMMs can finish these tasks well. Therefore, we propose PlanViz, a new
benchmark designed to evaluate image generation and editing for computer-use tasks. We focus on sub-tasks which frequently involve in daily life and require planning steps. Specifically, three representative sub-tasks are designed: route planning, work diagramming, and web&UI displaying. Data quality is ensured by including large human efforts on question, reference and criteria annotation. Additionally, a task-adaptive score, PlanScore, is proposed for comprehensive and exact evaluation. PlanScore helps understanding the correctness, visual quality and efficiency of generated images. Through experiments, we highlight key limitations and opportunities for future research on this topic. Codes and supplementary materials are in https://github.com/lijunxian111/PlanViz.
Read the full abstract ▾
All authors Junxian Li, Kai Liu, Leyang Chen, Weida Wang, Zhixin Wang, Jiaqi Xu, Fan Li, Renjing Pei, Linghe Kong, Yulun Zhang
Computer Vision · Cross-listed from cs.AI
Same Answer, Different Representations: Hidden instability in VLMs
Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena et al. (+6 more)
The robustness of
Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable
multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal
embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of
vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE
datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm
reasoning when they disrupt how models combine coarse and fine visual cues, but on the
hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.
Read the full abstract ▾
All authors Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena, Aryo Pradipta Gema, Wai-Chung Kwan, Fazl Barez, Maria Sofia Bucarelli, Fabrizio Silvestri, Pasquale Minervini
Computer Vision
CauCLIP: Bridging the Sim-to-Real Gap in Surgical Video Understanding via Causality-Inspired Vision-Language Modeling
Yuxin He, An Li and Cheng Xue
Surgical phase recognition is a critical component for context-aware decision support in intelligent operating rooms, yet training robust models is hindered by limited annotated clinical
videos and large domain gaps between synthetic and real
surgical data. To address this, we propose CauCLIP, a causality-inspired
vision-
language framework that leverages CLIP to learn domain-invariant representations for
surgical phase recognition without access to target domain data. Our approach integrates a frequency-based augmentation strategy to perturb domain-specific attributes while preserving semantic structures, and a causal suppression loss that mitigates non-causal biases and reinforces causal
surgical features. These components are combined in a unified training framework that enables the model to focus on stable causal factors underlying
surgical workflows. Experiments on the SurgVisDom hard adaptation
benchmark demonstrate that our method substantially outperforms all competing approaches, highlighting the effectiveness of causality-guided
vision-
language models for domain-generalizable
surgical video understanding.
Read the full abstract ▾