Computer Vision
Code World Model: Coding Agent as World Brain
Yiwen Chen, Guosheng Lin and Chi Zhang
World models aim to simulate how complex environments evolve under actions and events, yet existing
video-based world models primarily learn dynamics from visual observations, which reveal outcomes rather than the underlying knowledge, rules, and mechanisms governing world
evolution. This makes it difficult to maintain persistent consequences and support coherent, open-ended
evolution. We introduce Code World Model, a framework that separates world
evolution from visual realization by combining the
reasoning and coding capabilities of
language models with the generative priors of
video models. A coding
agent serves as the world
brain,
reasoning about events and their consequences and generating executable code to maintain persistent world state and perform rule-consistent
evolution. To connect executable state with visual generation, we introduce a proxy representation that encodes frame-wise spatiotemporal constraints and is compiled into a proxy
video, which conditions a
video model to render high-fidelity visual observations. We further develop data pipelines for constructing aligned proxy-observation pairs from gameplay and real-world
videos. After fine-tuning on paired gameplay data, MiniMax-H3 follows proxy-based spatiotemporal specifications from simple interactive worlds built by the coding
agent while preserving rich visual details and dynamics. These results demonstrate the potential of combining code for persistent world
evolution with
video models for flexible visual realization, providing a new path toward open-ended world models.
Read the full abstract ▾
Computer Vision
Visual General Intelligence: A White Paper
Hirokatsu Kataoka, Yoshihiro Fukuhara, Yonglong Tian et al. (+18 more)
This paper reconsiders intelligence from a
vision-centered perspective and examines whether intelligence emerging from visual experience and learning may provide a pathway toward AGI. In the
language domain, beginning with the introduction of the
Transformer architecture, the GPT series has demonstrated transfer to unseen tasks through autoregressive
language modeling on web-scale text combined with aggressive scaling. This raises a natural question, namely, what capabilities and forms of intelligence can emerge from visual modalities such as images,
videos, and geometry? In this paper, we discuss whether visual intelligence can serve as a pathway toward AGI, referred to in this paper as visual general intelligence (VGI), by bringing together contributors from diverse standpoints and affiliations. Our aim is not to offer a single definition of visual intelligence, but to clarify the principles that computer
vision should pursue in the AGI era, the visual input modalities, the
benchmarks, the learning paradigms, and the relationship between
vision, when taken as the core, and other modalities such as
language.
Read the full abstract ▾
All authors Hirokatsu Kataoka, Yoshihiro Fukuhara, Yonglong Tian, Shangzhe Wu, Oishi Deb, Ryousuke Yamada, Christian Rupprecht, Jianyuan Wang, Kohsuke Ide, Koichi Namekata, Xianzheng Ma, Yiming Chen, Robert Geirhos, Aditi Raghunathan, Yuki M. Asano, Deva Ramanan, David Fouhey, Andrew J. Davison, Yilun Du, Jiajun Wu, Zhuang Liu
Computer Vision
Embedding NDRE Trajectories into Contrastive Learning for Label-Free, Physiology-Aware Crop-Stress Staging and DSS Outputs
Shafqaat Ahmad
Timely detection of crop stress is critical for sustaining yields under increasing drought frequency, yet conventional vegetation index thresholds or image-based clustering often fail to capture stress progression, limiting their value for farm decision-making. To address this gap, we present EigenCL, a physiology-guided contrastive learning framework that stages crop stress from Sentinel-2 NDRE trajectories, with the goal of providing interpretable and transferable stress diagnostics for decision support systems (DSS). EigenCL was trained on 10,000 maize NDRE patches from drought-affected Iowa fields in 2020 and tested on Nebraska fields in 2023 without retraining, with validation incorporating soil-moisture records, U.S. Drought Monitor maps, and county-level yield statistics. The model produced four physiologically coherent stress clusters (Healthy, Mild, Moderate, Severe), significantly outperforming baselines including K-Means, SimCLR, ProtoCLR, and an ablation model (Silhouette = 0.748, DBI = 0.35, CHI = 49,624). Clusters aligned with maize growth stages, with severe stress peaking around tasseling-silking (VT-R1), a stage known to drive yield loss; moreover, EigenCL clusters correlated with soil moisture at 0-14-day lags (rho up to 0.72) and matched yield anomalies in drought-affected counties. By
embedding NDRE trajectory dynamics into contrastive learning, EigenCL enables early stress alerts and interpretable DSS outputs (e.g., heatmaps, scouting priorities, regional risk indices), extending beyond single-date NDRE thresholds and supporting scalable monitoring for
climate-smart agronomy.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.HC
Do Vision-Language Models Agree on the Affective Qualities of Shape? A Cross-Model Audit for Generative Design Interfaces
Luca Bux, Thiago Rios, Ingo Scholtes et al. (+2 more)
Generative design interfaces increasingly expose semantic controls that let users steer output with concepts such as "more elegant" or "more minimalist," typically encoded by a
vision-
language model (VLM). A practical question is whether state-of-the-art VLMs represent objects consistently in terms of the same concept. We audit 6 VLMs by ranking untextured 3D objects along Kansei adjective pairs, where Kansei describes affective impressions of product form, with each axis defined as the difference between the text representations of its two poles.
Geometric pairs serve as positive controls, and pairs of unrelated adjectives establish an empirical null. Across 10 categories of ShapeNet database, affective axes converge above the null (mean pairwise rank correlation 0.36 vs. 0.14) but below the
geometric ceiling (0.44). The agreement between models is partial and highly uneven: on the three axes shared by all categories, mean convergence ranges from 0.21 for bookshelves to 0.51 for jars. Convergence depends primarily on whether a category's representational variation aligns with the semantic direction being evaluated, rather than simply on how much the objects vary in shape overall. Cross-model convergence does not imply agreement with human judgments. Based on our findings, we implement a UI prototype that shows how the audit can inform which Kansei descriptors to expose as controls for a given object class and which to withhold.
Read the full abstract ▾
All authors Luca Bux, Thiago Rios, Ingo Scholtes, Stefan Menzel, Bernhard Sendhoff
Computer Vision
LUTSeg: A Longitudinal Multi-Expert Dataset for Ulcer Tissue Segmentation
Karen Sanchez, Carlos Hinojosa, Albert A. Ávila et al. (+5 more)
Quantifying wound tissue composition is essential for monitoring chronic ulcer progression and guiding treatment decisions. However, pixel-level annotations are costly, and multi-tissue wound
datasets remain scarce, particularly for neglected diseases such as leprosy. We introduce LUTSeg, a longitudinal chronic ulcer
dataset comprising 141 images from 39 patients with wound masks and five tissue categories annotated by five expert clinicians, including a multi-expert gold-standard subset for inter-rater agreement analysis. To establish an initial
benchmark for LUTSeg, we further propose TiSage, a semi-supervised tissue
segmentation framework that integrates multi-scale semantic priors from a frozen medical
vision-
language model within a teacher-student architecture. We evaluate TiSage on LUTSeg and DFUTissue, showing improvements over supervised and semi-supervised baselines in most low-label settings. Code & data: https://github.com/carlosh93/TiSage
Read the full abstract ▾
All authors Karen Sanchez, Carlos Hinojosa, Albert A. Ávila, Andrea C. Riano-Rojas, Diego H. Romero, Jenny C. Páez, Martina Llinás, Bernard Ghanem
Computer Vision
Learning Late, Guiding Early: Timestep-Decoupled Semantic Guidance for Fair Face Generation
Subir Kumar Parida, Rajbabu Velmurugan, Ketan Kotwal et al. (+2 more)
Demographic imbalance in synthetic face generation can propagate to downstream face recognition systems, making
fairness an important consideration when
diffusion models are used for data generation. Existing
fairness-aware generation approaches often require model retraining, architectural modifications, or repeated guidance throughout the reverse
diffusion process. In this work, we introduce Semantic Boundary Predictor (SBP), an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising. Our approach is motivated by the observation that latent representations at different
diffusion timesteps play distinct semantic roles: late-stage latents provide stronger demographic separability, whereas early-stage latents offer greater flexibility for semantic intervention. SBP leverages this timestep decoupling by learning linear semantic boundaries from late-stage latent representations while applying them only once at the initial noisy latent, allowing the remainder of the reverse denoising process to proceed unchanged. The method requires neither retraining nor fine-tuning of the underlying Latent
Diffusion Model and operates without external balanced
datasets. Experiments on CelebA-HQ demonstrate substantial improvements in demographic
fairness, reducing
fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race, while maintaining perceptual image quality across demographic groups. Owing to its one-shot inference strategy and model-agnostic design, SBP introduces only a small computational overhead and can be readily integrated with existing pre-trained latent
diffusion models.
Read the full abstract ▾
All authors Subir Kumar Parida, Rajbabu Velmurugan, Ketan Kotwal, R.S. Sengar, Swati Hiremath
Computer Vision
Precipitation Downscaling Using Foundation Model-Conditioned Diffusion
Victor Nascimento Ribeiro, Jorge Guevara, Jorge Sebastian Moraga et al. (+7 more)
High-resolution precipitation fields are essential for hydrological impact assessment, yet global
climate model outputs are too coarse and biased for direct use. AI-based statistical downscaling with
diffusion models offers a promising approach, but the mechanism by which large-scale atmospheric predictors condition generation remains largely unexplored. We investigate three conditioning strategies for a denoising
diffusion probabilistic model applied to daily precipitation downscaling: channel concatenation of upsampled coarse predictors, cross-
attention conditioning with a learned convolutional encoder, and cross-
attention conditioning with the frozen encoder of the pretrained Prithvi WxC weather foundation model. All strategies are evaluated against an unconditioned baseline under identical conditions using probabilistic, distributional, spectral, and extreme-event metrics for the Colorado River Basin. Concatenation conditioning achieves the lowest point-wise CRPS and MSE, but tends to produce over-smoothed fields that suppress high-intensity events. In contrast, cross-
attention conditioning provides substantially better distributional realism and modest improvements in spectral fidelity. Improvements are greatest for extremes: the Prithvi-WxC conditioned model retains over half of >100mm/day events, although estimates are uncertain due to limited samples. When trained on the full
dataset, the learned convolutional model performs similarly to the foundation model-conditioned approach while requiring lower computational resources. However, the Prithvi-WxC-conditioned model achieves comparable performance with only five years of training data. These results indicate that cross-
attention conditioning offers advantages over simple concatenation for probabilistic precipitation downscaling, and that pre-trained foundation model representations may offer benefits in data-limited settings.
Read the full abstract ▾
All authors Victor Nascimento Ribeiro, Jorge Guevara, Jorge Sebastian Moraga, Chris Lucas, Natalie Lord, Andrew Taylor, Edward Lockhart, Will Trojak, Johannes Schmude, Anne Jones
Computer Vision
DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors
Tuo Chen, Jie Gui, Minjing Dong et al. (+4 more)
Self-supervised learning (SSL) encoders are vulnerable to
backdoor attacks, posing threats to both visual SSL encoders and
vision-
language encoders. Existing defenses are typically designed for only one of these paradigms and rely on restrictive assumptions such as access to uninfected in-distribution data or precomputed pseudo-labels, which are difficult to satisfy in practice. To address these limitations, we propose DEFUSE, a generalizable
backdoor detection framework for SSL encoders. Inspired by
Bayesian posterior inference, we reformulate
backdoor detection as a representation-conditioned image likelihood estimation problem parameterized by a conditional
diffusion generative model. Uninfected representations tend to yield semantically consistent reconstructions, whereas backdoored ones are more likely to be mapped to the attacker's target class or semantically meaningless images, deviating from the original semantics and thereby exposing the
backdoor. However, we find that the exact likelihood is intractable, because highly abstracted representations discard the low-level information necessary for pixel-faithful reconstruction. We therefore relax the objective to semantic reconstruction and evaluate it in a well-separated representation space provided by a reference encoder. Rather than training from scratch, we fine-tune a pretrained
diffusion model, leveraging its generative prior to map data onto the natural image
manifold while preserving semantic content. Extensive experiments demonstrate that DEFUSE substantially outperforms existing detectors across diverse attack settings, generalizing to both visual SSL and
vision-
language encoders. Notably, our method greatly reduces the reliance on prior knowledge about the victim encoder or the attack strategy. The source code is available at https://github.com/jsrdcht/DEFUSE .
Read the full abstract ▾
All authors Tuo Chen, Jie Gui, Minjing Dong, Lanting Fang, Ju Jia, Benlei Cui, Jian Liu
Computer Vision
THA-Flow Generative Model: Prosthesis Geometry Prediction from Preoperative CT
Yiping Wang, Jie Li, Jingyu Shen et al. (+1 more)
Preoperative planning for total hip arthroplasty (THA) is commonly framed as selecting a single prosthesis configuration and placement for a patient's osseous anatomy. In practice, however, the same anatomy may admit several clinically reasonable solutions, making planning inherently a one-to-many problem that is better represented by a conditional probability distribution. We present THA-Flow, a conditional flow-matching model that generates three-dimensional prosthesis geometry directly from preoperative CT. Separate AutoencoderKL models compress preoperative bone anatomy and prosthesis geometry, while a three-dimensional UNet learns a rectified flow from Gaussian noise to the prosthesis latent space under spatial bone conditioning and optional structured prosthesis parameters. The retrospective cohort comprised 1,355 hips from 1,149 patients undergoing primary THA. Following rigid registration of postoperative CT to preoperative CT, the actual postoperative prostheses were transformed independently according to the pelvic and femoral registrations and represented as a dual-channel truncated signed distance field. The prosthesis autoencoder achieved a peak signal-to-noise ratio of 47.11 dB and a structural similarity index of 0.9964 on the validation set. Complete acetabular and femoral geometries were generated across seven major stem models representing 93.4% of the cohort. Repeated bone-conditioned sampling preserved component position,
alignment, and the principal bone-prosthesis interfaces while allowing limited local
geometric variation. To our knowledge, THA-Flow represents the first application of generative AI to three-dimensional
surgical planning for THA.
Read the full abstract ▾
All authors Yiping Wang, Jie Li, Jingyu Shen, Liao Wang
Computer Vision
Socialized Detector Learning: Trajectory-Guided and Reciprocal Distillation for Heterogeneous Object Detectors
Weihao Li, Yunqi Zhu, Zhihe Fan et al. (+5 more)
Object detection knowledge is fragmented across independently trained, heterogeneous detectors with complementary category supports. In socialized learning, this knowledge resides in a society, and learning aims to evolve the society collectively through exchange. However, aggregation-based socialization does not explicitly plan transfer order, whereas progressive multi-teacher
distillation considers order but remains a one-way student enhancement in a shared category space. Building on Socialized Learning, we formulate Socialized Detector Learning (SDL) for heterogeneous, category-specialized object detectors and propose Trajectory-Guided and Reciprocal
Distillation (TGRD).TGRD estimates directed operational Inter-Detector Transfer Difficulty (IDTD) from held-out feature-
alignment residuals, precomputes a fixed score table, and greedily constructs a carrier trajectory. Along the trajectory, knowledge is progressively consolidated into a union-category carrier and then returned to experts through reciprocal transfer. A conditional proxy-certificate analysis shows that, under stated assumptions, the progressive certificate is no larger than an aggregated-target counterpart. On MS COCO with four heterogeneous experts and two carrier initializations, final carriers outperform epoch-matched simultaneous aggregation controls by 2.6 AP in both settings. Reciprocal detectors attain 20.8–28.4 AP on previously unsupported categories while remaining within 1.3 AP of original expert-specific performance. These results support order-aware progressive consolidation followed by reciprocal transfer as a viable mechanism for detector-society
evolution.
Read the full abstract ▾
All authors Weihao Li, Yunqi Zhu, Zhihe Fan, Ruipu Zhao, Boan Tao, Xinjie Yao, Yan Fan, Pengfei Zhu
Computer Vision
FlowMoDL: Model-Based Deep Learning with Conjugate-Gradient Data Consistency for Highly Accelerated 4D Flow MRI Reconstruction
Tristan Gottwald, Michelle Bruch, Mubashir-Ul Hassan et al. (+6 more)
We present FlowMoDL, an unrolled neural network for highly accelerated 4D flow MRI reconstruction that directly optimizes for both anatomical magnitude and phase-derived velocity accuracy. Building on the MoDL framework, FlowMoDL alternates a learned (3+1)D spatiotemporal denoiser with conjugate-gradient data-consistency updates based on the SENSE forward model. A novel dual-pathway conditioning scheme adapts the denoiser features and data-consistency weighting, enabling a single model to handle varying acceleration factors ($10\times$ to $50\times$). To ensure physiological accuracy, the network is trained using a deep-supervision composite loss that explicitly penalizes velocity magnitude and angular errors, stabilized by a curriculum schedule. We evaluate FlowMoDL on the multi-center CMRx4DFlow
dataset against classical and deep-learning baselines (CG-SENSE, MoDL, FlowVN, and FlowMRI-Net). A key advantage of FlowMoDL is its superior gradient step efficiency. When evaluated under an equivalent, limited budget of gradient steps, competing flow-specific networks degrade significantly. In contrast, FlowMoDL robustly converges and strictly outperforms all competitors across all acceleration factors in magnitude SSIM, nRMSE, relative velocity error, and angular error, successfully recovering sharp structural details and temporally coherent velocity fields.
Read the full abstract ▾
All authors Tristan Gottwald, Michelle Bruch, Mubashir-Ul Hassan, Fatma Alickovic, Milan Kloiber, Daniel Tenbrinck, Torsten Panholzer, Melanie Schaller, Jana Hutter
Computer Vision
Steer the Sampling, Not the Kernel Grid: Geometry-Guided Sampling Operator for Volumetric Segmentation
Sizhe Wang, Himashi Peiris and Zhaolin Chen
Accurate 3D
segmentation is central to quantitative lesion assessment and anatomy mapping for clinical planning and follow-up. Thin, elongated, and fine anatomical/pathological structures (e.g., vessels) are a particularly challenging case: a one-voxel boundary error can disconnect a branch and change clinically relevant
topology. In encoder-decoder networks (e.g., U-Net), repeated downsampling and fixed-grid convolution blur or alias fine structures and weaken orientation cues, so early mistakes propagate across scales. We propose a geometry-guided local operator that steers where features are sampled, rather than deforming convolutional kernels, under a single formulation for both feature refinement (stride 1) and resolution reduction (stride > 1). At each voxel, it predicts a local orientation and bounded step sizes, samples symmetrically along these directions, and transforms paired samples into compact
geometric and boundary cues with lightweight mixing; a cross-scale consensus aligns encoder and decoder features at skip connections to reduce
geometric mismatch. Replacing all stride 1 and stride 2 operators in a 3D U-Net yields consistent improvements on BraTS, MSD Hepatic Vessel, and TDSC-ABUS, with notably better boundary metrics (e.g., BraTS Dice 86.1 to 88.9, HD95 7.1 to 6.2; TDSC-ABUS HD95 39.1 to 27.8) while reducing parameters from 2.3M to 0.8M. We further demonstrate that the operator can be integrated into other backbones (e.g., nnU-Net, Swin-UNETR, and MedNeXt) without changing their macro-architectures while providing consistent performance gains.
Read the full abstract ▾
Computer Vision
Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy
Juan Iñaki Larrea, Lucas Mansilla and Enzo Ferrante
Foundation models are increasingly deployed for medical image analysis. However, under the inter-institutional distribution shift typical of deployment, their performance varies widely and cannot be known without target-domain labels, which are rarely available. This leaves a practical question unresolved: given several candidate foundational models and labeled-data from a source domain, which one to deploy in an unlabeled target domain? We propose a label-free selection criterion built on SUDO, a framework for evaluating clinical AI systems without ground-truth annotations. SUDO partitions the unlabeled target data by predicted probability and, for each region, measures a pseudo-label discrepancy reflecting class contamination; aggregated across regions, this yields a score (AURCC) requiring neither target annotation nor fine-tuning. We show that AURCC can be used to rank a variety of
vision-
language models (BioMedCLIP, CXR-CLIP, CheXzero, MedCLIP, MedImageInsight, CLIP) on chest X-ray classification across three inter-hospital shift scenarios, under zero-shot and MLP-probe regimes. The AURCC ranking recovers the ground-truth ranking with Spearman rho up to 0.943 (p<0.05). Against the natural baseline of ranking by held-out source accuracy, AURCC is competitive when the labeled source is large and yields a more accurate ranking once it is small; the regime of interest in resource-constrained settings.
Read the full abstract ▾
Computer Vision
TDFNet: Tri-projection Deformable Fusion Network for Panoramic Salient Object Detection
Qiangqiang Zhou, Jiacong Yu, Jiawei Xu et al. (+3 more)
Recent years have witnessed the growing potential of panoramic salient object detection in robotic
vision, virtual reality, and related applications. However, projecting spherical scenes onto 2D planes inevitably introduces
geometric distortions, which fundamentally limit the effectiveness of existing projection-based methods. Specifically, Equirectangular Projection (ERP) suffers from severe polar stretching distortions, while cube map projection introduces discontinuities across cube-face boundaries, resulting in degraded feature discriminability and compromised
geometric consistency. To address these limitations, we propose TDFNet, the first Tri-projection Deformable Fusion Network for panoramic salient object detection, exploiting complementary projection representations to alleviate
geometric distortions and improve detection performance.Specifically, we design a cross-projection deformable
attention (CDA) module that leverages spatial correspondences between different projections to construct geometry-aware sampling locations, guiding deformable
attention for cross-projection contextual aggregation and enhancing robustness against projection-induced deformations. Furthermore, we introduce a latitude-guided fusion module, which utilizes spherical latitude priors to construct
geometric confidence weights for adaptively balancing ERP and CMP features. Meanwhile, LGF incorporates distortion-reduced semantic references from Tangent Projection to achieve cross-projection feature refinement and spatial
alignment.By constructing a three-branch encoding architecture based on ERP, CMP, and Tangent Projection, TDFNet simultaneously preserves global spatial continuity, local
geometric details, and fine-grained boundary information.
Read the full abstract ▾
All authors Qiangqiang Zhou, Jiacong Yu, Jiawei Xu, Yong Chen, Xin Huang, Ping Li
Computer Vision · Cross-listed from cs.LG
Learning from waste: Machine Learning for health risk prediction and computer vision-based sorting in Ghana
Hilda Adwubi Osei, Catherine Tenewaa Osei and Desdemona Yaa Asobayire
The inappropriate disposal of solid waste remains a significant public health and environmental concern worldwide, including in Ghana. Poor sanitation and improper waste management practices contribute to substantial economic costs and avoidable deaths annually. In 2022, a field study in Atonsu, Kumasi, Ghana, reported a community-perceived relationship between household waste disposal and illness patterns, but only through descriptive analysis without quantitative validation. This study extends that investigation using two data-driven approaches. First, a Random Forest classifier was developed to predict illness categories using waste disposal practices and demographic survey data. On a held-out group of respondents who reported illness (N=69), the model obtained a macro F1 score of 0.63, with disposal method emerging as the most important substantive predictor of illness type. Second, a MobileNetV2 image classification model enabled automated waste sorting via visual recognition, achieving 88.2% accuracy and a macro F1 score of 0.87 on the test set (N=415). The
vision-based approach offers an affordable, camera-driven alternative to complex multi-sensor systems, making it highly suitable for resource-constrained settings. Taken together, the findings provide quantitative evidence for a community health relationship previously documented only qualitatively. They demonstrate the potential for automated waste-sorting in low-resource environments. Importantly, the results illustrate that technological performance alone does not guarantee public health improvements; effective institutional support and implementation are equally necessary.
Read the full abstract ▾
Computer Vision
Moving Beyond More Views: Redundancy-Aware Ego-Exo Fusion for Proficiency Estimation
Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye et al. (+1 more)
EgoExo proficiency estimation aims to assess action quality by integrating fine-grained motion cues from
egocentric (1st-person) views with spatial context from multiple exocentric (3rd-person) views. Simply adding more exocentric views degrades EgoExo performance, as redundant or noisy perspectives dilute useful motion cues. Our analysis identifies two key causes: (1) Multiview redundancy - From the data perspective, certain views provide limited or noisy information, diluting discriminative cues; (2) Overfitting - From the feature perspective, conventional fusion increases representational complexity, causing the model to memorise view-specific patterns rather than learn generalisable representations. To address these issues, we propose two complementary modules: AdaMVS, which adaptively identifies and fuses the most informative view
tokens under weak supervision from the data perspective, and VIB-GB, which combines Gradient Blending and Variational Information Bottleneck regularisation from the feature perspective to compress redundant signals and suppress overfitting during training. Experiments on EgoExo-4D and EgoExo-Fitness demonstrate that our method learns both which view to look at and how to fuse them, achieving new state-of-the-art results. Our source code is available at https://github.com/dx199771/AdaMVS
Read the full abstract ▾
All authors Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye, Andrew Gilbert
Computer Vision
InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control
Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli et al. (+5 more)
Co-
speech gesture generation has made significant progress toward realistic full-body motion from speaker audio, yet existing models lack fine-grained spatial controllability of individual joints. To address this, we introduce InteractGesture, a model-agnostic, inference-time method for spatially controllable gesture generation. InteractGesture guides target latent estimates of a
diffusion sampler through a differentiable RVQ-VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. A primary challenge in streaming co-
speech generation is chunk-wise dependency: standard sequential inference freezes prior chunks, preventing spatial constraints in future chunks from adjusting preceding trajectories and causing boundary inconsistencies. To overcome this limitation, we propose Progressive Chunk Guidance, a chunk-window strategy that maintains an active set of editable chunk latents with staggered delays, enabling spatial constraints to propagate gradients backward across chunk boundaries during streaming generation. Experiments on the BEAT2
dataset show that InteractGesture improves multi-joint spatial control while preserving overall gesture quality. Furthermore, our approach supports diverse applications, including sparse joint positioning, dense joint trajectory control, and directional pointing. Our project page is available at https://exitudio.github.io/interactgesture-page .
Read the full abstract ▾
All authors Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli, Wenjing Zhang, Joanna Materzynska, Pu Wang, Vittorio Ferrari, Jie Shen
Computer Vision
MIMONet: Multi-scale Input and Multi-scale Output Network for Salient Object Detection
Zhaojian Yao, Wei Gao, Tiesong Zhao et al. (+2 more)
The existing methods for saliency detection task focus on the application of multi-level features, aiming to take advantage of the respective strengths of high- and low-level features. However, because the inputs of these models are single-size images, their multi-level features have difficulty in learning the knowledge of size variations of salient objects. Object-scale variation learning has great potential for detecting multi-scale objects, which has not been fully explored by existing methods. To improve the recognition ability of a model for objects with different sizes, we are inspired by the image pyramid to propose a Multi-scale Input and Multi-scale Output Network (MIMONet). In MIMONet, we extract multi-level features for three images with different resolutions to form three encoder branches, and information will be exchanged between the branches. The advantage of this approach is that the features of one branch can learn the knowledge of target size variation from the features of the other two branches. In addition, we design a Multi-scale Perception (MSP) module, in which the input feature layer is divided into several sub-layers with different resolutions. Capturing the multi-level structure information of the objects in these sub-layers can make the objects more fully perceived. For network training, we propose a Joint Saliency Loss (JSL), which can constrain multiple saliency maps output by the network to identify the same foreground objects, and induce their boundaries to be preserved clearly. Experimental results show that MIMONet has stronger detection capabilities and harvests better evaluation scores on multiple
datasets compared to existing models. The code of our model will be released.
Read the full abstract ▾
All authors Zhaojian Yao, Wei Gao, Tiesong Zhao, Hui Yuan, Sam Kwong
Computer Vision
LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding
Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase et al. (+3 more)
Long-
video MLLMs must model temporal change before a limited visual-
token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the
vision encoder and the LLM. Its grouped 2D fast weights adapt to each
video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream
reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than
attention- and fixed-state recurrent resamplers across three
benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic
memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five
video understanding
benchmarks.
Read the full abstract ▾
All authors Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase, Sam Ade Jacobs, Mathis Bode, Mohamed Elhoseiny
Computer Vision
Difficulty-Aware Sample Allocation for Adaptive Data Augmentation in Semantic Segmentation
Olasimbo Ayodeji Arigbabu and Abimbola Ismail Arigbabu
Data augmentation is a standard component of modern semantic
segmentation pipelines, but most augmentation techniques allocate transformations uniformly across training samples or adapt to a single difficulty signal such as loss. This ignores the fact that
segmentation difficulty is multi-factorial, since ambiguous predictions, persistent optimization errors, rare classes, and complex object boundaries can each make a sample informative in different ways. This paper introduces Difficulty-Aware Sample Allocation (DASA), an architecture-agnostic framework that assigns stronger augmentation to samples estimated to be more difficult. DASA combines prediction ambiguity, training loss, class rarity, and boundary complexity into a normalized difficulty score, then maps that score to sample-specific augmentation strength during iterative training. Experiments on Oxford-IIIT Pet and binary Pascal VOC
segmentation with U-Net, DeepLabV3, and SegFormer-B0 show that DASA improves over standard training and is competitive with or stronger than single-signal adaptive baselines. On Oxford-IIIT Pet, DASA improves DeepLabV3 from 0.633 to 0.740 mIoU. On binary Pascal VOC, DASA obtains the best foreground IoU for all three evaluated architectures. These results attest to the value of multi-factor difficulty estimation as a practical mechanism for directing augmentation where it is most useful.
Read the full abstract ▾
Computer Vision
Skeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretraining
Koshiro Nagano, Fumiaki Sato, Ryo Hachiuma et al. (+2 more)
We propose a novel pretraining strategy for skeleton-based zero-shot spatio-temporal action localization to estimate unseen actions for person instances while overcoming high annotation costs for training via new target actions and pretraining using large-scale action scenery
datasets. Specifically, our approach, termed Skeleton-
Language feature Pooling Switching, introduces a weakly-supervised
vision-
language pretraining mechanism. This mechanism transitions pooling kernels from pretraining, which aggregates skeleton features at the
video level and aligns them with each
video's known action text
embeddings, to the inference phase that computes instance-level features without training via target actions. Furthermore, we propose Scene-Mixed Discriminative Contrastive Learning to distinguish actions at the instance level within the combined scene through the MIL framework. Our experiments on four public spatio-temporal action localization and classification
datasets demonstrate that the proposed method effectively addresses annotation limitations.
Read the full abstract ▾
All authors Koshiro Nagano, Fumiaki Sato, Ryo Hachiuma, Kazuki Tsutsukawa, Taiki Sekii
Computer Vision
Unsupervised Anatomical Feature Learning via Diffusion Models: Enhanced Medical Image Segmentation with Denoising Diffusion Probabilistic Models
Akshat G, Divyansh Gupta, Shaleen Bhatnagar et al. (+2 more)
Acquiring pixel-level annotations for medical image
segmentation is a severe bottleneck. Traditional U-Net architectures, while effective, learn local texture patterns and lack awareness of global anatomical structures, leading to boundary delineation failures in low-data regimes. This research paper proposes utilizing unsupervised Denoising
Diffusion Probabilistic Models (DDPMs) to extract anatomical features. We train a DDPM on 21 unlabeled abdominal CT scans to learn structural representations, transferring the encoder weights to a downstream
segmentation task evaluated on the BTCV multi-organ
dataset.
Diffusion pretraining significantly improved liver
segmentation: Dice increased from $0.75\pm0.36$ to $0.93\pm0.16$ ($p < 5.33\times10^{-26}$, 0.529 Cohen's d), Average Surface Distance (ASD) decreased by 66%, and 95th-percentile Hausdorff Distance (HD95) reduced by 45%. For kidney
segmentation, Dice improved from $0.90\pm0.19$ to $0.95\pm0.10$ ($p < 4.01\times10^{-11}$). Multi-organ pooled performance showed a 68% variance reduction and a 74% improvement in boundary precision (Dice $0.95\pm0.07$). Crucially, frozen encoder models retained > 80% of fine-tuned performance without exposure to
segmentation labels, proving the existence of learned anatomical priors. In low-data scenarios,
diffusion-pretrained models maintained robust performance with only 50% (Dice: 0.92 liver, 0.94 kidney), 25%, and even 10% (Dice: 0.89 liver, 0.71 kidney) of labeled data. Using unlabeled images for
diffusion-based pretraining successfully embeds robust anatomical features prior to human supervision, transforming U-Nets into anatomy-aware systems.
Read the full abstract ▾
All authors Akshat G, Divyansh Gupta, Shaleen Bhatnagar, Shilpa Ankalaki, Tusar Kanti Mishra
Computer Vision
CloSeR: Unified Relational Distillation from Closed-Set Teachers for Category Discovery
Yuanpei Liu, Zhenqi He, Jialu Tang et al. (+1 more)
Generalized Category Discovery (GCD) is an intriguing open-world problem that has garnered increasing
attention: given partially labelled data, the goal is to correctly recognize known classes while discovering coherent novel categories from unlabelled samples. Recent GCD methods typically adapt foundation models by jointly optimizing supervised classification and unsupervised discovery objectives on mixed labelled and unlabelled data. While effective, this coupled training can entangle closed-set recognition and open-set discovery, leading to objective conflict and biased predictions, and may disturb the semantic geometry of pretrained representations under limited labels and noisy pseudo-labels. We propose CloSeR, a simple plug-and-play framework that injects Closed-Set Relational knowledge into GCD training. CloSeR first builds a domain-adapted closed-set teacher by tuning lightweight block-wise adapters on labelled known-class data while keeping the foundation model backbone frozen, thereby preserving pretrained priors at low training cost. It then transfers the teacher's knowledge to downstream GCD via Unified Relational
Distillation (URD), which distills complementary global sample-to-prototype relations to anchor known-class semantics and local sample-to-sample relations to preserve neighborhood structure, using separate feature pathways to reduce optimization interference. CloSeR is head-agnostic and readily integrates with both parametric and non-parametric GCD methods. Extensive experiments with DINO and DINOv2 backbones on six
benchmarks (CIFAR-10/100, ImageNet-100, CUB, Stanford-Cars, and FGVC-Aircraft) show consistent gains over GCD baselines, achieving state-of-the-art performance. Project page: https://visual-ai.github.io/closer/
Read the full abstract ▾
All authors Yuanpei Liu, Zhenqi He, Jialu Tang, Kai Han
Computer Vision
Deep Learning Segmentation of Diffusion-Weighted MRI Acute Ischaemic Stroke: A Pragmatic Evaluation Across Three Datasets
Atle Bjørnerud, Till Schellhorn, Thor H. Skattør et al. (+4 more)
Objective:
Diffusion-weighted MRI (DWI-MRI) is the gold standard for visualizing and quantifying acute ischaemic stroke (AIS). Although deep learning methods can accurately segment AIS lesions, the optimal image inputs and model architecture remain uncertain. We evaluated whether accurate AIS lesion
segmentation can be achieved using a pragmatic deep learning approach with minimal preprocessing and clinically feasible inference times. Materials and Methods: Self-configured nnU-Net models were trained on 1,744 DWI cases from local, national, and open-access
datasets and tested on 436 cases. Four experimental conditions were evaluated using five-fold cross-validation: with or without
brain extraction and using either DWI alone or DWI plus apparent
diffusion coefficient (ADC) images as inputs. Two architectures were compared: the baseline nnU-Net (base) and a residual encoder nnU-Net (ResEnc). Performance was benchmarked against the DeepISLES ensemble model from the 2022 ISLES challenge. Results: In the test set (n=436), the base model achieved a median (IQR) Dice similarity coefficient (DSC) of 0.84 (0.19). For the base model, only two of six pairwise comparisons between input configurations showed significant differences. ResEnc produced small but significant improvements in DSC compared with the base model for DWI, DWI+
brain extraction, and DWI+ADC inputs (all p<0.02), but not for DWI+ADC+
brain extraction (p>0.50). The base model significantly outperformed DeepISLES, particularly in patients with smaller infarct volumes (signed-rank test, p<0.01). Conclusions: A baseline nnU-Net trained on DWI alone, without preprocessing, enabled fast and accurate AIS lesion
segmentation. This streamlined approach may facilitate clinical research and support acute stroke imaging workflows
Read the full abstract ▾
All authors Atle Bjørnerud, Till Schellhorn, Thor H. Skattør, Terje Nome, Jon André Ottesen, Anne Hege Aamodt, Bradley J MacIntosh
Computer Vision
Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Models
Yiwen Liang, Hui Chen, Yizhe Xiong et al. (+7 more)
Test-time adaptation (TTA) has been widely explored in single-label recognition, effectively mitigating distribution shifts, especially when combined with
vision-
language models. However, real-world images often contain multiple objects, while the more practical multi-label test-time adaptation (MLTTA) has received little
attention so far. Recent cache-based TTA methods have shown promising efficiency and effectiveness, yet directly extending them to multi-label scenarios suffers from a one-to-many mapping problem: a shared global representation entangling co-occurring objects is stored as class-wise cache prototypes, inducing dominant-label bias and compromised cache calibration. While introducing region-level cues helps isolate class-specific evidence, such regional evidence can also be unreliable under distribution shifts, making its identification and utilization non-trivial. To address these issues, we introduce PuRF, a novel PuRiFication-driven cache-based method for multi-label test-time adaptation of
vision-
language models. Specifically, PuRF first performs region purification to identify reliable regions, providing comprehensive regional cues for multi-label recognition and enabling fine-grained
alignment. Based on these purified regions, PuRF conducts cache purification to enhance cache representation and adaptability, where episodic purification builds a discriminative region-based cache, and temporal refreshing further promotes long-term cache adaptability. Experiments demonstrate that PuRF consistently outperforms state-of-the-art methods, achieving a notable 4.05% mAP improvement on ViT-B/32 across five
datasets.
Read the full abstract ▾
All authors Yiwen Liang, Hui Chen, Yizhe Xiong, Mengyao Lyu, Yuhan Cao, Zijia Lin, Shuaicheng Niu, Sicheng Zhao, Jungong Han, Guiguang Ding
Computer Vision
Diffusion Transformers for Roof Graph Synthesis and Reconstruction
Daniel Panangian and Ksenia Bittner
We present RoofDiT, a generative framework for 2D roof graph synthesis and reconstruction. Roofs are compactly described as planar graphs of junctions and structural edges, but existing methods often rely on fixed
geometric rules or direct reconstruction objectives. RoofDiT instead models roof structures directly as vertex-edge graphs and learns a conditional generative prior over their geometry and connectivity. Our framework follows a two-stage design: a
diffusion transformer generates roof vertices, and an edge prediction module infers the corresponding graph
topology. To improve
geometric fidelity, RoofDiT combines relative geometry-aware
attention with footprint and aerial-image conditioning, while using an
alignment regularizer to encourage common horizontal, vertical, and diagonal roof patterns. The same model supports unconditional generation, footprint-conditioned synthesis, and image-guided reconstruction by changing the conditioning signal. Experiments show improved graph generation quality over a
diffusion baseline, favorable performance against a straight-skeleton prior in the footprint-conditioned setting, and the highest edge F1 among compared methods for image-guided reconstruction.
Read the full abstract ▾
Computer Vision
MAMA-FLUX.2: Image-to-Image Synthesis of Post-Contrast Breast DCE-MRI for the MAMA-SYNTH Challenge
Kamil Kwarciak and Marek Wodzinski
Dynamic contrast-enhanced breast MRI is central to cancer diagnosis and monitoring, but requires gadolinium-based contrast
agents. In this work, we address pre-to-post contrast breast MRI synthesis for the MAMA-SYNTH challenge. We propose MAMA-FLUX.2, a conditional latent flow-matching approach based on FLUX.2-Klein-4B. The pre-contrast image is encoded as spatial conditioning, while the model predicts the flow field associated with the post-contrast target latent. To adapt the pretrained model efficiently, we use LoRA fine-tuning and introduce a regional training objective combining global flow matching, tumor-region supervision, and stable foreground regularization. We further investigate LoRA rank, intensity windowing, and regional loss weights on axial slices, prioritizing clinically relevant tumor-focused metrics. Our ablation study shows that moderate tumor and stable-foreground weighting improves the trade-off between image fidelity and tumor-region accuracy. The final model achieves the best overall balance with LoRA rank/$\alpha=64/64$, $\mathrm{MHA}_{\max}=25$, $\lambda_{\mathrm{tumor}}=0.25$, and $\lambda_{\mathrm{stable}}=0.1$. These results demonstrate that compact pretrained rectified-flow
transformers can be adapted for contrast-enhanced MRI synthesis using parameter-efficient fine-tuning and task-aware regional losses.
Read the full abstract ▾
Computer Vision
SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering
Yaojun Hu, Danyang Tu, Yang Liu et al. (+8 more)
Volumetric medical VQA requires
reasoning over long and redundant 3D visual
token sequences, especially in multi-sequence MRI where complementary modalities provide diverse diagnostic cues but expose the decoder to many repeated anatomical regions. To investigate
reasoning under multi-sequence visual redundancy, we first introduce BreMRIs-VQA, a clinically curated breast MRI
benchmark with 1.19M QA pairs from 71.0K sequences and 12.9K patients, covering both free-text and multiple-choice questions. We further propose SeVeR, a selective visual exposure framework that compresses dense volumes into modality-wise prototypes and retrieves complementary multi-level evidence with change-aware gated
attention during decoding, trained with a marginal-utility self-consistency objective that suppresses unhelpful
retrieval. Experiments on BreMRIs-VQA and public
benchmarks show that SeVeR improves both discriminative and generative performance while exposing substantially fewer visual
tokens.
Read the full abstract ▾
All authors Yaojun Hu, Danyang Tu, Yang Liu, Jiajin Zhang, Wei Fang, Zhiqiang Liu, Chunlai Dong, Yingda Xia, Haochao Ying, Jian Wu, Ling Zhang
Computer Vision
Plans You Can Check: Verifier-Grounded Learning of an Open-Weight Planner for Executable Video-Editing
Haoyu Wang, Cheng Feng, Liuyang Bian et al. (+5 more)
Practical
video editing is not only pixel generation: an editor must turn a brief, a clip pool, music metadata, and hard constraints into an executable timeline. We study this decision layer as executable
video-editing planning and introduce RefineCut, which, unlike workflow systems that wrap a prompted frontier model, trains a compact open-weight planner for it. The planner edits a typed timeline through structured patches covering clip selection, trimming, ordering, transitions, and duration and music
alignment; a deterministic verifier applies each patch and checks it against an explicit constraint ledger. Because editing has no single ground-truth repair, we do not imitate teachers directly: RefineCut replays every multi-teacher branch through the verifier and keeps verifier-best repairs as supervision. A second stage, RefineCut-Evo, lets the student score its own repairs with the verifier and a task rubric and trains on high-margin preference pairs, so the final $8$B planner runs in a closed verifier loop with no teacher calls at inference. On RefineCut-Bench ($3{,}578$ tasks, $7{,}971$ captioned clips, $499$ music tracks, explicit ledgers), verifier-replayed
distillation lifts the planner from $0.620$ to $0.858$ on the protocol-specific
Video-Editing Score and RefineCut-Evo reaches $0.924$; the gain transfers to Llama-3.1-8B and GLM-4-9B, and in the same closed loop the $8$B planner matches or exceeds its frontier teachers. Code and RefineCut-Bench are publicly released; see the Data Availability statement.
Read the full abstract ▾
All authors Haoyu Wang, Cheng Feng, Liuyang Bian, Ruiyang Huang, Lei Wei, Yafei Wen, Xiaoxin Chen, Xiaoying Tang
Computer Vision
On the Separation of Human and AI-Generated Images in CLIP Embedding Space
Andrea Asperti
We identify a previously unreported phenomenon in CLIP representations: human and AI-generated paintings spontaneously separate along the dominant principal directions of their joint
embedding distribution, without any supervised objective designed to distinguish the two classes. Rather than exploiting this phenomenon for detection, our objective is to interpret it: we seek to identify the visual information underlying the separation and to trace it back from the
embedding space to the image domain. We pursue this objective through a progressive investigation combining interpretable image representations with gradient-based inversion, used systematically as an experimental probe of the relationships identified in feature space. Robustness experiments and increasingly expressive statistical descriptors progressively rule out several intuitive explanations based on global image properties and simple local statistics, and point instead to distributed multiscale image structure. Multiscale scattering provides the most informative interpretable representation considered, but offers only a partial account of the phenomenon. Direct inversion provides a complementary and striking observation: substantial displacements along the dominant CLIP directions can be induced by image perturbations that remain nearly imperceptible to human observers, showing that the directions involved in the separation are highly sensitive to image variations with very low perceptual salience for humans. Taken together, these results reveal a significant difference between the visual evidence reflected in CLIP representations and that readily accessible to human perception, raising broader questions about the relationship between artificial and human
vision and, ultimately, between artificial and human aesthetic judgment.
Read the full abstract ▾