Computer Vision
AnyThermal: Towards Learning Universal Representations for Thermal Perception
Parv Maheshwari, Jay Karhade, Yogesh Chawla et al. (+8 more)
We present AnyThermal, a thermal backbone that captures robust task-agnostic thermal features suitable for a variety of tasks such as cross-modal place recognition, thermal
segmentation, and
monocular depth estimation using thermal images. Existing thermal backbones that follow task-specific training from small-scale data result in utility limited to a specific environment and task. Unlike prior methods, AnyThermal can be used for a wide range of environments (indoor, aerial, off-road, urban) and tasks, all without task-specific training. Our key insight is to distill the feature representations from visual foundation models such as DINOv2 into a thermal encoder using thermal data from these multiple environments. To bridge the diversity gap of the existing RGB-Thermal
datasets, we introduce the TartanRGBT platform, the first open-source data collection platform with synced RGB-Thermal image acquisition. We use this payload to collect the TartanRGBT
dataset - a diverse and balanced
dataset collected in 4 environments. We demonstrate the efficacy of AnyThermal and TartanRGBT, achieving state-of-the-art results with improvements of up to 36% across diverse environments and downstream tasks on existing
datasets.
Read the full abstract ▾
All authors Parv Maheshwari, Jay Karhade, Yogesh Chawla, Isaiah Adu, Florian Heisen, Andrew Porco, Andrew Jong, Yifei Liu, Santosh Pitla, Sebastian Scherer, Wenshan Wang
Computer Vision
DeDPO: Debiased Direct Preference Optimization for Diffusion Models
Khiem Pham, Quang Nguyen, Tung Nguyen et al. (+4 more)
Direct Preference Optimization (DPO) has emerged as a predominant
alignment method for
diffusion models, facilitating off-policy training without explicit reward modeling. However, its reliance on large-scale, high-quality human preference labels presents a severe cost and scalability bottleneck. To overcome this, We propose a semi-supervised framework augmenting limited human data with a large corpus of unlabeled pairs annotated via cost-effective synthetic AI feedback. Our paper introduces Debiased DPO (DeDPO), which uniquely integrates a debiased estimation technique from causal inference into the DPO objective. By explicitly identifying and correcting the systematic bias and noise inherent in synthetic annotators, DeDPO ensures robust learning from imperfect feedback sources, including self-training and
Vision-
Language Models (VLMs). Experiments demonstrate that DeDPO is robust to the variations in synthetic labeling methods, achieving performance that matches and occasionally exceeds the theoretical upper bound of models trained on fully human-labeled data. This establishes DeDPO as a scalable solution for human-AI
alignment using inexpensive synthetic supervision.
Read the full abstract ▾
All authors Khiem Pham, Quang Nguyen, Tung Nguyen, Jingsen Zhu, Michele Santacatterina, Dimitris Metaxas, Ramin Zabih
Computer Vision
PhenoLIP: Integrating Phenotype Ontology Knowledge into Medical Vision-Language Pretraining
Cheng Liang, Chaoyi Wu, Weike Zhao et al. (+3 more)
Recent progress in large-scale CLIP-like
vision-
language models(VLMs) has greatly advanced medical image analysis. However, most existing medical VLMs still rely on coarse image-text contrastive objectives and fail to capture the systematic visual knowledge encoded in well-defined medical phenotype ontologies. To address this gap, we construct PhenoKG, the first large-scale, phenotype-centric
multimodal knowledge graph that encompasses over 520K high-quality image-text pairs linked to more than 3,000 phenotypes. Building upon PhenoKG, we propose PhenoLIP, a novel pretraining framework that explicitly incorporates structured phenotype knowledge into medical VLMs through a two-stage process. We first learn a knowledge-enhanced phenotype
embedding space from textual ontology data and then distill this structured knowledge into
multimodal pretraining via a teacher-guided knowledge
distillation objective. To support evaluation, we further introduce PhenoBench, an expert-verified
benchmark designed for phenotype recognition, comprising over 7,800 image–caption pairs covering more than 1,000 phenotypes. Extensive experiments demonstrate that PhenoLIP outperforms previous state-of-the-art baselines, improving upon BiomedCLIP in phenotype classification accuracy by 8.85% and BIOMEDICA in cross-modal
retrieval by 15.03%, underscoring the value of integrating phenotype-centric priors into medical VLMs for structured and interpretable medical image understanding.
Read the full abstract ▾
All authors Cheng Liang, Chaoyi Wu, Weike Zhao, Ya Zhang, Yanfeng Wang, Weidi Xie
Computer Vision
Unsupervised Anomaly Detection of Diseases in the Female Pelvis for Real-Time MR Imaging
Anika Knupfer, Johanna P. Müller, Jordina A. Verdera et al. (+9 more)
Pelvic diseases in women of reproductive age represent a major global health burden, with diagnosis frequently delayed due to high anatomical variability, complicating MRI interpretation. Existing AI approaches are largely disease-specific and lack real-time compatibility, limiting generalizability and clinical integration. To address these challenges, we establish a
benchmark framework for disease- and parameter-agnostic, real-time-compatible unsupervised anomaly detection in pelvic MRI. The method uses a residual variational autoencoder trained exclusively on healthy sagittal T2-weighted scans acquired across diverse imaging protocols to model normal pelvic anatomy. During inference, reconstruction error heatmaps indicate deviations from learned healthy structure, enabling detection of pathological regions without labeled abnormal data. The model is trained on 294 healthy scans and augmented with
diffusion-generated synthetic data to improve robustness. Quantitative evaluation on the publicly available Uterine Myoma MRI
Dataset yields an average area-under-the-curve (AUC) value of 0.736, with 0.828 sensitivity and 0.692 specificity. Additional inter-observer clinical evaluation extends analysis to endometrial cancer, endometriosis, and adenomyosis, revealing the influence of anatomical heterogeneity and inter-observer variability on performance interpretation. With a reconstruction time of approximately 92.6 frames per second, the proposed framework establishes a baseline for unsupervised anomaly detection in the female pelvis and supports future integration into real-time MRI. Code is available upon request (https://github.com/AniKnu/UADPelvis), prospective data sets are available for academic collaboration.
Read the full abstract ▾
All authors Anika Knupfer, Johanna P. Müller, Jordina A. Verdera, Martin Fenske, Claudius S. Mathy, Smiti Tripathy, Sebastian Arndt, Matthias May, Michael Uder, Matthias W. Beckmann, Stefanie Burghaus, Jana Hutter
Computer Vision
M3: High-fidelity Text-to-Image Generation via Multi-Modal, Multi-Agent and Multi-Round Visual Reasoning
Bangji Yang, Ruihan Guo, Jiajun Fan et al. (+2 more)
Generative models have achieved impressive fidelity in text-to-image synthesis, yet struggle with complex compositional
prompts involving multiple constraints. We introduce M3 (Multi-Modal, Multi-
Agent, Multi-Round), a training-free framework that systematically resolves these failures through iterative inference-time refinement. M3 orchestrates off-the-shelf foundation models in a robust multi-
agent loop: a Planner decomposes
prompts into verifiable checklists, while specialized Checker, Refiner, and Editor
agents surgically correct constraints one at a time, with a Verifier ensuring monotonic improvement. Applied to open-source models, M3 achieves remarkable results on the challenging OneIG-EN
benchmark, with our Qwen-Image+M3 surpassing commercial flagship systems including Imagen4 (0.515) and Seedream 3.0 (0.530), reaching state-of-the-art performance (0.532 overall). This demonstrates that intelligent multi-
agent reasoning can elevate open-source models beyond proprietary alternatives. M3 also substantially improves GenEval compositional metrics, effectively doubling spatial
reasoning performance on hardened test sets. As a plug-and-play module compatible with any pre-trained T2I model, M3 establishes a new paradigm for compositional generation without costly retraining.
Read the full abstract ▾
All authors Bangji Yang, Ruihan Guo, Jiajun Fan, Chaoran Cheng, Ge Liu
Computer Vision
MetaSSP: Enhancing Semi-supervised Implicit 3D Reconstruction through Meta-adaptive EMA and SDF-aware Pseudo-label Evaluation
Luoxi Zhang, Chun Xie and Itaru Kitahara
Implicit SDF-based methods for single-view 3D reconstruction achieve high-quality surfaces but require large labeled
datasets, limiting their scalability. We propose MetaSSP, a novel semi-supervised framework that exploits abundant unlabeled images. Our approach introduces gradient-based parameter importance estimation to regularize adaptive EMA updates and an SDF-aware pseudo-label weighting mechanism combining augmentation consistency with SDF variance. Beginning with a 10% supervised warm-up, the unified pipeline jointly refines labeled and unlabeled data. On the Pix3D
benchmark, our method reduces Chamfer Distance by approximately 20.61% and increases IoU by around 24.09% compared to existing semi-supervised baselines, setting a new state of the art.
Read the full abstract ▾
Computer Vision
Driving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Driving
Xuyang Chen, Conglang Zhang, Chuanheng Fu et al. (+9 more)
Driven by the emergence of Controllable
Video Diffusion, existing Sim2Real methods for autonomous driving
video generation typically rely on explicit intermediate representations to bridge the domain gap. However, these modalities face a fundamental Consistency-Realism Dilemma. Low-level signals (e.g., edges, blurred images) ensure precise control but compromise realism by "baking in" synthetic artifacts, whereas high-level priors (e.g., depth, semantics, HDMaps) facilitate photorealism but lack the structural detail required for consistent guidance. In this work, we present Driving with DINO (DwD), a novel framework that leverages
Vision Foundation Module (VFM) features as a unified bridge between the simulation and real-world domains. We first identify that these features encode a spectrum of information, from high-level semantics to fine-grained structure. To effectively utilize this, we employ Principal Subspace Projection to discard the high-frequency elements responsible for "texture baking," while concurrently introducing Random Channel Tail Drop to mitigate the structural loss inherent in rigid dimensionality reduction, thereby reconciling realism with control consistency. Furthermore, to fully leverage DINOv3's high-resolution capabilities for enhancing control precision, we introduce a learnable Spatial
Alignment Module that adapts these high-resolution features to the
diffusion backbone. Finally, we propose a Causal Temporal Aggregator employing causal convolutions to explicitly preserve historical motion context when integrating frame-wise DINO features, which effectively mitigates motion blur and guarantees temporal stability. Project page: https://albertchen98.github.io/DwD-project/
Read the full abstract ▾
All authors Xuyang Chen, Conglang Zhang, Chuanheng Fu, Zihao Yang, Kaixuan Zhou, Yizhi Zhang, Yanfeng Zhang, Mingwei Sun, Zhen Dong, Xiaoxiao Long, Zengmao Wang, Liqiu Meng
Computer Vision
MGP-KAD: Multimodal Geometric Priors and Kolmogorov-Arnold Decoder for Single-View 3D Reconstruction in Complex Scenes
Luoxi Zhang, Chun Xie and Itaru Kitahara
Single-view 3D reconstruction in complex real-world scenes is challenging due to noise, object diversity, and limited
dataset availability. To address these challenges, we propose MGP-KAD, a novel
multimodal feature fusion framework that integrates RGB and
geometric prior to enhance reconstruction accuracy. The
geometric prior is generated by sampling and clustering ground-truth object data, producing class-level features that dynamically adjust during training to improve
geometric understanding. Additionally, we introduce a hybrid decoder based on Kolmogorov-Arnold Networks (KAN) to overcome the limitations of traditional linear decoders in processing complex
multimodal inputs. Extensive experiments on the Pix3D
dataset demonstrate that MGP-KAD achieves state-of-the-art (SOTA) performance, significantly improving
geometric integrity, smoothness, and detail preservation. Our work provides a robust and effective solution for advancing single-view 3D reconstruction in complex scenes.
Read the full abstract ▾
Computer Vision
EgoAVU: Egocentric Audio-Visual Understanding
Ashish Seth, Xinhao Mei, Changsheng Zhao et al. (+9 more)
Understanding
egocentric videos plays a vital role for
embodied intelligence. Recent multi-modal large
language models (MLLMs) can accept both visual and audio inputs. However, due to the challenge of obtaining text labels with coherent joint-modality information, whether MLLMs can jointly understand both modalities in
egocentric videos remains under-explored. To address this problem, we introduce EgoAVU, a scalable data engine to automatically generate
egocentric audio-visual narrations, questions, and answers. EgoAVU enriches human narrations with
multimodal context and generates audio-visual narrations through cross-modal correlation modeling.
Token-based
video filtering and modular, graph-based curation ensure both data diversity and quality. Leveraging EgoAVU, we construct EgoAVU-Instruct, a large-scale training
dataset of 3M samples, and EgoAVU-Bench, a manually verified evaluation split covering diverse tasks. EgoAVU-Bench clearly reveals the limitations of existing MLLMs: they bias heavily toward visual signals, often neglecting audio cues or failing to correspond audio with the visual source. Finetuning MLLMs on EgoAVU-Instruct effectively addresses this issue, enabling up to 113% performance improvement on EgoAVU-Bench. Such benefits also transfer to other
benchmarks such as EgoTempo and EgoIllusion, achieving up to 28% relative performance gain. Code will be released to the community.
Read the full abstract ▾
All authors Ashish Seth, Xinhao Mei, Changsheng Zhao, Varun Nagaraja, Ernie Chang, Gregory P. Meyer, Gael Le Lan, Yunyang Xiong, Vikas Chandra, Yangyang Shi, Dinesh Manocha, Zhipeng Cai
Computer Vision · Cross-listed from cs.LG
Tempora: Characterising the Time-Contingent Utility of Online Test-Time Adaptation
Sudarshan Sreeram, Young D. Kwon and Cecilia Mascolo
Test-time adaptation (TTA) offers a compelling remedy for machine learning (ML) models that degrade under domain shifts, improving generalisation on-the-fly with only unlabelled samples. This flexibility suits real deployments, yet conventional evaluations unrealistically assume unbounded processing time, overlooking the accuracy-latency trade-off. As ML increasingly underpins latency-sensitive and user-facing use-cases, temporal pressure constrains the viability of adaptable inference; predictions arriving too late to act on are futile. We introduce Tempora, a framework for evaluating TTA under this pressure. It consists of temporal scenarios that model deployment constraints, evaluation protocols that operationalise measurement, and time-contingent utility metrics that quantify the accuracy-latency trade-off. We instantiate the framework with three such metrics: (1) discrete utility for asynchronous streams with hard deadlines, (2) continuous utility for interactive settings where value decays with latency, and (3) amortised utility for budget-constrained deployments. By applying Tempora to 11 TTA methods, we find that rank instability persists across 750+ temporal evaluations spanning diverse
datasets, models, and hardware platforms; i.e., conventional rankings do not predict rankings under temporal pressure. The highest-utility method varies with the shift and temporal pressure, with no clear winner. By enabling systematic evaluation across diverse temporal constraints for the first time, Tempora reveals when and why rankings change, offering practitioners a lens for method selection and researchers a target for deployable adaptation. Code: https://github.com/sudotensor/tempora.
Read the full abstract ▾
Computer Vision
From Blurry to Believable: Enhancing Low-quality Talking Heads with 3D Generative Priors
Ding-Jiun Huang, Yuanhao Wang, Shao-Ji Yuan et al. (+4 more)
Creating high-fidelity, animatable 3D talking heads is crucial for immersive applications, yet often hindered by the prevalence of low-quality image or
video sources, which yield poor 3D reconstructions. In this paper, we introduce SuperHead, a novel framework for enhancing low-resolution, animatable 3D head avatars. The core challenge lies in synthesizing high-quality geometry and textures, while ensuring both 3D and temporal consistency during animation and preserving subject identity. Despite recent progress in image,
video and 3D-based super-resolution (SR), existing SR techniques are ill-equipped to handle dynamic 3D inputs. To address this, SuperHead leverages the rich priors from pre-trained 3D generative models via a novel dynamics-aware 3D inversion scheme. This process optimizes the latent representation of the generative model to produce a super-resolved 3D Gaussian Splatting (3DGS) head model, which is subsequently rigged to an underlying parametric head model (e.g., FLAME) for animation. The inversion is jointly supervised using a sparse collection of upscaled 2D face renderings and corresponding depth maps, captured from diverse facial expressions and camera viewpoints, to ensure realism under dynamic facial motions. Experiments demonstrate that SuperHead generates avatars with fine-grained facial details under dynamic motions, significantly outperforming baseline methods in visual quality.
Read the full abstract ▾
All authors Ding-Jiun Huang, Yuanhao Wang, Shao-Ji Yuan, Albert Mosella-Montoro, Francisco Vicente Carrasco, Cheng Zhang, Fernando De la Torre
Computer Vision · Cross-listed from eess.IV
ALIEN: Analytic Latent Watermarking for Controllable Generation
Liangqi Lei, Keke Gai, Jing Yu et al. (+1 more)
Watermarking is a technical alternative to safeguarding intellectual property and reducing misuse. Existing methods focus on optimizing watermarked latent variables to balance watermark robustness and fidelity, as Latent
diffusion models (LDMs) are considered a powerful tool for generative tasks. However, reliance on computationally intensive heuristic optimization for iterative signal refinement results in high training overhead and local optima entrapment.To address these issues, we propose an \underline{A}na\underline{l}ytical Watermark\underline{i}ng Framework for Controllabl\underline{e} Generatio\underline{n} (ALIEN). We develop the first analytical derivation of the time-dependent modulation coefficient that guides the
diffusion of watermark residuals to achieve controllable watermark
embedding pattern.Experimental results show that ALIEN-Q outperforms the state-of-the-art by 33.1% across 5 quality metrics, and ALIEN-R demonstrates 14.0% improved robustness against generative variant and stability threats compared to the state-of-the-art across 15 distinct conditions. Code can be available at https://anonymous.4open.science/r/ALIEN/.
Read the full abstract ▾
All authors Liangqi Lei, Keke Gai, Jing Yu, Qi Wu
Computer Vision · Cross-listed from cs.SE
SVRepair: Structured Visual Reasoning for Automated Program Repair
Jincheng Wang, Liwei Luo, Xiaoxuan Tang et al. (+5 more)
Large
language models (LLMs) have recently been applied to Automated Program Repair (APR), yet most existing approaches remain unimodal and fail to use diagnostic signals contained in visual artifacts such as screenshots and control-flow graphs. In practice, many bug reports convey critical information visually (e.g., layout breakage or missing widgets), but directly using such dense visual inputs often causes context loss and noise, making it difficult for MLLMs to ground visual observations into precise fault localization and executable patches. To bridge this semantic gap, we propose SVRepair, a
multimodal APR framework with Structured Visual Representation (SVR). SVRepair first fine-tunes a
vision-
language model, SVR, to uniformly transform heterogeneous visual artifacts into a semantic scene graph that captures GUI elements and their structural relations (e.g., hierarchy), providing normalized, code-relevant context for downstream repair. Building on the graph, SVRepair drives a coding
agent to localize faults and synthesize patches, and further introduces an iterative visual-artifact
segmentation strategy that progressively narrows the input to bug-centered regions to suppress irrelevant context and reduce
hallucinations. Across primary repository-level APR
benchmarks, SVRepair resolves 186/517 SWE-Bench M instances (35.98% over all instances; 36.47% over submitted runs) and 4/19 visual OmniGIRL instances (21.05%). On supplementary structured
multimodal code
reasoning benchmarks, SVRepair reaches 38.02% on MMCode and 95.73% on CodeVision. Code is available at https://github.com/codefuse-ai/CodeFuse-SVR.
Read the full abstract ▾
All authors Jincheng Wang, Liwei Luo, Xiaoxuan Tang, Jingxuan Xu, Sheng Zhou, Dajun Chen, Wei Jiang, Yong Li
Computer Vision · Cross-listed from cs.LG
Shared LoRA Subspaces for almost Strict Continual Learning
Prakhar Kaushik, Ankit Vaidya, Shravan Chaudhari et al. (+2 more)
Adapting large pretrained models to new tasks efficiently and continually is crucial for real-world deployment but remains challenging due to catastrophic forgetting and the high cost of retraining. While parameter-efficient tuning methods like low rank adaptation (LoRA) reduce computational demands, they lack mechanisms for strict continual learning and knowledge integration, without relying on data replay, or multiple adapters. We propose Share, a novel approach to parameter efficient continual finetuning that learns and dynamically updates a single, shared low-rank subspace, enabling seamless adaptation across multiple tasks and modalities. Share constructs a foundational subspace that extracts core knowledge from past tasks and incrementally integrates new information by identifying essential subspace directions. Knowledge from each new task is incorporated into this evolving subspace, facilitating forward knowledge transfer, while minimizing catastrophic interference. This approach achieves up to 100x parameter reduction and 281x
memory savings over traditional LoRA methods, maintaining performance comparable to jointly trained models. A single Share model can replace hundreds of task-specific LoRA adapters, supporting scalable, asynchronous continual learning. Experiments across image classification, natural
language understanding, 3D pose estimation, and text-to-image generation validate its effectiveness, making Share a practical and scalable solution for lifelong learning in large-scale AI systems.
Read the full abstract ▾
All authors Prakhar Kaushik, Ankit Vaidya, Shravan Chaudhari, Rama Chellappa, Alan Yuille
Computer Vision · Cross-listed from cs.LG
Pseudo-Invertible Neural Networks
Yamit Ehrlich, Nimrod Berman and Assaf Shocher
The Moore-Penrose Pseudo-inverse (PInv) serves as the fundamental solution for linear systems. In this paper, we propose a natural generalization of PInv to the nonlinear regime in general and to neural networks in particular. We introduce Surjective Pseudo-invertible Neural Networks (SPNN), a class of architectures explicitly designed to admit a tractable non-linear PInv. The proposed non-linear PInv and its implementation in SPNN satisfy fundamental
geometric properties. One such property is null-space projection or "Back-Projection", $x' = x + A^\dagger(y-Ax)$, which moves a sample $x$ to its closest consistent state $x'$ satisfying $Ax=y$. We formalize Non-Linear Back-Projection (NLBP), a method that guarantees the same consistency constraint for non-linear mappings $f(x)=y$ via our defined PInv. We leverage SPNNs to expand the scope of zero-shot inverse problems.
Diffusion-based null-space projection has revolutionized zero-shot solving for linear inverse problems by exploiting closed-form back-projection. We extend this method to non-linear degradations. Here, "degradation" is broadly generalized to include any non-linear loss of information, spanning from optical distortions to semantic abstractions like classification. This approach enables zero-shot inversion of complex degradations and allows precise semantic control over generative outputs without retraining the
diffusion prior.
Read the full abstract ▾
Computer Vision
Predicting Camera Pose from Perspective Descriptions for Spatial Reasoning
Xuejun Zhang, Aditi Tiwari, Zhenhailong Wang et al. (+1 more)
Multi-image spatial
reasoning remains challenging for current
multimodal large
language models (MLLMs). While single-view perception is inherently 2D,
reasoning over multiple views requires building a coherent scene understanding across viewpoints. In particular, we study perspective taking, where a model must build a coherent 3D understanding from multi-view observations and use it to reason from a new,
language-specified viewpoint. We introduce CAMCUE, a pose-aware multi-image framework that uses camera pose as an explicit
geometric anchor for cross-view fusion and novel-view
reasoning. CAMCUE injects per-view pose into visual
tokens, grounds natural-
language viewpoint descriptions to a target camera pose, and synthesizes a pose-conditioned imagined target view to support answering. To support this setting, we curate CAMCUE-DATA with 27,668 training and 508 test instances pairing multi-view images and poses with diverse target-viewpoint descriptions and perspective-shift questions. We also include human-annotated viewpoint descriptions in the test split to evaluate generalization to human
language. CAMCUE improves overall accuracy by 9.06% and predicts target poses from natural-
language viewpoint descriptions with over 90% rotation accuracy within 20{\deg} and translation accuracy within a 0.5 error threshold. This direct grounding avoids expensive test-time search-and-match, reducing inference time from 256.6s to 1.45s per example and enabling fast, interactive use in real-world scenarios.
Read the full abstract ▾
All authors Xuejun Zhang, Aditi Tiwari, Zhenhailong Wang, Heng Ji
Computer Vision
SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMs
Jintao Tong, Shilin Yan, Hongwei Xue et al. (+5 more)
Multimodal Large
Language Models (MLLMs) have made remarkable progress in
multimodal perception and
reasoning by bridging
vision and
language. However, most existing MLLMs perform
reasoning primarily with textual CoT, which limits their effectiveness on
vision-intensive tasks. Recent approaches inject a fixed number of continuous hidden states as "visual thoughts" into the
reasoning process and improve visual performance, but often at the cost of degraded text-based logical
reasoning. We argue that the core limitation lies in a rigid, pre-defined
reasoning pattern that cannot adaptively choose the most suitable thinking modality for different user queries. We introduce SwimBird, a
reasoning-switchable MLLM that dynamically switches among three
reasoning modes conditioned on the input: (1) text-only
reasoning, (2)
vision-only
reasoning (continuous hidden states as visual thoughts), and (3) interleaved
vision-text
reasoning. To enable this capability, we adopt a hybrid autoregressive formulation that unifies next-
token prediction for textual thoughts with next-
embedding prediction for visual thoughts, and design a systematic
reasoning-mode curation strategy to construct SwimBird-SFT-92K, a diverse supervised fine-tuning
dataset covering all three
reasoning patterns. By enabling flexible, query-adaptive mode selection, SwimBird preserves strong textual logic while substantially improving performance on
vision-dense tasks. Experiments across diverse
benchmarks covering textual
reasoning and challenging visual understanding demonstrate that SwimBird achieves state-of-the-art results and robust gains over prior fixed-pattern
multimodal reasoning methods.
Read the full abstract ▾
All authors Jintao Tong, Shilin Yan, Hongwei Xue, Xiaojun Tang, Kunyu Shi, Guannan Zhang, Ruixuan Li, Yixiong Zou
Computer Vision · Cross-listed from cs.RO
CommCP: Efficient Multi-Agent Coordination via LLM-Based Communication with Conformal Prediction
Xiaopan Zhang, Zejin Wang, Zhixu Li et al. (+2 more)
To complete assignments provided by humans in natural
language,
robots must interpret commands, generate and answer relevant questions for scene understanding, and manipulate target objects. Real-world deployments often require multiple heterogeneous
robots with different manipulation capabilities to handle different assignments cooperatively. Beyond the need for specialized manipulation skills, effective information gathering is important in completing these assignments. To address this component of the problem, we formalize the information-gathering process in a fully cooperative setting as an underexplored multi-
agent multi-task
Embodied Question Answering (MM-EQA) problem, which is a novel extension of canonical
Embodied Question Answering (EQA), where effective communication is crucial for coordinating efforts without redundancy. To address this problem, we propose CommCP, a novel LLM-based decentralized communication framework designed for MM-EQA. Our framework employs conformal prediction to calibrate the generated messages, thereby minimizing receiver distractions and enhancing communication reliability. To evaluate our framework, we introduce an MM-EQA
benchmark featuring diverse, photo-realistic household scenarios with
embodied questions. Experimental results demonstrate that CommCP significantly enhances the task success rate and exploration efficiency over baselines. The experiment
videos, code, and
dataset are available on our project website: https://comm-cp.github.io.
Read the full abstract ▾
All authors Xiaopan Zhang, Zejin Wang, Zhixu Li, Jianpeng Yao, Jiachen Li
Computer Vision
Thinking with Geometry: Active Geometry Integration for Spatial Reasoning
Haoyuan Li, Qihang Cao, Tao Tang et al. (+6 more)
Recent progress in spatial
reasoning with
Multimodal Large
Language Models (MLLMs) increasingly leverages
geometric priors from 3D encoders. However, most existing integration strategies remain passive: geometry is exposed as a global stream and fused in an indiscriminate manner, which often induces semantic-geometry misalignment and redundant signals. We propose GeoThinker, a framework that shifts the paradigm from passive fusion to active perception. Instead of feature mixing, GeoThinker enables the model to selectively retrieve
geometric evidence conditioned on its internal
reasoning demands. GeoThinker achieves this through Spatial-Grounded Fusion applied at carefully selected VLM layers, where semantic visual priors selectively query and integrate task-relevant geometry via frame-strict cross-
attention, further calibrated by Importance Gating that biases per-frame
attention toward task-relevant structures. Comprehensive evaluation results show that GeoThinker sets a new state-of-the-art in spatial intelligence, achieving a peak score of 72.6 on the VSI-Bench. Furthermore, GeoThinker demonstrates robust generalization and significantly improved spatial perception across complex downstream scenarios, including
embodied referring and autonomous driving. Our results indicate that the ability to actively integrate spatial structures is essential for next-generation spatial intelligence. Code can be found at https://github.com/Li-Hao-yuan/GeoThinker.
Read the full abstract ▾
All authors Haoyuan Li, Qihang Cao, Tao Tang, Kun Xiang, Zihan Guo, Jianhua Han, JiaWang Bian, Hang Xu, Xiaodan Liang
Computer Vision
InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions
Sirui Xu, Samuel Schulter, Morteza Ziyadi et al. (+4 more)
Humans rarely plan whole-body interactions with objects at the level of explicit whole-body movements. High-level intentions, such as affordance, define the goal, while coordinated balance, contact, and manipulation can emerge naturally from underlying physical and motor priors. Scaling such priors is key to enabling
humanoids to compose and generalize loco-manipulation skills across diverse contexts while maintaining physically coherent whole-body coordination. To this end, we introduce InterPrior, a scalable framework that learns a unified generative controller through large-scale imitation pretraining and post-training by reinforcement learning. InterPrior first distills a full-reference imitation expert into a versatile, goal-conditioned variational policy that reconstructs motion from
multimodal observations and high-level intent. While the distilled policy reconstructs training behaviors, it does not generalize reliably due to the vast configuration space of large-scale human-object interactions. To address this, we apply data augmentation with physical perturbations, and then perform reinforcement learning finetuning to improve competence on unseen goals and initializations. Together, these steps consolidate the reconstructed latent skills into a valid
manifold, yielding a motion prior that generalizes beyond the training data, e.g., it can incorporate new behaviors such as interactions with unseen objects. We further demonstrate its effectiveness for user-interactive control and its potential for real
robot deployment.
Read the full abstract ▾
All authors Sirui Xu, Samuel Schulter, Morteza Ziyadi, Xialin He, Xiaohan Fei, Yu-Xiong Wang, Liangyan Gui
Computer Vision
Dongyang Chen, Chaoyang Wang, Dezhao Su et al. (+6 more)
Multimodal Large
Language Models (MLLMs) have recently been applied to universal
multimodal retrieval, where Chain-of-Thought (CoT)
reasoning improves candidate reranking. However, existing approaches remain largely
language-driven, relying on static visual encodings and lacking the ability to actively verify fine-grained visual evidence, which often leads to speculative
reasoning in visually ambiguous cases. We propose V-Retrver, an evidence-driven
retrieval framework that reformulates
multimodal retrieval as an agentic
reasoning process grounded in visual inspection. V-Retrver enables an MLLM to selectively acquire visual evidence during
reasoning via external visual tools, performing a
multimodal interleaved
reasoning process that alternates between hypothesis generation and targeted visual verification.To train such an evidence-gathering
retrieval agent, we adopt a curriculum-based learning strategy combining supervised
reasoning activation, rejection-based refinement, and reinforcement learning with an evidence-aligned objective. Experiments across multiple
multimodal retrieval benchmarks demonstrate consistent improvements in
retrieval accuracy (with 23.0% improvements on average), perception-driven
reasoning reliability, and generalization.
Read the full abstract ▾
All authors Dongyang Chen, Chaoyang Wang, Dezhao Su, Xi Xiao, Zeyu Zhang, Jing Xiong, Qing Li, Yuzhang Shang, Shichao Kan
Computer Vision
Splat and Distill: Augmenting Teachers with Feed-Forward 3D Reconstruction For 3D-Aware Distillation
David Shavin and Sagie Benaim
Vision Foundation Models (VFMs) have achieved remarkable success when applied to various downstream 2D tasks. Despite their effectiveness, they often exhibit a critical lack of 3D awareness. To this end, we introduce Splat and Distill, a framework that instills robust 3D awareness into 2D VFMs by augmenting the teacher model with a fast, feed-forward 3D reconstruction pipeline. Given 2D features produced by a teacher model, our method first lifts these features into an explicit 3D Gaussian representation, in a feedforward manner. These 3D features are then “splatted" onto novel viewpoints, producing a set of novel 2D feature maps used to supervise the student model, “distilling" geometrically grounded knowledge. By replacing slow per-scene optimization of prior work with our feed-forward lifting approach, our framework avoids feature-averaging artifacts, creating a dynamic learning process where the teacher's consistency improves alongside that of the student. We conduct a comprehensive evaluation on a suite of downstream tasks, including
monocular depth estimation, surface normal estimation, multi-view correspondence, and semantic
segmentation. Our method significantly outperforms prior works, not only achieving substantial gains in 3D awareness but also enhancing the underlying semantic richness of 2D features. Project page is available at https://davidshavin4.github.io/Splat-and-Distill/
Read the full abstract ▾
Computer Vision
Context Forcing: Consistent Autoregressive Video Generation with Long Context
Shuo Chen, Cong Wei, Sun Sun et al. (+5 more)
Recent approaches to real-time long
video generation typically employ streaming tuning strategies, attempting to train a long-context student using a short-context (memoryless) teacher. In these frameworks, the student performs long rollouts but receives supervision from a teacher limited to short 5-second windows. This structural discrepancy creates a critical student-teacher mismatch: the teacher's inability to access long-term history prevents it from guiding the student on global temporal dependencies, effectively capping the student's context length. To resolve this, we propose Context Forcing, a novel framework that trains a long-context student via a long-context teacher. By ensuring the teacher is aware of the full generation history, we eliminate the supervision mismatch, enabling the robust training of models capable of long-term consistency. To make this computationally feasible for extreme durations (e.g., 2 minutes), we introduce a context management system that transforms the linearly growing context into a Slow-Fast
Memory architecture, significantly reducing visual redundancy. Extensive results demonstrate that our method enables effective context lengths exceeding 20 seconds – 2 to 10 times longer than state-of-the-art methods like LongLive and Infinite-RoPE. By leveraging this extended context, Context Forcing preserves superior consistency across long durations, surpassing state-of-the-art baselines on various long
video evaluation metrics.
Read the full abstract ▾
All authors Shuo Chen, Cong Wei, Sun Sun, Ping Nie, Kai Zhou, Ge Zhang, Ming-Hsuan Yang, Wenhu Chen
Computer Vision
MambaVF: State Space Model for Efficient Video Fusion
Zixiang Zhao, Yukun Cui, Lilun Deng et al. (+4 more)
Video fusion aims to integrate complementary information from multiple source
videos while preserving temporal consistency. Effective modeling of temporal dynamics is essential to this goal, yet existing methods incur substantial computational overhead from optical flow estimation and feature warping. In this paper, we present MambaVF, an efficient
video fusion framework that uses state space model (SSM) to achieve temporal modeling without explicit motion estimation. First, by formulating
video fusion as a sequential state update process, MambaVF captures long-range temporal dependencies with linear complexity, significantly reducing computation and
memory costs. Second, the lightweight SSM-based fusion module eliminates conventional flow-guided
alignment. Instead, it introduces a mutual state fusion module and a spatio-temporal bidirectional scanning mechanism to enable information aggregation across
video streams. Experiments on multiple
benchmarks confirm that MambaVF reaches state-of-the-art performance in different
video fusion applications (multi-exposure, multi-focus, infrared-visible, medical), while reducing parameters by >90% and FLOPs by >80%, resulting in >50% shorter runtime. Project page: https://mambavf.github.io
Read the full abstract ▾
All authors Zixiang Zhao, Yukun Cui, Lilun Deng, Haowen Bai, Haotong Qin, Tao Feng, Konrad Schindler
Computer Vision
GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks?
Ruihang Li, Leigang Qu, Jingxu Zhang et al. (+6 more)
The rapid advancement of visual generation models has outpaced traditional evaluation approaches, necessitating the adoption of
Vision-
Language Models as surrogate judges. In this work, we systematically investigate the reliability of the prevailing absolute pointwise scoring standard, across a wide spectrum of visual generation tasks. Our analysis reveals that this paradigm is limited due to
stochastic inconsistency and poor
alignment with human perception. To resolve these limitations, we introduce GenArena, a unified evaluation framework that leverages a pairwise comparison paradigm to ensure stable and human-aligned evaluation. Crucially, our experiments uncover a transformative finding that simply adopting this pairwise protocol enables off-the-shelf open-source models to outperform top-tier proprietary models. Notably, our method boosts evaluation accuracy by over 20% and achieves a Spearman correlation of 0.86 with the authoritative LMArena leaderboard, drastically surpassing the 0.36 correlation of pointwise methods. Based on GenArena, we
benchmark state-of-the-art visual generation models across diverse tasks, providing the community with a rigorous and automated evaluation standard for visual generation.
Read the full abstract ▾
All authors Ruihang Li, Leigang Qu, Jingxu Zhang, Dongnan Gui, Mengde Xu, Xiaosong Zhang, Han Hu, Wenjie Wang, Jiaqi Wang
Computer Vision
VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generation
Jie Deng, Kaichun Yao and Libo Zhang
Screenshot-to-code generation aims to translate user interface screenshots into executable frontend code that faithfully reproduces the target layout and style. Existing
multimodal large
language models perform this mapping directly from screenshots but are trained without observing the visual outcomes of their generated code. In contrast, human developers iteratively render their implementation, compare it with the design, and learn how visual differences relate to code changes. Inspired by this process, we propose VisRefiner, a training framework that enables models to learn from visual differences between rendered predictions and reference designs. We construct difference-aligned supervision that associates visual discrepancies with corresponding code edits, allowing the model to understand how appearance variations arise from implementation changes. Building on this, we introduce a reinforcement learning stage for self-refinement, where the model improves its generated code by observing both the rendered output and the target design, identifying their visual differences, and updating the code accordingly. Experiments show that VisRefiner substantially improves single-step generation quality and layout fidelity, while also endowing models with strong self-refinement ability. These results demonstrate the effectiveness of learning from visual differences for advancing screenshot-to-code generation.
Read the full abstract ▾
Computer Vision
RISE-Video: Can Video Generators Decode Implicit World Rules?
Mingxin Liu, Shuran Ma, Shibei Meng et al. (+9 more)
While generative
video models have achieved remarkable visual fidelity, their capacity to internalize and reason over implicit world rules remains a critical yet under-explored frontier. To bridge this gap, we present RISE-
Video, a pioneering
reasoning-oriented
benchmark for Text-Image-to-
Video (TI2V) synthesis that shifts the evaluative focus from surface-level aesthetics to deep cognitive
reasoning. RISE-
Video comprises 467 meticulously human-annotated samples spanning eight rigorous categories, providing a structured testbed for probing model intelligence across diverse dimensions, ranging from commonsense and spatial dynamics to specialized subject domains. Our framework introduces a multi-dimensional evaluation protocol consisting of four metrics:
Reasoning Alignment, Temporal Consistency, Physical Rationality, and Visual Quality. To further support scalable evaluation, we propose an automated pipeline leveraging Large
Multimodal Models (LMMs) to emulate human-centric assessment. Extensive experiments on 11 state-of-the-art TI2V models reveal pervasive deficiencies in simulating complex scenarios under implicit constraints, offering critical insights for the advancement of future world-simulating generative models.
Read the full abstract ▾
All authors Mingxin Liu, Shuran Ma, Shibei Meng, Xiangyu Zhao, Zicheng Zhang, Shaofeng Zhang, Zhihang Zhong, Peixian Chen, Haoyu Cao, Xing Sun, Haodong Duan, Xue Yang
Computer Vision
LSA: Localized Semantic Alignment for Enhancing Temporal Consistency in Traffic Video Generation
Mirlan Karimov, Teodora Spasojevic, Markus Braun et al. (+3 more)
Controllable
video generation has emerged as a versatile tool for autonomous driving, enabling realistic synthesis of traffic scenarios. However, existing methods depend on control signals at inference time to guide the generative model towards temporally consistent generation of dynamic objects, limiting their utility as scalable and generalizable data engines. In this work, we propose Localized Semantic
Alignment (LSA), a simple yet effective framework for fine-tuning pre-trained
video generation models. LSA enhances temporal consistency by aligning semantic features between ground-truth and generated
video clips. Specifically, we compare the output of an off-the-shelf feature extraction model between the ground-truth and generated
video clips localized around dynamic objects inducing a semantic feature consistency loss. We fine-tune the base model by combining this loss with the standard
diffusion loss. The model fine-tuned for a single epoch with our novel loss outperforms the baselines in common
video generation evaluation metrics. To further test the temporal consistency in generated
videos we adapt two additional metrics from object detection task, namely mAP and mIoU. Extensive experiments on nuScenes and KITTI
datasets show the effectiveness of our approach in enhancing temporal consistency in
video generation without the need for external control signals during inference and any computational overheads.
Read the full abstract ▾
All authors Mirlan Karimov, Teodora Spasojevic, Markus Braun, Julian Wiederer, Vasileios Belagiannis, Marc Pollefeys
Computer Vision
Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matching
Junwan Kim, Jiho Park, Seonghu Jeon et al. (+1 more)
Flow matching has recently emerged as a promising alternative to
diffusion-based generative models, particularly for text-to-image generation. Despite its flexibility in allowing arbitrary source distributions, most existing approaches rely on a standard Gaussian distribution, a choice inherited from
diffusion models, and rarely consider the source distribution itself as an optimization target in such settings. In this work, we show that principled design of the source distribution is not only feasible but also beneficial at the scale of modern text-to-image systems. Specifically, we propose learning a condition-dependent source distribution under flow matching objective that better exploit rich conditioning signals. We identify key failure modes that arise when directly incorporating conditioning into the source, including distributional collapse and instability, and show that appropriate variance regularization and directional
alignment between source and target are critical for stable and effective learning. We further analyze how the choice of target representation space impacts flow matching with structured sources, revealing regimes in which such designs are most effective. Extensive experiments across multiple text-to-image
benchmarks demonstrate consistent and robust improvements, including up to a 3x faster convergence in FID, highlighting the practical benefits of a principled source distribution design for conditional flow matching.
Read the full abstract ▾
All authors Junwan Kim, Jiho Park, Seonghu Jeon, Seungryong Kim
Computer Vision
Multi-Scale Global-Instance Prompt Tuning for Continual Test-time Adaptation in Medical Image Segmentation
Lingrui Li, Yanfeng Zhou, Nan Pu et al. (+2 more)
Distribution shift is a common challenge in medical images obtained from different clinical centers, significantly hindering the deployment of pre-trained semantic
segmentation models in real-world applications across multiple domains. Continual Test-Time Adaptation(CTTA) has emerged as a promising approach to address cross-domain shifts during continually evolving target domains. Most existing CTTA methods rely on incrementally updating model parameters, which inevitably suffer from error accumulation and catastrophic forgetting, especially in long-term adaptation. Recent
prompt-tuning-based works have shown potential to mitigate the two issues above by updating only visual
prompts. While these approaches have demonstrated promising performance, several limitations remain:1)lacking multi-scale
prompt diversity, 2)inadequate incorporation of instance-specific knowledge, and 3)risk of
privacy leakage. To overcome these limitations, we propose Multi-scale Global-Instance
Prompt Tuning(MGIPT), to enhance scale diversity of
prompts and capture both global- and instance-level knowledge for robust CTTA. Specifically, MGIPT consists of an Adaptive-scale Instance
Prompt(AIP) and a Multi-scale Global-level
Prompt(MGP). AIP dynamically learns lightweight and instance-specific
prompts to mitigate error accumulation with adaptive optimal-scale selection mechanism. MGP captures domain-level knowledge across different scales to ensure robust adaptation with anti-forgetting capabilities. These complementary components are combined through a weighted ensemble approach, enabling effective dual-level adaptation that integrates both global and local information. Extensive experiments on medical image
segmentation benchmarks demonstrate that our MGIPT outperforms state-of-the-art methods, achieving robust adaptation across continually changing target domains.
Read the full abstract ▾
All authors Lingrui Li, Yanfeng Zhou, Nan Pu, Xin Chen, Zhun Zhong