Computer Vision
Joint Geometric and Trajectory Consistency Learning for One-Step Real-World Super-Resolution
Chengyan Deng, Zhangquan Chen, Li Yu et al. (+3 more)
Diffusion-based Real-World Image Super-Resolution (Real-ISR) achieves impressive perceptual quality but suffers from high computational costs due to iterative sampling. While recent
distillation approaches leveraging large-scale Text-to-Image (T2I) priors have enabled one-step generation, they are typically hindered by prohibitive parameter counts and the inherent capability bounds imposed by teacher models. As a lightweight alternative, Consistency Models offer efficient inference but struggle with two critical limitations: the accumulation of consistency drift inherent to transitive training, and a phenomenon we term "
Geometric Decoupling" - where the generative trajectory achieves pixel-wise
alignment yet fails to preserve structural coherence. To address these challenges, we propose GTASR (
Geometric Trajectory
Alignment Super-Resolution), a simple yet effective consistency training paradigm for Real-ISR. Specifically, we introduce a Trajectory
Alignment (TA) strategy to rectify the tangent vector field via full-path projection, and a Dual-Reference Structural Rectification (DRSR) mechanism to enforce strict structural constraints. Extensive experiments verify that GTASR delivers superior performance over representative baselines while maintaining minimal latency. The code and model will be released at https://github.com/Blazedengcy/GTASR.
Read the full abstract ▾
All authors Chengyan Deng, Zhangquan Chen, Li Yu, Kai Zhang, Xue Zhou, Wang Zhang
Computer Vision
Enhancing Spatial Understanding in Image Generation via Reward Modeling
Zhenyu Tang, Chaoran Feng, Yufan Deng et al. (+5 more)
Recent progress in text-to-image generation has greatly advanced visual fidelity and creativity, but it has also imposed higher demands on
prompt complexity-particularly in encoding intricate spatial relationships. In such cases, achieving satisfactory results often requires multiple sampling attempts. To address this challenge, we introduce a novel method that strengthens the spatial understanding of current image generation models. We first construct the SpatialReward-
Dataset with over 80k preference pairs. Building on this
dataset, we build SpatialScore, a reward model designed to evaluate the accuracy of spatial relationships in text-to-image generation, achieving performance that even surpasses leading proprietary models on spatial evaluation. We further demonstrate that this reward model effectively enables online reinforcement learning for the complex spatial generation. Extensive experiments across multiple
benchmarks show that our specialized reward model yields significant and consistent gains in spatial understanding for image generation.
Read the full abstract ▾
All authors Zhenyu Tang, Chaoran Feng, Yufan Deng, Jie Wu, Xiaojie Li, Rui Wang, Yunpeng Chen, Daquan Zhou
Computer Vision
MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy
Albert Dominguez Mantes, Gioele La Manno and Martin Weigert
Modern microscopy routinely produces gigapixel images that contain structures across multiple spatial scales, from fine cellular morphology to broader tissue organization. Many analysis tasks require combining these scales, yet most
vision models operate at a single resolution or derive multi-scale features from one view, limiting their ability to exploit the inherently multi-resolution nature of microscopy data. We introduce MuViT, a
transformer architecture built to fuse true multi-resolution observations from the same underlying image. MuViT embeds all patches into a shared world-coordinate system and extends rotary positional
embeddings to these coordinates, enabling
attention to integrate wide-field context with high-resolution detail within a single encoder. Across synthetic
benchmarks, kidney histopathology, and high-resolution mouse-
brain microscopy, MuViT delivers consistent improvements over strong ViT and CNN baselines. Multi-resolution MAE pretraining further produces scale-consistent representations that enhance downstream tasks. These results demonstrate that explicit world-coordinate modelling provides a simple yet powerful mechanism for leveraging multi-resolution information in large-scale microscopy analysis.
Read the full abstract ▾
Computer Vision
SenCache: Accelerating Diffusion Model Inference via Sensitivity-Aware Caching
Yasaman Haghighi and Alexandre Alahi
Diffusion models achieve state-of-the-art
video generation quality, but their inference remains expensive due to the large number of sequential denoising steps. This has motivated a growing line of research on accelerating
diffusion inference. Among training-free acceleration methods, caching reduces computation by reusing previously computed model outputs across timesteps. Existing caching methods rely on heuristic criteria to choose cache/reuse timesteps and require extensive tuning. We address this limitation with a principled sensitivity-aware caching framework. Specifically, we formalize the caching error through an analysis of the model output sensitivity to perturbations in the denoising inputs, i.e., the noisy latent and the timestep, and show that this sensitivity is a key predictor of caching error. Based on this analysis, we propose Sensitivity-Aware Caching (SenCache), a dynamic caching policy that adaptively selects caching timesteps on a per-sample basis. Our framework provides a theoretical basis for adaptive caching, explains why prior empirical heuristics can be partially effective, and extends them to a dynamic, sample-specific approach. Experiments on Wan 2.1, CogVideoX, and LTX-
Video show that SenCache achieves better visual quality than existing caching methods under similar computational budgets.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume
Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin et al. (+1 more)
Despite their capabilities,
Multimodal Large
Language Models (MLLMs) may produce plausible but erroneous outputs, hindering reliable deployment. Accurate uncertainty metrics could enable escalation of unreliable queries to human experts or larger models for improved performance. However, existing uncertainty metrics have practical constraints, such as being designed only for specific modalities, reliant on external tools, or computationally expensive. We introduce UMPIRE, a training-free uncertainty quantification framework for MLLMs that works efficiently across various input and output modalities without external tools, relying only on the models' own internal modality features. UMPIRE computes the incoherence-adjusted semantic volume of sampled MLLM responses for a given task instance, effectively capturing both the global semantic diversity of samples and the local incoherence of responses based on internal model confidence. We propose uncertainty desiderata for MLLMs and provide theoretical analysis motivating UMPIRE's design. Extensive experiments show that UMPIRE consistently outperforms baseline metrics in error detection and uncertainty calibration across image, audio, and
video-text
benchmarks, including
adversarial and out-of-distribution settings. We also demonstrate UMPIRE's generalization to non-text output tasks, including image and audio generation.
Read the full abstract ▾
All authors Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin, Bryan Kian Hsiang Low
Computer Vision
A multimodal slice discovery framework for systematic failure detection and explanation in medical image classification
Yixuan Liu, Kanwal K. Bhatia and Ahmed E. Fetit
Despite advances in machine learning-based medical image classifiers, the safety and reliability of these systems remain major concerns in practical settings. Existing
auditing approaches mainly rely on unimodal features or metadata-based subgroup analyses, which are limited in interpretability and often fail to capture hidden systematic failures. To address these limitations, we introduce the first automated
auditing framework that extends slice discovery methods to
multimodal representations specifically for medical applications. Comprehensive experiments were conducted under common failure scenarios using the MIMIC-CXR-JPG
dataset, demonstrating the framework's strong capability in both failure discovery and explanation generation. Our results also show that
multimodal information generally allows more comprehensive and effective
auditing of classifiers, while unimodal variants beyond image-only inputs exhibit strong potential in scenarios where resources are constrained.
Read the full abstract ▾
Computer Vision
A Mixed Diet Makes DINO An Omnivorous Vision Encoder
Rishabh Kabra, Maks Ovsjanikov, Drew A. Hudson et al. (+5 more)
Pre-trained
vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their features are poorly aligned across different visual modalities. For instance, the feature
embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous
Vision Encoder, a post-training framework that learns a modality-agnostic feature space. We fine-tune the encoder with a dual objective: first, to maximize the feature
alignment between different modalities of the same scene; and second, a
distillation objective that anchors the learned representations to a fully frozen teacher. The resulting student encoder becomes "omnivorous" by producing more consistent
embeddings for a given scene, regardless of the input modality (RGB, Depth,
Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model. Omnivorous model weights are available at https://github.com/google-deepmind/representations4d.
Read the full abstract ▾
All authors Rishabh Kabra, Maks Ovsjanikov, Drew A. Hudson, Ye Xia, Skanda Koppula, Andre Araujo, Joao Carreira, Niloy J. Mitra
Computer Vision
GeoDiff4D: Geometry-Aware Diffusion for 4D Head Avatar Reconstruction
Chao Xu, Xiaochen Zhao, Xiang Deng et al. (+4 more)
Reconstructing photorealistic and animatable 4D head avatars from a single portrait image remains a fundamental challenge in computer
vision. While
diffusion models have enabled remarkable progress in image and
video generation for avatar reconstruction, existing methods primarily rely on 2D priors and struggle to achieve consistent 3D geometry. We propose a novel framework that leverages geometry-aware
diffusion to learn strong geometry priors for high-fidelity head avatar reconstruction. Our approach jointly synthesizes portrait images and corresponding surface normals, while a pose-free expression encoder captures implicit expression representations. Both synthesized images and expression latents are incorporated into 3D Gaussian-based avatars, enabling photorealistic rendering with accurate geometry. Extensive experiments demonstrate that our method substantially outperforms state-of-the-art approaches in visual quality, expression fidelity, and cross-identity generalization, while supporting real-time rendering.
Read the full abstract ▾
All authors Chao Xu, Xiaochen Zhao, Xiang Deng, Jingxiang Sun, Donglin Di, Zhuo Su, Yebin Liu
Computer Vision
Manifold-Preserving Superpixel Hierarchies and Embeddings for the Exploration of High-Dimensional Images
Alexander Vieth, Boudewijn Lelieveldt, Elmar Eisemann et al. (+2 more)
High-dimensional images, or images with a high-dimensional attribute vector per pixel, are commonly explored with coordinated views of a low-dimensional
embedding of the attribute space and a conventional image representation. Nowadays, such images can easily contain several million pixels. For such large
datasets, hierarchical
embedding techniques are better suited to represent the high-dimensional attribute space than flat dimensionality reduction methods. However, available hierarchical dimensionality reduction methods construct the hierarchy purely based on the attribute information and ignore the spatial layout of pixels in the images. This impedes the exploration of regions of interest in the image space, since there is no congruence between a region of interest in image space and the associated attribute abstractions in the hierarchy. In this paper, we present a superpixel hierarchy for high-dimensional images that takes the high-dimensional attribute
manifold into account during construction. Through this, our method enables consistent exploration of high-dimensional images in both image and attribute space. We show the effectiveness of this new image-guided hierarchy in the context of
embedding exploration by comparing it with classical hierarchical
embedding-based image exploration in two use cases.
Read the full abstract ▾
All authors Alexander Vieth, Boudewijn Lelieveldt, Elmar Eisemann, Anna Vilanova, Thomas Höllt
Computer Vision
Martial Guidez, Stefan Duffner and Christophe Garcia
Vision transformers have recently made a breakthrough in computer
vision showing excellent performance in terms of precision for numerous applications. However, their computational cost is very high compared to alternative approaches such as Convolutional Neural Networks. To address this problem, we propose a novel framework for image classification called RAViT based on a multi-branch network that operates on several copies of the same image with different resolutions to reduce the computational cost while preserving the overall accuracy. Furthermore, our framework includes an early exit mechanism that makes our model adaptive and allows to choose the appropriate trade-off between accuracy and computational cost at run-time. For example in a two-branch architecture, the original image is first resized to reduce its resolution, then a prediction is performed on it using a first
transformer and the resulting prediction is reused together with the original-size image to perform a final prediction on a second
transformer with less computation than a classical
Vision transformer architecture. The early-exit process allows the model to make a final prediction at intermediate branches, saving even more computation. We evaluated our approach on CIFAR-10, Tiny ImageNet, and ImageNet. We obtained an equivalent accuracy to the classical
Vision transformer model with only around 70% of FLOPs.
Read the full abstract ▾
Computer Vision
HumanOrbit: 3D Human Reconstruction as 360{\deg} Orbit Generation
Keito Suzuki, Kunyao Chen, Lei Wang et al. (+5 more)
We present a method for generating a full 360{\deg} orbit
video around a person from a single input image. Existing methods typically adapt image-based
diffusion models for multi-view synthesis, but yield inconsistent results across views and with the original identity. In contrast, recent
video diffusion models have demonstrated their ability in generating photorealistic results that align well with the given
prompts. Inspired by these results, we propose HumanOrbit, a
video diffusion model for multi-view human image generation. Our approach enables the model to synthesize continuous camera rotations around the subject, producing geometrically consistent novel views while preserving the appearance and identity of the person. Using the generated multi-view frames, we further propose a reconstruction pipeline that recovers a textured mesh of the subject. Experimental results validate the effectiveness of HumanOrbit for multi-view image generation and that the reconstructed 3D models exhibit superior completeness and fidelity compared to those from state-of-the-art baselines.
Read the full abstract ▾
All authors Keito Suzuki, Kunyao Chen, Lei Wang, Bang Du, Runfa Blark Li, Peng Liu, Ning Bi, Truong Nguyen
Computer Vision
Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation
Muquan Li, Hang Gou, Yingyi Ma et al. (+3 more)
Decoupled
dataset distillation (DD) compresses large corpora into a few synthetic images by matching a frozen teacher's statistics. However, current residual-matching pipelines rely on static real patches, creating a fit-complexity gap and a pull-to-anchor effect that reduce intra-class diversity and hurt generalization. To address these issues, we introduce RETA – a
Retrieval and
Topology Alignment framework for decoupled DD. First, Dynamic
Retrieval Connection (DRC) selects a real patch from a prebuilt pool by minimizing a fit-complexity score in teacher feature space; the chosen patch is injected via a residual connection to tighten feature fit while controlling injected complexity. Second, Persistent
Topology Alignment (PTA) regularizes synthesis with persistent homology: we build a mutual k-NN feature graph, compute persistence images of components and loops, and penalize
topology discrepancies between real and synthetic sets, mitigating pull-to-anchor effect. Across CIFAR-100, Tiny-ImageNet, ImageNet-1K, and multiple ImageNet subsets, RETA consistently outperforms various baselines under comparable time and
memory, especially reaching 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at 50 images per class, +3.1% over the best prior.
Read the full abstract ▾
All authors Muquan Li, Hang Gou, Yingyi Ma, Rongzheng Wang, Ke Qin, Tao He
Computer Vision
Omar Mohamed, Edoardo Fazzari, Ayah Al-Naji et al. (+5 more)
Automated recognition of
surgical phases and steps is a fundamental capability for intraoperative decision support, workflow automation, and skill assessment in robotic-assisted surgery. Existing approaches either depend on large-scale annotated
surgical datasets or require expensive domain-specific pretraining on thousands of labeled
videos, limiting their practical deployability across diverse robotic platforms and clinical environments. In this work, we propose TASOT (Text-Augmented Action
Segmentation Optimal Transport), an annotation-free framework for
surgical temporal
segmentation that requires no task-specific annotations or
surgical-domain pretraining. TASOT extends the Action
Segmentation Optimal Transport (ASOT) formulation by incorporating temporally aligned textual descriptions generated directly from the input
video, fusing visual and semantic cues within a unified unbalanced Gromov-Wasserstein optimal transport objective. Visual representations are extracted using DINOv3, while temporal captions produced by a
vision-
language model are encoded via CLIP and temporally aligned to individual frames, providing complementary semantic structure to the transport cost. We evaluate TASOT on three public
surgical datasets and four
benchmark settings spanning laparoscopic and robotic procedures, showing substantial improvements over the strongest zero-shot baselines: +18.9 F1 on Cholec80, +33.7 on AutoLaparo, +23.7 on StrasByPass70, and +4.5 on BernByPass70. These results suggest that fine-grained
surgical workflow understanding in robotic settings can be achieved without manual training annotations or
surgical-specific pretraining pipelines, offering a promising alternative for real-world robotic
surgical systems.
Read the full abstract ▾
All authors Omar Mohamed, Edoardo Fazzari, Ayah Al-Naji, Hamdan Alhadhrami, Khalfan Hableel, Saif Alkindi, Ivan Laptev, Cesare Stefanini
Computer Vision
Prune Wisely, Reconstruct Sharply: Compact 3D Gaussian Splatting via Adaptive Pruning and Difference-of-Gaussian Primitives
Haoran Wang, Guoxi Huang, Fan Zhang et al. (+2 more)
Recent significant advances in 3D scene representation have been driven by 3D Gaussian Splatting (3DGS), which has enabled real-time rendering with photorealistic quality. 3DGS often requires a large number of primitives to achieve high fidelity, leading to redundant representations and high resource consumption, thereby limiting its scalability for complex or large-scale scenes. Consequently, effective
pruning strategies and more expressive primitives that can reduce redundancy while preserving visual quality are crucial for practical deployment. We propose an efficient, integrated reconstruction-aware
pruning strategy that adaptively determines
pruning timing and refining intervals based on reconstruction quality, thus reducing model size while enhancing rendering quality. Moreover, we introduce a 3D Difference-of-Gaussians primitive that jointly models both positive and negative densities in a single primitive, improving the expressiveness of Gaussians under compact configurations. Our method significantly improves model compactness, achieving up to 90% reduction in Gaussian-count while delivering visual quality that is similar to, or in some cases better than, that produced by state-of-the-art methods. Code will be made publicly available.
Read the full abstract ▾
All authors Haoran Wang, Guoxi Huang, Fan Zhang, David Bull, Nantheera Anantrasirichai
Computer Vision
AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation
Zhengren Wang, Dongsheng Ma, Huaping Zhong et al. (+4 more)
The expansion of
retrieval-augmented generation (RAG) into
multimodal domains has intensified the challenge for processing complex visual documents, such as financial reports. While page-level chunking and
retrieval is a natural starting point, it creates a critical bottleneck: delivering entire pages to the generator introduces excessive extraneous context. This not only overloads the generator's
attention mechanism but also dilutes the most salient evidence. Moreover, compressing these information-rich pages into a limited visual
token budget further increases the risk of
hallucinations. To address this, we introduce AgenticOCR, a dynamic parsing paradigm that transforms optical character recognition (OCR) from a static, full-text process into a query-driven, on-demand extraction system. By autonomously analyzing document layout in a "thinking with images" manner, AgenticOCR identifies and selectively recognizes regions of interest. This approach performs on-demand decompression of visual
tokens precisely where needed, effectively decoupling
retrieval granularity from rigid page-level chunking. AgenticOCR has the potential to serve as the "third building block" of the visual document RAG stack, operating alongside and enhancing standard
Embedding and Reranking modules. Experimental results demonstrate that AgenticOCR improves both the efficiency and accuracy of visual RAG systems, achieving expert-level performance in long document understanding. Code and models are available at https://github.com/OpenDataLab/AgenticOCR.
Read the full abstract ▾
All authors Zhengren Wang, Dongsheng Ma, Huaping Zhong, Jiayu Li, Wentao Zhang, Bin Wang, Conghui He
Computer Vision
FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Tracking
Sifan Zhou, Jiahao Nie, Ziyu Zhao et al. (+2 more)
In 3D point cloud object tracking, the motion-centric methods have emerged as a promising avenue due to its superior performance in modeling inter-frame motion. However, existing two-stage motion-based approaches suffer from fundamental limitations: (1) error accumulation due to decoupled optimization caused by explicit foreground
segmentation prior to motion estimation, and (2) computational bottlenecks from sequential processing. To address these challenges, we propose FocusTrack, a novel one-stage paradigms tracking framework that unifies motion-semantics co-modeling through two core innovations: Inter-frame Motion Modeling (IMM) and Focus-and-Suppress
Attention. The IMM module employs a temp-oral-difference siamese encoder to capture global motion patterns between adjacent frames. The Focus-and-Suppress
attention that enhance the foreground semantics via motion-salient feature gating and suppress the background noise based on the temporal-aware motion context from IMM without explicit
segmentation. Based on above two designs, FocusTrack enables end-to-end training with compact one-stage pipeline. Extensive experiments on prominent 3D tracking
benchmarks, such as KITTI, nuScenes, and Waymo, demonstrate that the FocusTrack achieves new SOTA performance while running at a high speed with 105 FPS.
Read the full abstract ▾
All authors Sifan Zhou, Jiahao Nie, Ziyu Zhao, Yichao Cao, Xiaobo Lu
Computer Vision
Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification
Vikash Singh, Debargha Ganguly, Haotian Yu et al. (+5 more)
Vision-
language models (VLMs) show promise in drafting radiology reports, yet they frequently suffer from logical inconsistencies, generating diagnostic impressions unsupported by their own perceptual findings or missing logically entailed conclusions. Standard lexical metrics heavily penalize clinical paraphrasing and fail to capture these deductive failures in reference-free settings. Toward guarantees for clinical
reasoning, we introduce a neurosymbolic verification framework that deterministically audits the internal consistency of VLM-generated reports. Our pipeline autoformalizes free-text radiographic findings into structured propositional evidence, utilizing an SMT solver (Z3) and a clinical knowledge base to verify whether each diagnostic claim is mathematically entailed, hallucinated, or omitted. Evaluating seven VLMs across five chest X-ray
benchmarks, our verifier exposes distinct
reasoning failure modes, such as conservative observation and
stochastic hallucination, that remain invisible to traditional metrics. On labeled
datasets, enforcing solver-backed entailment acts as a rigorous post-hoc guarantee, systematically eliminating unsupported
hallucinations to significantly increase diagnostic soundness and precision in generative clinical assistants.
Read the full abstract ▾
All authors Vikash Singh, Debargha Ganguly, Haotian Yu, Chengwei Zhou, Prerna Singh, Brandon Lee, Vipin Chaudhary, Gourav Datta
Computer Vision
DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer
Yuxuan Zhang, Katarína Tóthová, Zian Wang et al. (+7 more)
Simulation is essential to the development and evaluation of autonomous
robots such as self-driving vehicles. Neural reconstruction is emerging as a promising solution as it enables simulating a wide variety of scenarios from real-world data alone in an automated and scalable way. However, while methods such as NeRF and 3D Gaussian Splatting can produce visually compelling results, they often exhibit artifacts particularly when rendering novel views, and fail to realistically integrate inserted dynamic objects, especially when they were captured from different scenes. To overcome these limitations, we introduce DiffusionHarmonizer, an online generative enhancement framework that transforms renderings from such imperfect scenes into temporally consistent outputs while improving their realism. At its core is a single-step temporally-conditioned enhancer that is converted from a pretrained multi-step image
diffusion model, capable of running in online simulators on a single GPU. The key to training it effectively is a custom data curation pipeline that constructs synthetic-real pairs emphasizing appearance harmonization, artifact correction, and lighting realism. The result is a scalable system that significantly elevates simulation fidelity in both research and production environments.
Read the full abstract ▾
All authors Yuxuan Zhang, Katarína Tóthová, Zian Wang, Kangxue Yin, Haithem Turki, Riccardo de Lutio, Yen-Yu Chang, Or Litany, Sanja Fidler, Zan Gojcic
Computer Vision
FoV-Net: Rotation-Invariant CAD B-rep Learning via Field-of-View Ray Casting
Matteo Ballegeer and Dries F. Benoit
Learning directly from boundary representations (B-reps) has significantly advanced 3D CAD analysis. However, state-of-the-art B-rep learning methods rely on absolute coordinates and normals to encode global context, making them highly sensitive to rotations. Our experiments reveal that models achieving over 95% accuracy on aligned
benchmarks can collapse to as low as 10% under arbitrary $\mathbf{SO}(3)$ rotations. To address this, we introduce FoV-Net, the first B-rep learning framework that captures both local surface geometry and global structural context in a rotation-invariant manner. Each face is represented by a Local Reference Frame (LRF) UV-grid that encodes its local surface geometry, and by Field-of-View (FoV) grids that capture the surrounding 3D context by casting rays and recording intersections with neighboring faces. Lightweight CNNs extract per-face features, which are propagated over the B-rep graph using a graph
attention network. FoV-Net achieves state-of-the-art performance on B-rep classification and
segmentation benchmarks, demonstrating robustness to arbitrary rotations while also requiring less training data to achieve strong results.
Read the full abstract ▾
Computer Vision
EvalMVX: A Unified Benchmarking for Neural 3D Reconstruction under Diverse Multiview Setups
Zaiyan Yang, Jieji Ren, Xiangyi Wang et al. (+5 more)
Recent advancements in neural surface reconstruction have significantly enhanced 3D reconstruction. However, current real world
datasets mainly focus on benchmarking multiview stereo (MVS) based on RGB inputs. Multiview photometric stereo (MVPS) and multiview shape from polarization (MVSfP), though indispensable on high-fidelity surface reconstruction and sparse inputs, have not been quantitatively assessed together with MVS. To determine the working range of different MVX (MVS, MVSfP, and MVPS) techniques, we propose EvalMVX, a real-world
dataset containing $25$ objects, each captured with a polarized camera under $20$ varying views and $17$ light conditions including OLAT and natural illumination, leading to $8,500$ images. Each object includes aligned ground-truth 3D mesh, facilitating quantitative benchmarking of MVX methods simultaneously. Based on our EvalMVX, we evaluate $13$ MVX methods published in recent years, record the best-performing methods, and identify open problems under diverse
geometric details and reflectance types. We hope EvalMVX and the benchmarking results can inspire future research on multiview 3D reconstruction.
Read the full abstract ▾
All authors Zaiyan Yang, Jieji Ren, Xiangyi Wang, zonglin li, Xu Cao, Heng Guo, Zhanyu Ma, Boxin Shi
Computer Vision
Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization
Chenwei Jia, Baoting Li, Xuchong Zhang et al. (+3 more)
Post-Training Quantization (PTQ) has emerged as an effective technique for alleviating the substantial computational and
memory overheads of
Vision-
Language Models (VLMs) by compressing both weights and activations without retraining the full model. Existing PTQ methods primarily rely on static identification and global compensation of sensitive or outlier channels, yet they often overlook the distributional differences of these important channels across inputs, leading to unsatisfactory quantization. In this work, we observe that the distributions and occurrence frequencies of important channels vary significantly both across modalities and among
tokens, even within the same modality. Accordingly, we propose Quant Experts (QE), a
token-aware adaptive error compensation with mixture-of-experts for VLMs quantization. QE divides the important channels into
token-independent and
token-dependent groups. For the former, a shared expert is designed for most
tokens to compensate for global quantization error using a low-rank adapter. For the latter, routed experts including multiple routed low-rank adapters are elaborated to compensate for local quantization error related to specific
tokens. Extensive experiments demonstrate that QE consistently enhances task accuracy across various quantization settings and model scales, ranging from 2B to 70B parameters, while maintaining performance comparable to full-precision models.
Read the full abstract ▾
All authors Chenwei Jia, Baoting Li, Xuchong Zhang, Mingzhuo Wei, Bochen Lin, Hongbin Sun
Computer Vision
Spatio-Temporal Garment Reconstruction Using Diffusion Mapping via Pattern Coordinates
Yingxuan You, Ren Li, Corentin Dumery et al. (+3 more)
Reconstructing 3D clothed humans from
monocular images and
videos is a fundamental problem with applications in virtual try-on, avatar creation, and mixed reality. Despite significant progress in human body recovery, accurately reconstructing garment geometry, particularly for loose-fitting clothing, remains an open challenge. We propose a unified framework for high-fidelity 3D garment reconstruction from both single images and
video sequences. Our approach combines Implicit Sewing Patterns (ISP) with a generative
diffusion model to learn expressive garment shape priors in 2D UV space. Leveraging these priors, we introduce a mapping model that establishes correspondences between image pixels, UV pattern coordinates, and 3D geometry, enabling accurate and detailed garment reconstruction from single images. We further extend this formulation to dynamic reconstruction by introducing a spatio-temporal
diffusion scheme with test-time guidance to enforce long-range temporal consistency. We also develop analytic projection-based constraints that preserve image-aligned geometry in visible regions while enforcing coherent completion in occluded areas over time. Although trained exclusively on synthetically simulated cloth data, our method generalizes well to real-world imagery and consistently outperforms existing approaches on both tight- and loose-fitting garments. The reconstructed garments preserve fine
geometric detail while exhibiting realistic dynamic motion, supporting downstream applications such as texture editing, garment retargeting, and animation.
Read the full abstract ▾
All authors Yingxuan You, Ren Li, Corentin Dumery, Cong Cao, Hao Li, Pascal Fua
Computer Vision
Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation
Xingyu Zhu, Kesen Zhao, Liang Yi et al. (+4 more)
Multimodal large
language models (MLLMs) have achieved remarkable progress in
vision-
language reasoning, yet they remain vulnerable to
hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent
vision enhancement methods attempt to address this issue by reinforcing visual
tokens during decoding, but they typically inject all
tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based
token reduction condenses the large pool of visual
tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the
alignment between hidden states and patch
embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates
hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces
hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.
Read the full abstract ▾
All authors Xingyu Zhu, Kesen Zhao, Liang Yi, Shuo Wang, Zhicai Wang, Beier Zhu, Hanwang Zhang
Computer Vision
Xingyu Zhu, Beier Zhu, Junfeng Fang et al. (+4 more)
Large
vision-
language models (LVLMs) have achieved remarkable progress in
vision-
language reasoning tasks, yet ensuring their safety remains a critical challenge. Recent input-side defenses detect unsafe images with CLIP and prepend safety prefixes to
prompts, but they still suffer from inaccurate detection in complex scenes and unstable safety signals during decoding. To address these issues, we propose GuardAlign, a training-free defense framework that integrates two strategies. First, OT-enhanced safety detection leverages optimal transport to measure distribution distances between image patches and unsafe semantics, enabling accurate identification of malicious regions without additional computational cost. Second, cross-modal attentive calibration strengthens the influence of safety prefixes by adaptively reallocating
attention across layers, ensuring that safety signals remain consistently activated throughout generation. Extensive evaluations on six representative MLLMs demonstrate that GuardAlign reduces unsafe response rates by up to 39% on SPA-VL, while preserving utility, achieving an improvement on VQAv2 from 78.51% to 79.21%.
Read the full abstract ▾
All authors Xingyu Zhu, Beier Zhu, Junfeng Fang, Shuo Wang, Yin Zhang, Xiang Wang, Xiangnan He
Computer Vision
Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection
Zhaolin Cai, Fan Li, Huiyu Duan et al. (+2 more)
Video anomaly detection (VAD) aims to identify abnormal events in
videos. Traditional VAD methods generally suffer from the high costs of labeled data and full training, thus some recent works have explored leveraging frozen multi-modal large
language models (MLLMs) in a tuning-free manner to perform VAD. However, their performance is limited as they directly inherit pre-training biases and cannot adapt internal representations to specific
video contexts, leading to difficulties in handling subtle or ambiguous anomalies. To address these limitations, we propose a novel intervention framework, termed SteerVAD, which advances MLLM-based VAD by shifting from passively reading to actively steering and rectifying internal representations. Our approach first leverages the gradient-free representational separability analysis (RSA) to identify top
attention heads as latent anomaly experts (LAEs) which are most discriminative for VAD. Then a hierarchical meta-controller (HMC) generates dynamic rectification signals by jointly conditioning on global context and these LAE outputs. The signals execute targeted, anisotropic scaling directly upon the LAE representation
manifolds, amplifying anomaly-relevant dimensions while suppressing inherent biases. Extensive experiments on mainstream
benchmarks demonstrate our method achieves state-of-the-art performance among tuning-free approaches requiring only 1% of training data, establishing it as a powerful new direction for
video anomaly detection. The code will be released upon the publication.
Read the full abstract ▾
All authors Zhaolin Cai, Fan Li, Huiyu Duan, Lijun He, Guangtao Zhai
Computer Vision
SR3R: Rethinking Super-Resolution 3D Reconstruction With Feed-Forward Gaussian Splatting
Xiang Feng, Xiangbo Wang, Tieshi Zhong et al. (+7 more)
3D super-resolution (3DSR) aims to reconstruct high-resolution (HR) 3D scenes from low-resolution (LR) multi-view images. Existing methods rely on dense LR inputs and per-scene optimization, which restricts the high-frequency priors for constructing HR 3D Gaussian Splatting (3DGS) to those inherited from pretrained 2D super-resolution (2DSR) models. This severely limits reconstruction fidelity, cross-scene generalization, and real-time usability. We propose to reformulate 3DSR as a direct feed-forward mapping from sparse LR views to HR 3DGS representations, enabling the model to autonomously learn 3D-specific high-frequency geometry and appearance from large-scale, multi-scene data. This fundamentally changes how 3DSR acquires high-frequency knowledge and enables robust generalization to unseen scenes. Specifically, we introduce SR3R, a feed-forward framework that directly predicts HR 3DGS representations from sparse LR views via the learned mapping network. To further enhance reconstruction fidelity, we introduce Gaussian offset learning and feature refinement, which stabilize reconstruction and sharpen high-frequency details. SR3R is plug-and-play and can be paired with any feed-forward 3DGS reconstruction backbone: the backbone provides an LR 3DGS scaffold, and SR3R upscales it to an HR 3DGS. Extensive experiments across three 3D
benchmarks demonstrate that SR3R surpasses state-of-the-art (SOTA) 3DSR methods and achieves strong zero-shot generalization, even outperforming SOTA per-scene optimization methods on unseen scenes.
Read the full abstract ▾
All authors Xiang Feng, Xiangbo Wang, Tieshi Zhong, Chengkai Wang, Yiting Zhao, Tianxiang Xu, Zhenzhong Kuang, Feiwei Qin, Xuefei Yin, Yanming Zhu
Computer Vision
Interpretable Debiasing of Vision-Language Models for Social Fairness
Na Min An, Yoonna Jang, Yusuke Hirota et al. (+3 more)
The rapid advancement of
Vision-
Language models (VLMs) has raised growing concerns that their black-box
reasoning processes could lead to unintended forms of social bias. Current debiasing approaches focus on mitigating surface-level bias signals through post-hoc learning or test-time
algorithms, while leaving the internal dynamics of the model largely unexplored. In this work, we introduce an interpretable, model-agnostic bias mitigation framework, DeBiasLens, that localizes social attribute
neurons in VLMs through sparse autoencoders (SAEs) applied to
multimodal encoders. Building upon the disentanglement ability of SAEs, we train them on facial image or caption
datasets without corresponding social attribute labels to uncover
neurons highly responsive to specific demographics, including those that are underrepresented. By selectively deactivating the social
neurons most strongly tied to bias for each group, we effectively mitigate socially biased behaviors of VLMs without degrading their semantic knowledge. Our research lays the groundwork for future
auditing tools, prioritizing social
fairness in emerging real-world AI systems.
Read the full abstract ▾
All authors Na Min An, Yoonna Jang, Yusuke Hirota, Ryo Hachiuma, Isabelle Augenstein, Hyunjung Shim
Computer Vision
Ordinal Diffusion Models for Color Fundus Images
Gustav Schmidt, Philipp Berens and Sarah Müller
Generative image models such as
diffusion models can improve performance on clinically relevant tasks by offering deep learning models supplementary training data. However, most conditional
diffusion models treat disease stages as independent classes, ignoring the continuous nature of disease progression. This mismatch is problematic in medical imaging because continuous pathological processes are typically only observed through coarse, discrete but ordered labels as in ophthalmology for diabetic retinopathy (DR). We propose an ordinal latent
diffusion model for generating color fundus images that explicitly incorporates the ordered structure of DR severity into the generation process. Instead of categorical conditioning, we used a scalar disease representation, enabling a smooth transition between adjacent stages. We evaluated our approach using visual realism metrics and classification-based clinical consistency analysis on the EyePACS
dataset. Compared to a standard conditional
diffusion model, our model reduced the Fréchet inception distance for four of the five DR stages and increased the quadratic weighted $\kappa$ from 0.79 to 0.87. Furthermore, interpolation experiments showed that the model captured a continuous spectrum of disease progression learned from ordered, coarse class labels. Code available at https://github.com/berenslab/OrdinalDiffusionModels.
Read the full abstract ▾
Computer Vision
Accelerating Masked Image Generation by Learning Controlled Latent Dynamics
Kaiwen Zhu, Quansheng Zeng, Yuandong Pu et al. (+9 more)
Masked Image Generation Models (MIGMs) have achieved great success, yet their efficiency is hampered by the multiple steps of bi-directional
attention. In fact, there exists notable redundancy in their computation: when sampling discrete
tokens, the rich semantics contained in the continuous features are lost. Some existing works attempt to cache the features to approximate future features. However, they exhibit considerable approximation error under aggressive acceleration settings. We attribute this to their limited expressivity and the failure to account for sampling information. To fill this gap, we propose learning a lightweight model that incorporates both previous features and sampled
tokens, and regresses the average velocity field of feature
evolution. The model has moderate complexity that suffices to capture the subtle dynamics while keeping lightweight compared to the original base model. We apply our method to two representative MIGMs and tasks. In particular, on the state-of-the-art Lumina-DiMOO, it achieves over 4x acceleration of text-to-image generation while maintaining quality, significantly pushing the Pareto frontier of masked image generation. The code and model weights are available at https://github.com/Kaiwen-Zhu/MIGM-Shortcut.
Read the full abstract ▾
All authors Kaiwen Zhu, Quansheng Zeng, Yuandong Pu, Shuo Cao, Xiaohui Li, Yi Xin, Qi Qin, Jiayang Li, Juncheng Yan, Yu Qiao, Jinjin Gu, Yihao Liu
Computer Vision · Cross-listed from cs.LG
MINT: Multimodal Imaging-to-Speech Knowledge Transfer for Early Alzheimer's Screening
Vrushank Ahire, Yogesh Kumar, Anouck Girard et al. (+1 more)
Alzheimer's disease is a progressive neurodegenerative disorder in which mild cognitive impairment (MCI) precedes dementia. Structural MRI provides biomarkers but requires costly infrastructure, limiting population-scale deployment.
Speech offers a non-invasive alternative, yet
speech-only classifiers are developed independently of neuroimaging and lack biological grounding for CN-versus-MCI classification. We propose MINT (
Multimodal Imaging-to-
Speech Knowledge Transfer), a three-stage framework that transfers MRI-derived biomarker structure to
speech during training. An MRI teacher defines a compact
embedding space for CN-versus-MCI classification, while a residual projection head aligns
speech representations to this space using a combined
geometric loss. The frozen MRI classifier enables imaging-free inference. On ADNI-4, aligned
speech achieves performance comparable to
speech baselines, while
multimodal fusion improves over MRI alone. Ablations identify dropout regularization and self-supervised pretraining as important design choices. To our knowledge, MINT is the first demonstration of MRI-to-
speech knowledge transfer for early Alzheimer's screening without imaging at inference.
Read the full abstract ▾
All authors Vrushank Ahire, Yogesh Kumar, Anouck Girard, M. A. Ganaie