Computer Vision · Cross-listed from cs.LG
Rethinking Dataset Distillation: Hard Truths about Soft Labels
Priyam Dey, Aditya Sahdev, Sunny Bhati et al. (+2 more)
Despite the perceived success of large-scale
dataset distillation (DD) methods, recent evidence finds that simple random image baselines perform on-par with state-of-theart DD methods like SRe2L due to the use of soft labels during downstream model training. This is in contrast with the findings in coreset literature, where high-quality coresets consistently outperform random subsets in the hardlabel (HL) setting. To understand this discrepancy, we perform a detailed scalability analysis to examine the role of data quality under different label regimes, ranging from abundant soft labels (termed as SL+KD regime) to fixed soft labels (SL) and hard labels (HL). Our analysis reveals that high-quality coresets fail to convincingly outperform the random baseline in both SL and SL+KD regimes. In the SL+KD setting, performance further approaches nearoptimal levels relative to the full
dataset, regardless of subset size or quality, for a given compute budget. This performance saturation calls into question the widespread practice of using soft labels for model evaluation, where unlike the HL setting, subset quality has negligible influence. A subsequent systematic evaluation of five large-scale and four small-scale DD methods in the HL setting reveals that only RDED reliably outperforms random baselines on ImageNet-1K, but can still lag behind strong coreset methods due to its over-reliance on easy sample patches. Based on this, we introduce CAD-Prune, a compute-aware
pruning metric that efficiently identifies samples of optimal difficulty for a given compute budget, and use it to develop CA2D, a compute-aligned DD method, outperforming current DD methods on ImageNet-1K at various IPC settings. Together, our findings uncover many insights into current DD research and establish useful tools to advance dataefficient learning for both coresets and DD.
Read the full abstract ▾
All authors Priyam Dey, Aditya Sahdev, Sunny Bhati, Konda Reddy Mopuri, R. Venkatesh Babu
Computer Vision
Geometric Decoupling: Diagnosing the Structural Instability of Latent
Yuanbang Liang, Zhengwen Chen and Yu-Kun Lai
Latent
Diffusion Models (LDMs) achieve high-fidelity synthesis but suffer from latent space brittleness, causing discontinuous semantic jumps during editing. We introduce a Riemannian framework to diagnose this instability by analyzing the generative Jacobian, decomposing geometry into Local Scaling (capacity) and Local Complexity (
curvature). Our study uncovers a “
Geometric Decoupling": while
curvature in normal generation functionally encodes image detail, OOD generation exhibits a functional decoupling where extreme
curvature is wasted on unstable semantic boundaries rather than perceptible details. This
geometric misallocation identifies “
Geometric Hotspots" as the structural root of instability, providing a robust intrinsic metric for diagnosing generative reliability.
Read the full abstract ▾
Computer Vision
LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models
Zhiyuan Jiang, Weihao Hong, Xinlei Guan et al. (+8 more)
Vision-
Language Models (VLMs) are increasingly deployed in settings where reliable visual grounding carries operational consequences, yet their behavior under progressively coercive
prompt phrasing remains undercharacterized. Existing
hallucination benchmarks predominantly rely on neutral
prompts and binary detection, leaving open how both the incidence and the intensity of fabrication respond to graded linguistic pressure across structurally distinct task types. We present Ghost-100, a procedurally constructed
benchmark of 800 synthetically generated images spanning eight categories across three task families: text-illegibility, time-reading, and object-absence, each designed under a negative-ground-truth principle that guarantees the queried target is absent, illegible, or indeterminate by construction. Every image is paired with five
prompts drawn from a structured 5-Level
Prompt Intensity Framework, holding the image and task identity fixed while varying only directive force, so that tone is isolated as the sole independent variable. We adopt a dual-track evaluation protocol: a rule-based H-Rate measuring the proportion of responses in which a model crosses from grounded refusal into unsupported positive commitment, and a GPT-4o-mini-judged H-Score on a 1-5 scale characterizing the confidence and specificity of fabrication once it occurs. We additionally release a three-stage automated validation workflow, which retrospectively confirms 717 of 800 images as strictly compliant. Evaluating nine open-weight VLMs, we find that H-Rate and H-Score dissociate substantially across model families, reading-style and presence-detection subsets respond to
prompt pressure in qualitatively different ways, and several models exhibit non-monotonic sensitivity peaking at intermediate tone levels: patterns that aggregate metrics obscure.
Read the full abstract ▾
All authors Zhiyuan Jiang, Weihao Hong, Xinlei Guan, Tejaswi Dhandu, Miles Q. Li, Meng Xu, Kuan Huang, Umamaheswara Rao Tida, Bingyu Shen, Daehan Kwak, Boyang Li
Computer Vision
CrossPan: A Comprehensive Benchmark for Cross-Sequence Pancreas MRI Segmentation and Generalization
Linkai Peng, Cuiling Sun, Zheyuan Zhang et al. (+10 more)
Automatic pancreas
segmentation is fundamental to abdominal MRI analysis, yet deep learning models trained on one MRI sequence often fail catastrophically when applied to another-a challenge that has received little systematic investigation. We introduce CrossPan, a multi-institutional
benchmark comprising 1,386 3D scans across three routinely acquired sequences (T1-weighted, T2-weighted, and Out-of-Phase) from eight centers. Our experiments reveal three key findings. First, cross-sequence domain shifts are far more severe than cross-center variability: models achieving Dice scores above 0.85 in-domain collapse to near-zero (<0.02) when transferred across sequences. Second, state-of-the-art domain generalization methods provide negligible benefit under these physics-driven contrast inversions, whereas foundation models like MedSAM2 maintain moderate zero-shot performance through contrast-invariant shape priors. Third, semi-supervised learning offers gains only under stable intensity distributions and becomes unstable on sequences with high intra-organ variability. These results establish cross-sequence generalization-not model architecture or center diversity-as the primary barrier to clinically deployable pancreas MRI
segmentation.
Dataset and code are available at https://crosspan.netlify.app/.
Read the full abstract ▾
All authors Linkai Peng, Cuiling Sun, Zheyuan Zhang, Wanying Dou, Halil Ertugrul Aktas, Andrea M Bejar, Elif Keles, Tamas Gonda, Michael B Wallace, Zongwei Zhou, Gorkem Durak, Rajesh N Keswani, Ulas Bagci
Computer Vision
EfficientPENet: Real-Time Depth Completion from Sparse LiDAR via Lightweight Multi-Modal Fusion
Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi et al. (+4 more)
Depth completion from sparse
LiDAR measurements and corresponding RGB images is a prerequisite for accurate 3D perception in robotic systems. Existing methods achieve high accuracy on standard
benchmarks but rely on heavy backbone architectures that preclude real-time deployment on embedded hardware. We present EfficientPENet, a two-branch depth completion network that replaces the conventional ResNet encoder with a modernized ConvNeXt backbone, introduces sparsity-invariant convolutions for the depth stream, and refines predictions through a Convolutional Spatial Propagation Network (CSPN). The RGB branch leverages ImageNet-pretrained ConvNeXt blocks with Layer Normalization, 7x7 depthwise convolutions, and
stochastic depth regularization. Features from both branches are merged via late fusion and decoded through a multi-scale deep supervision strategy. We further introduce a position-aware test-time augmentation scheme that corrects coordinate tensors during horizontal flipping, yielding consistent error reduction at inference. On the KITTI depth completion
benchmark, EfficientPENet achieves an RMSE of 631.94 mm with 36.24M parameters and a latency of 20.51 ms, operating at 48.76 FPS. This represents a 3.7 times reduction in parameters and a 23 times speedup relative to BP-Net, while maintaining competitive accuracy. These results establish EfficientPENet as a practical solution for real-time depth completion on resource-constrained edge platforms such as the NVIDIA Jetson.
Read the full abstract ▾
All authors Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi, Anton Netchaev, Steven Sloan, Ken Pathak, Kendall N. Niles
Computer Vision
CAHAL: Clinically Applicable resolution enHAncement for Low-resolution MRI scans
Sergio Morell-Ortega, Ángela González-Cebrián, Boris Mansencal et al. (+8 more)
Large-scale automated morphometric analysis of
brain MRI is limited by the thick-slice, anisotropic acquisitions prevalent in routine clinical practice. Existing generative super-resolution (SR) methods produce visually compelling isotropic volumes but often introduce anatomical
hallucinations, systematic volumetric overestimation, and structural distortions that compromise downstream quantitative analysis and diagnostic safety. To address this, we propose CAHAL (Clinically Applicable resolution enHAncement for Low-resolution MRI scans), a
hallucination-robust, physics-informed resolution enhancement framework that operates directly in the patient's native acquisition space. CAHAL employs a deterministic bivariate Mixture of Experts (MoE) architecture routing each input through specialised residual 3D U-Net experts conditioned on both volumetric resolution and acquisition anisotropy, two independent descriptors of clinical MRI acquisition. Experts are optimised with a composite loss combining edge-penalised spatial reconstruction,
Fourier-domain spectral coherence matching, and a
segmentation-guided semantic consistency constraint. Training pairs are generated on-the-fly via physics-based degradation sampled from a large-scale real-world database, ensuring robust generalisation. Validated on T1-weighted and FLAIR sequences against generative baselines, CAHAL achieves state-of-the-art results, improving the best related methods in terms of accuracy and efficiency.
Read the full abstract ▾
All authors Sergio Morell-Ortega, Ángela González-Cebrián, Boris Mansencal, Marien Gadea, Roberto Vivo-Hernando, Gregorio Rubio, Fernando Aparici, Maria de la Iglesia-Vaya, Gwenaelle Catheline, Pierrick Coupé, José V. Manjón
Computer Vision
REVEAL: Multimodal Vision-Language Alignment of Retinal Morphometry and Clinical Risks for Incident AD and Dementia Prediction
Seowung Leem, Lin Gu, Chenyu You et al. (+2 more)
The retina provides a unique, noninvasive window into Alzheimer's disease (AD) and dementia, capturing early structural changes through morphometric features, while systemic and lifestyle risk factors reflect well-established contributors to disease susceptibility long before clinical symptom onset. However, current retinal analysis frameworks typically model imaging and risk factors separately, limiting their ability to capture joint
multimodal patterns critical for early risk prediction. Moreover, existing methods rarely incorporate mechanisms to organize or align patients with similar retinal and clinical characteristics, constraining the learning of coherent cross-modal associations. To address these limitations, we introduce REVEAL (REtinal-risk
Vision-
Language Early Alzheimer's Learning), a framework that aligns color fundus photographs with individualized disease-specific risk profiles for predicting incident AD and dementia, on average 8 years before diagnosis (range: 1-11 years). Because real-world risk factors are structured questionnaire data, we translate them into clinically interpretable narratives compatible with pretrained
vision-
language models (VLMs). We further propose a group-aware contrastive learning (GACL) strategy that clusters patients with similar retinal morphometry and risk factors as positive pairs, strengthening
multimodal alignment. This unified representation learning framework substantially outperforms state-of-the-art retinal imaging models paired with clinical text encoders, as well as general-purpose VLMs, demonstrating the value of jointly modeling retinal biomarkers and clinical risk factors. By providing a generalizable and noninvasive approach for early AD and dementia risk stratification, REVEAL has the potential to enable earlier intervention and improve preventive care at the population level.
Read the full abstract ▾
All authors Seowung Leem, Lin Gu, Chenyu You, Kuang Gong, Ruogu Fang
Computer Vision
URoPE: Universal Relative Position Embedding across Geometric Spaces
Yichen Xie, Depu Meng, Chensheng Peng et al. (+4 more)
Relative position
embedding has become a standard mechanism for encoding positional information in
Transformers. However, existing formulations are typically limited to a fixed
geometric space, namely 1D sequences or regular 2D/3D grids, which restricts their applicability to many computer
vision tasks that require
geometric reasoning across camera views or between 2D and 3D spaces. To address this limitation, we propose URoPE, a universal extension of Rotary Position
Embedding (RoPE) to cross-view or cross-dimensional
geometric spaces. For each key/value image patch, URoPE samples 3D points along the corresponding camera ray at predefined depth anchors and projects them into the query image plane. Standard 2D RoPE can then be applied using the projected pixel coordinates. URoPE is a parameter-free and intrinsics-aware relative position
embedding that is invariant to the choice of global coordinate systems, while remaining fully compatible with existing RoPE-optimized
attention kernels. We evaluate URoPE as a plug-in positional encoding for
transformer architectures across a diverse set of tasks, including novel view synthesis, 3D object detection, object tracking, and depth estimation, covering 2D-2D, 2D-3D, and temporal scenarios. Experiments show that URoPE consistently improves the performance of
transformer-based models across all tasks, demonstrating its effectiveness and generality for
geometric reasoning. Our code is available on our project website: https://urope-pe.github.io/.
Read the full abstract ▾
All authors Yichen Xie, Depu Meng, Chensheng Peng, Yihan Hu, Quentin Herau, Masayoshi Tomizuka, Wei Zhan
Computer Vision
DeltaSeg: Tiered Attention and Deep Delta Learning for Multi-Class Structural Defect Segmentation
Enrique Hernandez Noguera, Md Meftahul Ferdaus, Elias Ioup et al. (+1 more)
Automated
segmentation of structural defects from visual inspection imagery remains challenging due to the diversity of damage types, extreme class imbalance, and the need for precise boundary delineation. This paper presents DeltaSeg, a U-shaped encoder-decoder architecture with a tiered
attention strategy that integrates Squeeze-and-Excitation (SE) channel
attention in the encoder, Coordinate
Attention at the bottleneck and decoder, and a novel Deep Delta
Attention (DDA) mechanism in the skip connections. The encoder uses depthwise separable convolutions with dilated stages to maintain spatial resolution while expanding the receptive field. Atrous Spatial Pyramid Pooling (ASPP) at the bottleneck captures multi-scale context. The DDA module refines skip connections through a dual-path scheme combining a learned delta operator for nuisance feature suppression with spatial
attention gates conditioned on decoder signals. Deep supervision through multi-scale auxiliary heads further strengthens gradient flow and encourages semantically meaningful features at intermediate decoder stages. We evaluate DeltaSeg on two
datasets: the S2DS
dataset (7 classes) and the Culvert-Sewer Defect
Dataset (CSDD, 9 classes). Across both
benchmarks, DeltaSeg consistently outperforms 12 competing architectures including U-Net, SA-UNet, UNet3+, SegFormer, Swin-UNet, EGE-UNet, FPN, and Mobile-UNETR, demonstrating strong generalization across damage types, imaging conditions, and structural geometries.
Read the full abstract ▾
All authors Enrique Hernandez Noguera, Md Meftahul Ferdaus, Elias Ioup, Mahdi Abdelguerfi
Computer Vision
Match-Any-Events: Zero-Shot Motion-Robust Feature Matching Across Wide Baselines for Event Cameras
Ruijun Zhang, Hang Su, Kostas Daniilidis et al. (+1 more)
Event cameras have recently shown promising capabilities in instantaneous motion estimation due to their robustness to low light and fast motions. However, computing wide-baseline correspondence between two arbitrary views remains a significant challenge, since event appearance changes substantially with motion, and learning-based approaches are constrained by both scalability and limited wide-baseline supervision. We therefore introduce the first event matching model that achieves cross-
dataset wide-baseline correspondence in a zero-shot manner: a single model trained once is deployed on unseen
datasets without any target-domain fine-tuning or adaptation. To enable this capability, we introduce a motion-robust and computationally efficient
attention backbone that learns multi-timescale features from event streams, augmented with sparsity-aware event
token selection, making large-scale training on diverse wide-baseline supervision computationally feasible. To provide the supervision needed for wide-baseline generalization, we develop a robust event motion synthesis framework to generate large-scale event-matching
datasets with augmented viewpoints, modalities, and motions. Extensive experiments across multiple
benchmarks show that our framework achieves a 37.7% improvement over the previous best event feature matching methods. Code and data are available at: https://github.com/spikelab-jhu/Match-Any-Events.
Read the full abstract ▾
All authors Ruijun Zhang, Hang Su, Kostas Daniilidis, Ziyun Wang
Computer Vision
Autonomous Skeletal Landmark Localization towards Agentic C-Arm Control
Jay Jung, Ahmad Arrabi, Jax Luo et al. (+2 more)
Purpose: Automated C-arm positioning ensures timely treatment in patients requiring emergent interventions. When a conventional Deep Learning (DL) approach for C-arm control fails, clinicians must revert to manual operation, resulting in additional delays. Consequently, an agentic C-arm control framework based on
multimodal large
language models (MLLMs) is highly desirable, as it can incorporate clinician feedback and use
reasoning to make adjustments toward more accurate positioning. Skeletal landmark localization is essential for C-arm control, and we investigate adapting MLLMs for autonomous landmark localization. Methods: We used an annotated synthetic X-ray
dataset and a real X-ray
dataset. Each X-ray in both
datasets is paired with several skeletal landmarks. We fine-tuned two MLLMs and tasked them with retrieving the closest landmarks from each X-ray. Quantitative evaluations of landmark localization were performed and compared against a leading DL approach. We further conducted qualitative experiments demonstrating: (1) how an MLLM can correct an initially incorrect prediction through
reasoning, and (2) how the MLLM can sequentially navigate the C-arm toward a target location. Results: On both
datasets, fine-tuned MLLMs demonstrate competitive performance across all localization tasks when compared with the DL approach. In the qualitative experiments, the MLLMs provide evidence of
reasoning and spatial awareness. Conclusion: This study shows that fine-tuned MLLMs achieve accurate skeletal landmark localization and hold promise for agentic autonomous C-arm control. Our code is available athttps://github.com/marszzibros/C-arm-localization-LLMs.git
Read the full abstract ▾
All authors Jay Jung, Ahmad Arrabi, Jax Luo, Scott Raymond, Safwan Wshah
Computer Vision
Colour Extraction Pipeline for Odonates using Computer Vision
Megan Mirnalini Sundaram Rajaraman, Fons J. Verbeek, Vincent J. Kalkman et al. (+1 more)
The correlation between insect morphological traits and
climate has been documented in physiological studies, but such studies remain limited by the time-consuming nature of the data analysis. In particular, the open source
datasets often lack annotations of species' morphological traits, making dedicated annotations campaigns necessary; these efforts are typically local in scale and costly. In this paper, we propose a pipeline to identify and segment body parts of Odonates (dragonflies and damselflies) using deep neural networks, with the ultimate goal of extracting body parts' colouration. The pipeline is trained on a limited annotated
dataset and refined with pseudo supervised data. We show that, by using open source images from citizen science platforms, our approach can segment each visible subject (Odonates) into head, thorax, abdomen, and wings and then extract a colour palette for each body part. This will enable large-scale statistical analysis of ecological correlations (e.g., between colouration and
climate change, habitat loss, or geolocation) which are crucial for quantifying and assessing ecosystem biodiversity status.
Read the full abstract ▾
All authors Megan Mirnalini Sundaram Rajaraman, Fons J. Verbeek, Vincent J. Kalkman, Rita Pucci
Computer Vision · Cross-listed from eess.IV
A Controlled Benchmark of Visual State-Space Backbones with Domain-Shift and Boundary Analysis for Remote-Sensing Segmentation
Nichula Wasalathilaka, Dineth Perera, Oshadha Samarakoon et al. (+4 more)
Visual state-space models (SSMs) are increasingly promoted as efficient alternatives to
Vision Transformers, yet their practical advantages remain unclear under fair comparison because existing studies rarely isolate encoder effects from decoder and training choices. We present a strictly controlled
benchmark of representative visual SSM families, including VMamba, MambaVision, and Spatial-Mamba, for remote-sensing semantic
segmentation, in which only the encoder varies across experiments. Evaluated on LoveDA and ISPRS Potsdam under a unified 4-stage feature interface and a fixed lightweight decoder, the
benchmark reveals three main findings, intra-family scaling yields only modest gains, cross-domain generalization is strongly asymmetric, and boundary delineation is the dominant failure mode under distribution shift. Although visual SSMs achieve favorable accuracy-efficiency trade-offs relative to the controlled CNN and
Transformer baselines considered here, the results suggest that future improvements are more likely to come from robustness-oriented design and boundary-aware decoding than from encoder scaling alone. By isolating encoder behavior under a unified and reproducible protocol, this study establishes a practical reference
benchmark for the design and evaluation of future Mamba-based
segmentation backbones
Read the full abstract ▾
All authors Nichula Wasalathilaka, Dineth Perera, Oshadha Samarakoon, Buddhi Wijenayake, Roshan Godaliyadda, Vijitha Herath, Parakrama Ekanayake
Computer Vision
Align then Refine: Text-Guided 3D Prostate Lesion Segmentation
Cuiling Sun, Linkai Peng, Adam Murphy et al. (+9 more)
Automated 3D
segmentation of prostate lesions from biparametric MRI (bp-MRI) is essential for reliable algorithmic analysis, but achieving high precision remains challenging. Volumetric methods must combine multiple modalities while ensuring anatomical consistency, but current models struggle to integrate cross-modal information reliably. While
vision-
language models (VLMs) are replacing the currently used architectural designs, they still lack the fine-grained, lesion-level semantics required for effective localized guidance. To address these limitations, we propose a new multi-encoder U-Net architecture incorporating three key innovations: (1) an
alignment loss that enhances foreground text-image similarity to inject lesion semantics; (2) a heatmap loss that calibrates the similarity map and suppresses spurious background activations; and (3) a final-stage, confidence-gated multi-head cross-
attention refiner that performs localized boundary edits in high-confidence regions. A phase-scheduled training regime stabilizes the optimization of these components. Our method consistently outperforms prior approaches, establishing a new state-of-the-art on the PI-CAI
dataset through enhanced multi-modal fusion and localized text guidance. Our code is available at https://github.com/NUBagciLab/Prostate-Lesion-
Segmentation.
Read the full abstract ▾
All authors Cuiling Sun, Linkai Peng, Adam Murphy, Elif Keles, Hiten D. Patel, Ashley Ross, Frank Miller, Baris Turkbey, Andrea Mia Bejar, Halil Ertugrul Aktas, Gorkem Durak, Ulas Bagci
Computer Vision
DanceCrafter: Fine-Grained Text-Driven Controllable Dance Generation via Choreographic Syntax
Hang Yuan, Xiaolin Hu, Yan Wan et al. (+8 more)
Text-driven controllable dance generation remains under-explored, primarily due to the severe scarcity of high-quality
datasets and the inherent difficulty of articulating complex choreographies. Characterizing dance is particularly challenging owing to its intricate spatial dynamics, strong directionality, and the highly decoupled movements of distinct body parts. To overcome these bottlenecks, we bridge principles from dance studies, human anatomy, and biomechanics to propose Choreographic Syntax, a novel theoretical framework with a tailored annotation system. Grounded in this syntax, we combine professional dance archives with high-fidelity motion capture data to construct DanceFlow, the most fine-grained dance
dataset to date. It encompasses 41 hours of high-quality motions paired with 6.34 million words of detailed descriptions. At the model level, we introduce DanceCrafter, a tailored motion
transformer built upon the Momentum Human Rig. To circumvent optimization instabilities, we construct a continuous
manifold motion representation paired with a hybrid normalization strategy. Furthermore, we design an anatomy-aware loss to explicitly regulate the decoupled nature of body parts. Together, these adaptations empower DanceCrafter to achieve the high-fidelity and stable generation of complex dance sequences. Extensive evaluations and user studies demonstrate our state-of-the-art performance in motion quality, fine-grained controllability, and generation naturalness.
Read the full abstract ▾
All authors Hang Yuan, Xiaolin Hu, Yan Wan, Menglin Gao, Wenzhe Yu, Cong Huang, Fei Xu, Qing Li, Christina Dan Wang, Zhou Yu, Kai Chen
Computer Vision
MUA: Mobile Ultra-detailed Animatable Avatars
Heming Zhu, Guoxing Sun and Marc Habermann
Building photorealistic, animatable full-body digital humans remains a longstanding challenge in computer graphics and
vision. Recent advances in animatable avatar modeling have largely progressed along two directions: improving the fidelity of dynamic geometry and appearance, or reducing computational complexity to enable deployment on resource-constrained platforms, e.g., VR headsets. However, existing approaches fail to achieve both goals simultaneously: Ultra-high-fidelity avatars typically require substantial computation on server-class GPUs, whereas lightweight avatars often suffer from limited surface dynamics, reduced appearance details, and noticeable artifacts. To bridge this gap, we propose a novel animatable avatar representation, termed Wavelet-guided Multi-level Spatial Factorized Blendshapes, and a corresponding
distillation pipeline that transfers motion-aware clothing dynamics and fine-grained appearance details from a pre-trained ultra-high-quality avatar model into a compact, efficient representation. By coupling multi-level wavelet spectral decomposition with low-rank structural factorization in texture space, our method achieves up to 2000X lower computational cost and a 10X smaller model size than the original high-quality teacher avatar model, while preserving visually plausible dynamics and appearance details closely resemble those of the teacher model. Extensive comparisons with state-of-the-art methods show that our approach significantly outperforms existing avatar approaches designed for mobile settings and achieves comparable or superior rendering quality to most approaches that can only run on servers. Importantly, our representation substantially improves the practicality of high-fidelity avatars for immersive applications, achieving over 180 FPS on a desktop PC and real-time native on-device performance at 24 FPS on a standalone Meta Quest 3.
Read the full abstract ▾
Computer Vision
ReCap: Lightweight Referential Grounding for Coherent Story Visualization
Aditya Arora, Akshita Gupta, Pau Rodriguez et al. (+1 more)
Story Visualization aims to generate a sequence of images that faithfully depicts a textual narrative that preserve character identity, spatial configuration, and stylistic coherence as the narratives unfold. Maintaining such cross-frame consistency has traditionally relied on explicit
memory banks, architectural expansion, or auxiliary
language models, resulting in substantial parameter growth and inference overhead. We introduce ReCap, a lightweight consistency framework that improves character stability and visual fidelity without modifying the base
diffusion backbone. ReCap's CORE (COnditional frame REferencing) module treats anaphors, in our case pronouns, as visual anchors, activating only when characters are referred to by a pronoun and conditioning on the preceding frame to propagate visual identity. This selective design avoids unconditional cross-frame conditioning and introduces only 149K additional parameters, a fraction of the cost of
memory-bank and LLM-augmented approaches. To further stabilize identity, we incorporate SemDrift (Guided Semantic Drift Correction) applied only during training. When text is vague or referential, the denoiser lacks a visual anchor for identity-defining attributes, causing character appearance to drift across frames, SemDrift corrects this by aligning denoiser representations with pretrained DINOv3 visual
embeddings, enforcing semantic identity stability at zero inference cost. ReCap outperforms previous state-of-the-art, StoryGPT-V, on the two main
benchmarks for story visualization by 2.63% Character-Accuracy on FlintstonesSV and by 5.65% on PororoSV, establishing a new state-of-the-art character consistency on both
benchmarks. Furthermore, we extend story visualization to human-centric narratives derived from real films, demonstrating the capability of ReCap beyond stylized cartoon domains.
Read the full abstract ▾
All authors Aditya Arora, Akshita Gupta, Pau Rodriguez, Marcus Rohrbach
Computer Vision
T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability
Savya Khosla, Sethuraman T V, Aryan Chadha et al. (+2 more)
Despite recent progress,
vision-
language encoders struggle with two core limitations: (1) weak
alignment between
language and dense
vision features, which hurts tasks like open-vocabulary semantic
segmentation; and (2) high
token counts for fine-grained visual representations, which limits scalability to long
videos. This work addresses both limitations. We propose T-REN (Text-aligned Region Encoder Network), an efficient encoder that maps visual data to a compact set of text-aligned region-level representations (or region
tokens). T-REN achieves this through a lightweight network added on top of a frozen
vision backbone, trained to pool patch-level representations within each semantic region into region
tokens and align them with region-level text annotations. With only 3.7% additional parameters compared to the
vision-
language backbone, this design yields substantially stronger dense cross-modal understanding while reducing the
token count by orders of magnitude. Specifically, T-REN delivers +5.9 mIoU on ADE20K open-vocabulary
segmentation, +18.4% recall on COCO object-level text-image
retrieval, +15.6% recall on Ego4D
video object localization, and +17.6% mIoU on VSPW
video scene parsing, all while reducing
token counts by more than 24x for images and 187x for
videos compared to the patch-based
vision-
language backbone. The code and model are available at https://github.com/savya08/T-REN.
Read the full abstract ▾
All authors Savya Khosla, Sethuraman T V, Aryan Chadha, Alex Schwing, Derek Hoiem
Computer Vision
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
A. Sophia Koepke, Daniil Zverev, Shiry Ginosar et al. (+1 more)
The Platonic Representation Hypothesis posits that neural networks trained on different modalities (e.g., text and images) converge toward a shared representation of reality. If true, this has significant implications for whether modality choice matters at all. In this paper, we show that the evidence for this claim is substantially weaker than subsequent work suggests. The mutual $k$-nearest-neighbor metric used on 1024 text-image pairs in the original study captures only coarse structure. To keep the
alignment from collapsing as one scales up the data, $k$ has to grow proportionally, undercutting the argument for fine-grained representational convergence. The reported increase in
alignment with
language model strength saturates for recent models. Moreover, the one-to-one text-image pairing favors
alignment, while
alignment decreases with non-bijective data. We further find that image and text representations indeed share coarse semantic structure, but neither stronger
language models nor richer captions yield fine-grained
alignment. Thus,
multimodal representations share coarse structure without evidence of convergence to a shared representation – arguably, full representational convergence would require fine-grained
alignment.
Read the full abstract ▾
All authors A. Sophia Koepke, Daniil Zverev, Shiry Ginosar, Alexei A. Efros
Computer Vision
MultiWorld: Scalable Multi-Agent Multi-View Video World Models
Haoyu Wu, Jiwen Yu, Yingtian Zou et al. (+1 more)
Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or
agents. They are modeled as action-conditioned
video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-
agent scenarios and fail to capture the complex interactions inherent in real-world multi-
agent systems. We present MultiWorld, a unified framework for multi-
agent multi-view world modeling that enables accurate control of multiple
agents while maintaining multi-view consistency. We introduce the Multi-
Agent Condition Module to achieve precise multi-
agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of
agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-
robot manipulation tasks demonstrate that MultiWorld outperforms baselines in
video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/
Read the full abstract ▾
All authors Haoyu Wu, Jiwen Yu, Yingtian Zou, Xihui Liu
Computer Vision
Rui Qian, Chuanhang Deng, Qiang Huang et al. (+6 more)
Reasoning segmentation requires models to ground complex, implicit textual queries into precise pixel-level masks. Existing approaches rely on a single
segmentation token $\texttt{<SEG>}$, whose hidden state implicitly encodes both semantic
reasoning and spatial localization, limiting the model's ability to explicitly disentangle what to segment from where to segment. We introduce AnchorSeg, which reformulates
reasoning segmentation as a structured conditional generation process over image
tokens, conditioned on
language grounded query banks. Instead of compressing all semantic
reasoning and spatial localization into a single
embedding, AnchorSeg constructs an ordered sequence of query banks: latent
reasoning tokens that capture intermediate semantic states, and a
segmentation anchor
token that provides explicit spatial grounding. We model spatial conditioning as a factorized distribution over image
tokens, where the anchor query determines localization signals while contextual queries provide semantic modulation. To bridge
token-level predictions and pixel-level supervision, we propose
Token–Mask Cycle Consistency (TMCC), a bidirectional training objective that enforces
alignment across resolutions. By explicitly decoupling spatial grounding from semantic
reasoning through structured
language grounded query banks, AnchorSeg achieves state-of-the-art results on ReasonSeg test set (67.7% gIoU and 68.1% cIoU). All code and models are publicly available at https://github.com/rui-qian/AnchorSeg.
Read the full abstract ▾
All authors Rui Qian, Chuanhang Deng, Qiang Huang, Jian Xiong, Mingxuan Li, Yingbo Zhou, Wei Zhai, Jintao Chen, Dejing Dou
Computer Vision
SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy
Wei Yao, Haohan Ma, Hongwen Zhang et al. (+5 more)
Controllable cooperative
humanoid manipulation is a fundamental yet challenging problem for
embodied intelligence, due to severe data scarcity, complexities in multi-
agent coordination, and limited generalization across objects. In this paper, we present SynAgent, a unified framework that enables scalable and physically plausible cooperative manipulation by leveraging Solo-to-Cooperative
Agent Synergy to transfer skills from single-
agent human-object interaction to multi-
agent human-object-human scenarios. To maintain semantic integrity during motion transfer, we introduce an interaction-preserving retargeting method based on an Interact Mesh constructed via Delaunay tetrahedralization, which faithfully maintains spatial relationships among humans and objects. Building upon this refined data, we propose a single-
agent pretraining and adaptation paradigm that distills synergistic collaborative behaviors from abundant single-human data through decentralized training and multi-
agent PPO. Finally, we develop a trajectory-conditioned generative policy using a conditional VAE, trained via multi-teacher
distillation from motion imitation priors to achieve stable and controllable object-level trajectory execution. Extensive experiments demonstrate that SynAgent significantly outperforms existing baselines in both cooperative imitation and trajectory-conditioned control, while generalizing across diverse object geometries. Codes and data will be available after publication. Project Page: https://yw0208.github.io/synagent/
Read the full abstract ▾
All authors Wei Yao, Haohan Ma, Hongwen Zhang, Liangjun Xing, Zhile Yang, Yuanjun Guo, Yunlian Sun, Yebin Liu
Computer Vision
Advancing Vision Transformer with Enhanced Spatial Priors
Qihang Fan, Huaibo Huang, Mingrui Chen et al. (+2 more)
In recent years, the
Vision Transformer (ViT) has garnered significant
attention within the computer
vision community. However, the core component of ViT, Self-
Attention, lacks explicit spatial priors and suffers from quadratic computational complexity, limiting its applicability. To address these issues, we have proposed RMT, a robust
vision backbone with explicit spatial priors for general purposes. RMT utilizes Manhattan distance decay to introduce spatial information and employs a horizontal and vertical decomposition
attention method to model global information. Building on the strengths of RMT, Euclidean enhanced
Vision Transformer (EVT) is an expanded version that incorporates several key improvements. Firstly, EVT uses a more reasonable Euclidean distance decay to enhance the modeling of spatial information, allowing for a more accurate representation of spatial relationships compared to the Manhattan distance used in RMT. Secondly, EVT abandons the decomposed
attention mechanism featured in RMT and instead adopts a simpler spatially-independent grouping approach, providing the model with greater flexibility in controlling the number of
tokens within each group. By addressing these modifications, EVT offers a more sophisticated and adaptable approach to incorporating spatial priors into the Self-
Attention mechanism, thus overcoming some of the limitations associated with RMT and further enhancing its applicability in various computer
vision tasks. Extensive experiments on Image Classification, Object Detection, Instance
Segmentation, and Semantic
Segmentation demonstrate that EVT exhibits exceptional performance. Without additional training data, EVT achieves 86.6% top1-acc on ImageNet-1k.
Read the full abstract ▾
All authors Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, Ran He
Computer Vision
MetaCloak-JPEG: JPEG-Robust Adversarial Perturbation for Preventing Unauthorized DreamBooth-Based Deepfake Generation
Tanjim Rahaman Fardin, S M Zunaid Alam, Mahadi Hasan Fahim et al. (+1 more)
The rapid progress of subject-driven text-to-image synthesis, and in particular DreamBooth, has enabled a consent-free deepfake pipeline: an adversary needs only 4-8 publicly available face images to fine-tune a personalized
diffusion model and produce photorealistic harmful content. Current
adversarial face-protection systems – PhotoGuard, Anti-DreamBooth, and MetaCloak – perturb user images to disrupt surrogate fine-tuning, but all share a structural blindness: none backpropagates gradients through the JPEG compression pipeline that every major social-media platform applies before adversary access. Because JPEG quantization relies on round(), whose derivative is zero almost everywhere,
adversarial energy concentrates in high-frequency DCT bands that JPEG discards, eliminating 60-80% of the protective signal. We introduce MetaCloak-JPEG, which closes this gap by inserting a Differentiable JPEG (DiffJPEG) layer built on the Straight-Through
Estimator (STE): the forward pass applies standard JPEG compression, while the backward pass replaces round() with the identity. DiffJPEG is embedded in a JPEG-aware EOT distribution (~70% of augmentations include DiffJPEG) and a curriculum quality-factor schedule (QF: 95 to 50) inside a bilevel meta-learning loop. Under an l-inf perturbation budget of eps=8/255, MetaCloak-JPEG attains 32.7 dB PSNR, a 91.3% JPEG survival rate, and outperforms PhotoGuard on all 9 evaluated JPEG quality factors (9/9 wins, mean denoising-loss gain +0.125) within a 4.1 GB training-
memory budget.
Read the full abstract ▾
All authors Tanjim Rahaman Fardin, S M Zunaid Alam, Mahadi Hasan Fahim, Md Faysal Mahfuz
Computer Vision
UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models
Jiaqi Wang, Haoge Deng, Ting Pan et al. (+5 more)
Uniform Discrete
Diffusion Model (UDM) has recently emerged as a promising paradigm for discrete generative modeling; however, its integration with reinforcement learning remains largely unexplored. We observe that naively applying GRPO to UDM leads to training instability and marginal performance gains. To address this, we propose UDM-GRPO, the first framework to integrate UDM with RL. Our method is guided by two key insights: (i) treating the final clean sample as the action provides more accurate and stable optimization signals; and (ii) reconstructing trajectories via the
diffusion forward process better aligns probability paths with the pretraining distribution. Additionally, we introduce two strategies, Reduced-Step and CFG-Free, to further improve training efficiency. UDM-GRPO significantly improves base model performance across multiple T2I tasks. Notably, GenEval accuracy improves from $69\%$ to $96\%$ and PickScore increases from $20.46$ to $23.81$, achieving state-of-the-art performance in both continuous and discrete settings. On the OCR
benchmark, accuracy rises from $8\%$ to $57\%$, further validating the generalization ability of our method. Code is available at https://github.com/Yovecent/UDM-GRPO.
Read the full abstract ▾
All authors Jiaqi Wang, Haoge Deng, Ting Pan, Yang Liu, Chengyuan Wang, Fan Zhang, Yonggang Qi, Xinlong Wang
Computer Vision
S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models
Nitish Shukla, Surgan Jandial and Arun Ross
Vision-
Language Models (VLMs) have demonstrated remarkable progress in single-image understanding, yet effective
reasoning across multiple images remains challenging. We identify a critical capability gap in existing multi-image
alignment approaches: current methods focus primarily on localized
reasoning with pre-specified image indices (“Look at Image 3 and...”), bypassing the essential skills of global visual search and autonomous cross-image comparison. To address this limitation, we introduce a Simple-to-Hard (S2H) learning framework that systematically constructs multi-image preference data across three hierarchical
reasoning levels requiring an increasing level of capabilities: (1) single-image localized
reasoning, (2) multi-image localized comparison, and (3) global visual search. Unlike prior work that relies on model-specific attributes, such as
hallucinations or
attention heuristics, to generate preference pairs, our approach leverages
prompt-driven complexity to create chosen/rejected pairs that are applicable across different models. Through extensive evaluations on LLaVA and Qwen-VL models, we show that our diverse multi-image
reasoning data significantly enhances multi-image
reasoning performance, yielding significant improvements over baseline methods across
benchmarks. Importantly, our approach maintains strong single-image
reasoning performance while simultaneously strengthening multi-image understanding capabilities, thus advancing the state of the art for holistic visual preference
alignment.
Read the full abstract ▾
Computer Vision
Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
Jinghui Lu, Jiayi Guan, Zhijian Huang et al. (+47 more)
Chain-of-Thought (CoT)
reasoning has become a powerful driver of trajectory prediction in VLA-based autonomous driving, yet its autoregressive nature imposes a latency cost that is prohibitive for real-time deployment. Latent CoT methods attempt to close this gap by compressing
reasoning into continuous hidden states, but consistently fall short of their explicit counterparts. We suggest that this is due to purely linguistic latent representations compressing a symbolic abstraction of the world, rather than the causal dynamics that actually govern driving. Thus, we present OneVL (One-step latent
reasoning and planning with
Vision-
Language explanations), a unified VLA and World Model framework that routes
reasoning through compact latent
tokens supervised by dual auxiliary decoders. Alongside a
language decoder that reconstructs text CoT, we introduce a visual world model decoder that predicts future-frame
tokens, forcing the latent space to internalize the causal dynamics of road geometry,
agent motion, and environmental change. A three-stage training pipeline progressively aligns these latents with trajectory,
language, and visual objectives, ensuring stable joint optimization. In inference, the auxiliary decoders are discarded, and all latent
tokens are prefilled in a single parallel pass, matching the speed of answer-only prediction. Across four
benchmarks, OneVL becomes the first latent CoT method to surpass explicit CoT, delivering superior accuracy at answer-only latency. These results show that with world model supervision, latent CoT produces more generalizable representations than verbose
token-by-
token reasoning. Code has been open-sourced to the community. Project Page: https://xiaomi-
embodied-intelligence.github.io/OneVL
Read the full abstract ▾
All authors Jinghui Lu, Jiayi Guan, Zhijian Huang, Jinlong Li, Guang Li, Lingdong Kong, Yingyan Li, Han Wang, Shaoqing Xu, Yuechen Luo, Fang Li, Chenxu Dang, Junli Wang, Tao Xu, Jing Wu, Jianhua Wu, Xiaoshuai Hao, Wen Zhang, Tianyi Jiang, Lingfeng Zhang, Lei Zhou, Yingbo Tang, Jie Wang, Yinfeng Gao, Xizhou Bu, Haochen Tian, Yihang Qiu, Feiyang Jia, Lin Liu, Yigu Ge, Hanbing Li, Yuannan Shen, Jianwei Cui, Hongwei Xie, Bing Wang, Haiyang Sun, Jingwei Zhao, Jiahui Huang, Pei Liu, Zeyu Zhu, Yuncheng Jiang, Zibin Guo, Chuhong Gong, Hanchao Leng, Kun Ma, Naiyan Wang, Guang Chen, Kuiyuan Yang, Hangjun Ye, Long Chen
Computer Vision
XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environments
Kangan Qian, ChuChu Xie, Yang Zhong et al. (+13 more)
Vision-
Language-Action (VLA) models drive next-generation autonomous systems, but training them requires scalable, high-quality annotations from complex environments. Current cloud pipelines rely on generic
vision-
language models (VLMs) that lack
geometric reasoning and domain semantics due to their 2D image-text pretraining. To address this mismatch, we propose XEmbodied, a cloud-side foundation model that endows VLMs with intrinsic 3D
geometric awareness and interaction with physical cues (e.g., occupancy grids, 3D boxes). Instead of treating geometry as auxiliary input, XEmbodied integrates
geometric representations via a structured 3D Adapter and distills physical signals into context
tokens using an Efficient Image-
Embodied Adapter. Through progressive domain curriculum and reinforcement learning post-training, XEmbodied preserves general capabilities while demonstrating robust performance across 18 public
benchmarks. It significantly improves spatial
reasoning, traffic semantics,
embodied affordance, and out-of-distribution generalization for large-scale scenario mining and
embodied VQA.
Read the full abstract ▾
All authors Kangan Qian, ChuChu Xie, Yang Zhong, Jingrui Pang, Siwen Jiao, Sicong Jiang, Zilin Huang, Yunlong Wang, Kun Jiang, Mengmeng Yang, Hao Ye, Guanghao Zhang, Hangjun Ye, Guang Chen, Long Chen, Diange Yang
Computer Vision
SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection
Hao Vo, Khoa Vo, Thinh Phan et al. (+5 more)
Camera-only 3D object detection has emerged as a cost-effective and scalable alternative to
LiDAR for autonomous driving, yet existing methods primarily prioritize overall performance while overlooking the severe long-tail imbalance inherent in real-world
datasets. In practice, many rare but safety-critical categories such as children, strollers, or emergency vehicles are heavily underrepresented, leading to biased learning and degraded performance. This challenge is further exacerbated by pronounced inter-class ambiguity (e.g., visually similar subclasses) and substantial intra-class diversity (e.g., objects varying widely in appearance, scale, pose, or context), which together hinder reliable long-tail recognition. In this work, we introduce SemLT3D, a Semantic-Guided Expert
Distillation framework designed to enrich the representation space for underrepresented classes through semantic priors. SemLT3D consists of: (1) a
language-guided mixture-of-experts module that routes 3D queries to specialized experts according to their semantic affinity, enabling the model to better disentangle confusing classes and specialize on tail distributions; and (2) a semantic projection
distillation pipeline that aligns 3D queries with CLIP-informed 2D semantics, producing more coherent and discriminative features across diverse visual manifestations. Although motivated by long-tail imbalance, the semantically structured learning in SemLT3D also improves robustness under broader appearance variations and challenging corner cases, offering a principled step toward more reliable camera-only 3D perception.
Read the full abstract ▾
All authors Hao Vo, Khoa Vo, Thinh Phan, Ngo Xuan Cuong, Gianfranco Doretto, Hien Nguyen, Anh Nguyen, Ngan Le
Computer Vision
Asset Harvester: Extracting 3D Assets from Autonomous Driving Logs for Simulation
Tianshi Cao, Jiawei Ren, Yuxuan Zhang et al. (+12 more)
Closed-loop simulation is a core component of autonomous vehicle (AV) development, enabling scalable testing, training, and safety validation before real-world deployment. Neural scene reconstruction converts driving logs into interactive 3D environments for simulation, but it does not produce complete 3D object assets required for
agent manipulation and large-viewpoint novel-view synthesis. To address this challenge, we present Asset Harvester, an image-to-3D model and end-to-end pipeline that converts sparse, in-the-wild object observations from real driving logs into complete, simulation-ready assets. Rather than relying on a single model component, we developed a system-level design for real-world AV data that combines large-scale curation of object-centric training tuples, geometry-aware preprocessing across heterogeneous sensors, and a robust training recipe that couples sparse-view-conditioned multiview generation with 3D Gaussian lifting. Within this system, SparseViewDiT is explicitly designed to address limited-angle views and other real-world data challenges. Together with hybrid data curation, augmentation, and self-
distillation, this system enables scalable conversion of sparse AV object observations into reusable 3D assets.
Read the full abstract ▾
All authors Tianshi Cao, Jiawei Ren, Yuxuan Zhang, Jaewoo Seo, Jiahui Huang, Shikhar Solanki, Haotian Zhang, Mingfei Guo, Haithem Turki, Muxingzi Li, Yue Zhu, Sipeng Zhang, Zan Gojcic, Sanja Fidler, Kangxue Yin