Computer Vision
Transferable Multi-Bit Watermarking Across Frozen Diffusion Models via Latent Consistency Bridges
Hong-Hanh Nguyen-Le, Van-Tuan Tran, Thuc D. Nguyen et al. (+1 more)
As generative AI advances, global governance frameworks increasingly mandate verifiable content provenance. However, existing watermarking techniques face a critical policy-to-technology disconnect: sampling-based methods require computationally prohibitive inversion, while fine-tuning approaches are tethered to specific model checkpoints, hindering standardized, cross-model oversight. To bridge this gap, we introduce DiffMark, a plug-and-play multi-bit watermarking framework. DiffMark embeds a persistent, learned perturbation into every denoising step of a frozen
diffusion model, accumulating a recoverable signal in the final latent space. To enable efficient training through the frozen network, we utilize Latent Consistency Models (LCMs) as a differentiable training bridge. DiffMark achieves 64-bit extraction in a single 16.4 ms forward pass, which is a $45\times$ speed-up over inversion baselines. By enabling per-image key flexibility and cross-architecture transferability without retraining, DiffMark provides the practical, scalable technical tooling necessary to operationalize user accountability and enforce emerging AI governance mandates.
Read the full abstract ▾
All authors Hong-Hanh Nguyen-Le, Van-Tuan Tran, Thuc D. Nguyen, Nhien-An Le-Khac
Computer Vision
InjectFlow: Weak Guides Strong via Orthogonal Injection for Flow Matching
Dayu Wang, Jiaye Yang, Weikang Li et al. (+2 more)
Flow Matching (FM) has recently emerged as a leading approach for high-fidelity visual generation, offering a robust continuous-time alternative to ordinary differential equation (ODE) based models. However, despite their success, FM models are highly sensitive to
dataset biases, which cause severe semantic degradation when generating out-of-distribution or minority-class samples. In this paper, we provide a rigorous mathematical formalization of the “Bias
Manifold” within the FM framework. We identify that this performance drop is driven by conditional expectation smoothing, a mechanism that inevitably leads to trajectory lock-in during inference. To resolve this, we introduce InjectFlow, a novel, training-free method by injecting orthogonal semantics during the initial velocity field computation, without requiring any changes to the random seeds. This design effectively prevents the latent drift toward majority modes while maintaining high generative quality. Extensive experiments demonstrate the effectiveness of our approach. Notably, on the GenEval
dataset, InjectFlow successfully fixes 75% of the
prompts that standard flow matching models fail to generate correctly. Ultimately, our theoretical analysis and
algorithm provide a ready-to-use solution for building more fair and robust visual foundation models.
Read the full abstract ▾
All authors Dayu Wang, Jiaye Yang, Weikang Li, Jiahui Liang, Yang Li
Computer Vision
HSI Image Enhancement Classification Based on Knowledge Distillation: A Study on Forgetting
Songfeng Zhu
In incremental classification tasks for hyperspectral images, catastrophic forgetting is an unavoidable challenge. While
memory recall methods can mitigate this issue, they heavily rely on samples from old categories. This paper proposes a teacher-based knowledge retention method for incremental image classification. It alleviates model forgetting of old category samples by utilizing incremental category samples, without depending on old category samples. Additionally, this paper introduces a mask-based partial category knowledge
distillation algorithm. By decoupling knowledge
distillation, this approach filters out potentially misleading information that could misguide the student model, thereby enhancing overall accuracy. Comparative and ablation experiments demonstrate the proposed method's robust performance.
Read the full abstract ▾
Computer Vision
dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3
Saikat Dutta, Biplab Banerjee and Hamid Rezatofighi
Open-Vocabulary Semantic
Segmentation (OVSS) assigns pixel-level labels from an open set of text-defined categories, demanding reliable generalization to unseen classes at inference. Although modern
vision-
language models (VLMs) support strong open-vocabulary recognition, their representations learned through global contrastive objectives remain suboptimal for dense prediction, prompting many OVSS methods to depend on limited adaptation or refinement of image-text similarity maps. This, in turn, restricts spatial precision and robustness in complex, cluttered scenes. We introduce dinov3.seg, extending dinov3.txt into a dedicated framework for OVSS. Our contributions are four-fold. First, we design a task-specific architecture tailored to this backbone, systematically adapting established design principles from prior open-vocabulary
segmentation work. Second, we jointly leverage text
embeddings aligned with both the global [CLS]
token and local patch-level visual features from ViT-based encoder, effectively combining semantic discrimination with fine-grained spatial locality. Third, unlike prior approaches that rely primarily on post hoc similarity refinement, we perform early refinement of visual representations prior to image-text interaction, followed by late refinement of the resulting image-text correlation features, enabling more accurate and robust dense predictions in cluttered scenes. Finally, we propose a high-resolution local-global inference strategy based on sliding-window aggregation, which preserves spatial detail while maintaining global context. We conduct extensive experiments on five widely adopted OVSS
benchmarks to evaluate our approach. The results demonstrate its effectiveness and robustness, consistently outperforming current state-of-the-art methods.
Read the full abstract ▾
Computer Vision
SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions
Vasco Xu, Brian Chen, Eric J. Gonzalez et al. (+4 more)
Mid-air gestures in Extended Reality (XR) often cause fatigue and imprecision. Surface-based interactions offer improved accuracy and comfort, but current
egocentric vision methods struggle due to hand tracking challenges and unreliable surface plane estimation. We introduce SurfaceXR, a sensor fusion approach combining headset-based hand tracking with smartwatch IMU data to enable robust inputs on everyday surfaces. Our insight is that these modalities are complementary: hand tracking provides 3D positional data while IMUs capture high-frequency motion. A 21-participant study validates SurfaceXR's effectiveness for touch tracking and 8-class gesture recognition, demonstrating significant improvements over single-modality approaches.
Read the full abstract ▾
All authors Vasco Xu, Brian Chen, Eric J. Gonzalez, Andrea Colaço, Henry Hoffmann, Mar Gonzalez-Franco, Karan Ahuja
Computer Vision
Recognising BSL Fingerspelling in Continuous Signing Sequences
Alyssa Chan, Taein Kwon and Andrew Zisserman
Fingerspelling is a critical component of British Sign
Language (BSL), used to spell proper names, technical terms, and words that lack established lexical signs. Fingerspelling recognition is challenging due to the rapid pace of signing and common letter omissions by native signers, while existing BSL fingerspelling
datasets are either small in scale or temporally and letter-wise inaccurate. In this work, we introduce a new large-scale BSL fingerspelling
dataset, FS23K, constructed using an iterative annotation framework. In addition, we propose a fingerspelling recognition model that explicitly accounts for bi-manual interactions and mouthing cues. As a result, with refined annotations, our approach halves the character error rate (CER) compared to the prior state of the art on fingerspelling recognition. These findings demonstrate the effectiveness of our method and highlight its potential to support future research in sign
language understanding and scalable, automated annotation pipelines. The project page can be found at https://taeinkwon.com/projects/fs23k/.
Read the full abstract ▾
Computer Vision
ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding
Oishi Banerjee, Sung Eun Kim, Alexandra N. Willauer et al. (+9 more)
Everyday photographs taken with ordinary cameras are already widely used in telemedicine and other online health conversations, yet no comprehensive
benchmark evaluates whether
vision-
language models can interpret their medical content. Analyzing these images requires both fine-grained natural image understanding and domain-specific medical
reasoning, a combination that challenges both general-purpose and specialized models. We introduce ReXInTheWild, a
benchmark of 955 clinician-verified multiple-choice questions spanning seven clinical topics across 484 photographs sourced from the biomedical literature. When evaluated on ReXInTheWild, leading
multimodal large
language models show substantial performance variation: Gemini-3 achieves 78% accuracy, followed by Claude Opus 4.5 (72%) and GPT-5 (68%), while the medical specialist model MedGemma reaches only 37%. A systematic error analysis also reveals four categories of common errors, ranging from low-level
geometric errors to high-level
reasoning failures and requiring different mitigation strategies. ReXInTheWild provides a challenging, clinically grounded
benchmark at the intersection of natural image understanding and medical
reasoning. The
dataset is available on HuggingFace.
Read the full abstract ▾
All authors Oishi Banerjee, Sung Eun Kim, Alexandra N. Willauer, Julius M. Kernbach, Abeer Rihan Alomaish, Reema Abdulwahab S. Alghamdi, Hassan Rayhan Alomaish, Mohammed Baharoon, Xiaoman Zhang, Julian Nicolas Acosta, Christine Zhou, Pranav Rajpurkar
Computer Vision
Sheng Lu, Hao Chen, Rui Yin et al. (+3 more)
Recent
vision-
language models (VLMs) have shown strong generalization and
multimodal reasoning abilities in natural domains. However, their application to medical diagnosis remains limited by the lack of comprehensive and structured
datasets that capture real clinical workflows. To advance the development of VLMs for clinical applications, particularly in gastric cancer, we introduce Gastric-X, a large-scale
multimodal benchmark for gastric cancer analysis providing 1.7K cases. Each case in Gastric-X includes paired resting and dynamic CT scans, endoscopic image, a set of structured biochemical indicators, expert-authored diagnostic notes, and bounding box annotations of tumor regions, reflecting realistic clinical conditions. We systematically examine the capability of recent VLMs on five core tasks: Visual Question Answering (VQA), report generation, cross-modal
retrieval, disease classification, and lesion localization. These tasks simulate critical stages of clinical workflow, from visual understanding and
reasoning to
multimodal decision support. Through this evaluation, we aim not only to assess model performance but also to probe the nature of VLM understanding: Can current VLMs meaningfully correlate biochemical signals with spatial tumor features and textual reports? We envision Gastric-X as a step toward aligning machine intelligence with the cognitive and evidential
reasoning processes of physicians, and as a resource to inspire the development of next-generation medical VLMs.
Read the full abstract ▾
All authors Sheng Lu, Hao Chen, Rui Yin, Juyan Ba, Yu Zhang, Yuanzhe Li
Computer Vision
FedAgain: A Trust-Based and Robust Federated Learning Strategy for an Automated Kidney Stone Identification in Ureteroscopy
Ivan Reyes-Amezcua, Francisco Lopez-Tiro, Clément Larose et al. (+3 more)
The reliability of artificial intelligence (AI) in medical imaging critically depends on its robustness to heterogeneous and corrupted images acquired with diverse devices across different hospitals which is highly challenging. Therefore, this paper introduces FedAgain, a trust-based
Federated Learning (
Federated Learning) strategy designed to enhance robustness and generalization for automated kidney stone identification from endoscopic images. FedAgain integrates a dual trust mechanism that combines
benchmark reliability and model divergence to dynamically weight client contributions, mitigating the impact of noisy or
adversarial updates during aggregation. The framework enables the training of collaborative models across multiple institutions while preserving data
privacy and promoting stable convergence under real-world conditions. Extensive experiments across five
datasets, including two canonical
benchmarks (MNIST and CIFAR-10), two private multi-institutional kidney stone
datasets, and one public
dataset (MyStone), demonstrate that FedAgain consistently outperforms standard
Federated Learning baselines under non-identically and independently distributed (non-IID) data and corrupted-client scenarios. By maintaining diagnostic accuracy and performance stability under varying conditions, FedAgain represents a practical advance toward reliable,
privacy-preserving, and clinically deployable
federated AI for medical imaging.
Read the full abstract ▾
All authors Ivan Reyes-Amezcua, Francisco Lopez-Tiro, Clément Larose, Christian Daul, Andres Mendez-Vazquez, Gilberto Ochoa-Ruiz
Computer Vision
Vision Tiny Recursion Model (ViTRM): Parameter-Efficient Image Classification via Recursive State Refinement
Ange-Clément Akazan, Abdoulaye Koroko, Verlon Roel Mbingui et al. (+3 more)
The success of deep learning in computer
vision has been driven by models of increasing scale, from deep Convolutional Neural Networks (CNN) to large
Vision Transformers (ViT). While effective, these architectures are parameter-intensive and demand significant computational resources, limiting deployment in resource-constrained environments. Inspired by Tiny Recursive Models (TRM), which show that small recursive networks can solve complex
reasoning tasks through iterative state refinement, we introduce the
Vision Tiny Recursion Model (ViTRM): a parameter-efficient architecture that replaces the $L$-layer ViT encoder with a single tiny $k$-layer block ($k{=}3$) applied recursively $N$ times. Despite using up to $6 \times $ and $84 \times$ fewer parameters than CNN based models and ViT respectively, ViTRM maintains competitive performance on CIFAR-10 and CIFAR-100. This demonstrates that recursive computation is a viable, parameter-efficient alternative to architectural depth in
vision.
Read the full abstract ▾
All authors Ange-Clément Akazan, Abdoulaye Koroko, Verlon Roel Mbingui, Choukouriyah Arinloye, Hassan Fifen, Rose Bandolo
Computer Vision · Cross-listed from cs.AI
Teaching an Agent to Sketch One Part at a Time
Xiaodan Du, Ruize Xu, David Yunis et al. (+2 more)
We develop a method for producing vector sketches one part at a time. To do this, we train a multi-modal
language model-based
agent using a novel multi-turn process-reward reinforcement learning following supervised fine-tuning. Our approach is enabled by a new
dataset we call ControlSketch-Part, containing rich part-level annotations for sketches, obtained using a novel, generic automatic annotation pipeline that segments vector sketches into semantic parts and assigns paths to parts with a structured multi-stage labeling process. Our results indicate that incorporating structured part-level data and providing
agent with the visual feedback through the process enables interpretable, controllable, and locally editable text-to-vector sketch generation.
Read the full abstract ▾
All authors Xiaodan Du, Ruize Xu, David Yunis, Yael Vinker, Greg Shakhnarovich
Computer Vision
VeloxNet: Efficient Spatial Gating for Lightweight Embedded Image Classification
Md Meftahul Ferdaus, Elias Ioup, Mahdi Abdelguerfi et al. (+4 more)
Deploying deep learning models on embedded devices for tasks such as aerial disaster monitoring and infrastructure inspection requires architectures that balance accuracy with strict constraints on model size,
memory, and latency. This paper introduces VeloxNet, a lightweight CNN architecture that replaces SqueezeNet's fire modules with gated multi-layer perceptron (gMLP) blocks for embedded image classification. Each gMLP block uses a spatial gating unit (SGU) that applies learned spatial projections and multiplicative gating, enabling the network to capture spatial dependencies across the full feature map in a single layer. Unlike fire modules, which are limited to local receptive fields defined by small convolutional kernels, the SGU provides global spatial modeling at each layer with fewer parameters. We evaluate VeloxNet on three aerial image
datasets: the Aerial Image Database for Emergency Response (AIDER), the Comprehensive Disaster
Dataset (CDD), and the Levee Defect
Dataset (LDD), comparing against eleven baselines including MobileNet variants, ShuffleNet, EfficientNet, and recent
vision transformers. VeloxNet reduces the parameter count by 46.1% relative to SqueezeNet (from 740,970 to 399,366) while improving weighted F1 scores by 6.32% on AIDER, 30.83% on CDD, and 2.51% on LDD. These results demonstrate that substituting local convolutional modules with spatial gating blocks can improve both classification accuracy and parameter efficiency for resource-constrained deployment. The source code will be made publicly available upon acceptance of the paper.
Read the full abstract ▾
All authors Md Meftahul Ferdaus, Elias Ioup, Mahdi Abdelguerfi, Anton Netchaev, Steven Sloan, Ken Pathak, Kendall N. Niles
Computer Vision
Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
Myeongkyun Kang, Soopil Kim, Xiaoxiao Li et al. (+1 more)
Large
vision language models (LVLMs) have demonstrated impressive performance across a wide range of tasks. These capabilities largely stem from visual instruction tuning, which fine-tunes models on
datasets consisting of curated image-instruction-output triplets. However, in the medical domain, constructing large-scale, high-quality instruction
datasets is particularly challenging due to the need for specialized expert knowledge. To address this issue, we propose an instruction-free tuning approach that reduces reliance on handcrafted instructions, leveraging only image-description pairs for fine-tuning. Specifically, we introduce a momentum proxy instruction as a replacement for curated text instructions, which preserves the instruction-following capability of the pre-trained LVLM while promoting updates to parameters that remain valid during inference. Consequently, the fine-tuned LVLM can flexibly respond to domain-specific instructions, even though explicit instructions are absent during fine-tuning. Additionally, we incorporate a response shuffling strategy to mitigate the model's over-reliance on previous words, facilitating more effective fine-tuning. Our approach achieves state-of-the-art accuracy on multiple-choice visual question answering tasks across SKINCON, WBCAtt, CBIS, and MIMIC-CXR
datasets, significantly enhancing the fine-tuning efficiency of LVLMs in medical domains.
Read the full abstract ▾
All authors Myeongkyun Kang, Soopil Kim, Xiaoxiao Li, Sang Hyun Park
Computer Vision
Narrative Aligned Long Form Video Question Answering
Rahul Jain, Keval Doshi, Burak Uzkent et al. (+1 more)
Recent progress in
multimodal large
language models (MLLMs) has led to a surge of
benchmarks for long-
video reasoning. However, most existing
benchmarks rely on localized cues and fail to capture narrative
reasoning, the ability to track intentions, connect distant events, and reconstruct causal chains across an entire movie. We introduce NA-VQA, a
benchmark designed to evaluate deep temporal and narrative
reasoning in long-form
videos. NA-VQA contains 88 full-length movies and 4.4K open-ended question-answer pairs, each grounded in multiple evidence spans labeled as Short, Medium, or Far to assess long-range dependencies. By requiring generative, multi-scene answers, NA-VQA tests whether models can integrate dispersed narrative information rather than rely on shallow pattern matching. To address the limitations of existing approaches, we propose
Video-NaRA, a narrative-centric framework that builds event-level chains and stores them in a structured
memory for
retrieval during
reasoning. Extensive experiments show that state-of-the-art MLLMs perform poorly on questions requiring far-range evidence, highlighting the need for explicit narrative modeling.
Video-NaRA improves long-range
reasoning performance by up to 3 percent, demonstrating its effectiveness in handling complex narrative structures. We will release NA-VQA upon publication.
Read the full abstract ▾
All authors Rahul Jain, Keval Doshi, Burak Uzkent, Garin Kessler
Computer Vision
ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models
Thomas De Min, Subhankar Roy, Stéphane Lathuilière et al. (+2 more)
Effective collaboration begins with knowing when to ask for help. For example, when trying to identify an occluded object, a human would ask someone to remove the obstruction. Can MLLMs exhibit a similar "proactive" behavior by requesting simple user interventions? To investigate this, we introduce ProactiveBench, a
benchmark built from seven repurposed
datasets that tests proactiveness across different tasks such as recognizing occluded objects, enhancing image quality, and interpreting coarse sketches. We evaluate 22 MLLMs on ProactiveBench, showing that (i) they generally lack proactiveness; (ii) proactiveness does not correlate with model capacity; (iii) "hinting" at proactiveness yields only marginal gains. Surprisingly, we found that conversation histories and in-context learning introduce negative biases, hindering performance. Finally, we explore a simple fine-tuning strategy based on reinforcement learning: its results suggest that proactiveness can be learned, even generalizing to unseen scenarios. We publicly release ProactiveBench as a first step toward building proactive
multimodal models.
Read the full abstract ▾
All authors Thomas De Min, Subhankar Roy, Stéphane Lathuilière, Elisa Ricci, Massimiliano Mancini
Computer Vision
In-the-Wild Camouflage Attack on Vehicle Detectors through Controllable Image Editing
Xiao Fang, Yiming Gong, Stanislav Panev et al. (+4 more)
Deep neural networks (DNNs) have achieved remarkable success in computer
vision but remain highly vulnerable to
adversarial attacks. Among them, camouflage attacks manipulate an object's visible appearance to deceive detectors while remaining stealthy to humans. In this paper, we propose a new framework that formulates vehicle camouflage attacks as a conditional image-editing problem. Specifically, we explore both image-level and scene-level camouflage generation strategies, and fine-tune a ControlNet to synthesize camouflaged vehicles directly on real images. We design a unified objective that jointly enforces vehicle structural fidelity, style consistency, and
adversarial effectiveness. Extensive experiments on the COCO and LINZ
datasets show that our method achieves significantly stronger attack effectiveness, leading to more than 38% AP50 decrease, while better preserving vehicle structure and improving human-perceived stealthiness compared to existing approaches. Furthermore, our framework generalizes effectively to unseen black-box detectors and exhibits promising transferability to the physical world. Project page is available at https://humansensinglab.github.io/CtrlCamo
Read the full abstract ▾
All authors Xiao Fang, Yiming Gong, Stanislav Panev, Celso de Melo, Shuowen Hu, Shayok Chakraborty, Fernando De la Torre
Computer Vision
LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray
Myeongkyun Kang, Yanting Yang and Xiaoxiao Li
Fine-grained representation learning is crucial for
retrieval and phrase grounding in chest X-rays, where clinically relevant findings are often spatially confined. However, the lack of region-level supervision in contrastive models and the limited ability of large
vision language models to capture fine-grained representations in external validation lead to suboptimal performance on these tasks. To address these limitations, we propose Location-aware Fine-grained representation learning (LoFi), which jointly optimizes sigmoid, captioning, and location-aware captioning losses using a lightweight large
language model. The location-aware captioning loss enables region-level supervision through grounding and dense captioning objectives, thereby facilitating fine-grained representation learning. Building upon these representations, we integrate a fine-grained encoder into
retrieval-based in-context learning to enhance chest X-ray grounding across diverse settings. Extensive experiments demonstrate that our method achieves superior
retrieval and phrase grounding performance on MIMIC-CXR and PadChest-GR.
Read the full abstract ▾
Computer Vision
Factored Levenberg-Marquardt for Diffeomorphic Image Registration: An efficient optimizer for FireANTs
Rohit Jena, Pratik Chaudhari and James C. Gee
FireANTs introduced a novel Eulerian descent method for plug-and-play behavior with arbitrary optimizers adapted for diffeomorphic image registration as a test-time optimization problem, with a GPU-accelerated implementation. FireANTs uses Adam as its default optimizer for fast and more robust optimization. However, Adam requires storing state variables (i.e. momentum and squared-momentum estimates), each of which can consume significant
memory, prohibiting its use for significantly large images. In this work, we propose a modified Levenberg-Marquardt (LM) optimizer that requires only a single scalar damping parameter as optimizer state, that is adaptively tuned using a trust region approach. The resulting optimizer reduces
memory by up to 24.6% for large volumes, and retaining performance across all four
datasets. A single hyperparameter configuration tuned on
brain MRI transfers without modification to lung CT and cross-modal abdominal registration, matching or outperforming Adam on three of four
benchmarks. We also perform ablations on the effectiveness of using Metropolis-Hastings style rejection step to prevent updates that worsen the loss function.
Read the full abstract ▾
Computer Vision
AURORA: Adaptive Unified Representation for Robust Ultrasound Analysis
Ufaq Khan, L. D. M. S. Sai Teja, Ayuba Shakiru et al. (+4 more)
Ultrasound images vary widely across scanners, operators, and anatomical targets, which often causes models trained in one setting to generalize poorly to new hospitals and clinical conditions. The Foundation Model Challenge for Ultrasound Image Analysis (FMC-UIA) reflects this difficulty by requiring a single model to handle multiple tasks, including
segmentation, detection, classification, and landmark regression across diverse organs and
datasets. We propose a unified multi-task framework based on a
transformer visual encoder from the Qwen3-VL family. Intermediate
token features are projected into spatial feature maps and fused using a lightweight multi-scale feature pyramid, enabling both pixel-level predictions and global
reasoning within a shared representation. Each task is handled by a small task-specific prediction head, while training uses task-aware sampling and selective loss balancing to manage heterogeneous supervision and reduce task imbalance. Our method is designed to be simple to optimize and adaptable across a wide range of ultrasound analysis tasks. The performance improved from 67% to 85% on the validation set and achieved an average score of 81.84% on the official test set across all tasks. The code is publicly available at: https://github.com/saitejalekkala33/FMCUIA-ISBI.git
Read the full abstract ▾
All authors Ufaq Khan, L. D. M. S. Sai Teja, Ayuba Shakiru, Mai A. Shaaban, Yutong Xie, Muhammad Bilal, Muhammad Haris Khan
Computer Vision
Diffusion-Guided Semantic Consistency for Multimodal Heterogeneity
Jing Liu, Zhengliang Guo, Yan Wang et al. (+4 more)
Federated learning (FL) is severely challenged by non-independent and identically distributed (non-IID) client data, a problem that degrades global model performance, especially in
multimodal perception settings. Conventional methods often fail to address the underlying semantic discrepancies between clients, leading to suboptimal performance for multimedia systems requiring robust perception. To overcome this, we introduce SemanticFL, a novel framework that leverages the rich semantic representations of pre-trained
diffusion models to provide
privacy-preserving guidance for local training. Our approach leverages multi-layer semantic representations from a pre-trained Stable
Diffusion model (including VAE-encoded latents and U-Net hierarchical features) to create a shared latent space that aligns heterogeneous clients, facilitated by an efficient client-server architecture that offloads heavy computation to the server. A unified consistency mechanism, employing cross-modal contrastive learning, further stabilizes convergence. We conduct extensive experiments on
benchmarks including CIFAR-10, CIFAR-100, and TinyImageNet under diverse heterogeneity scenarios. Our results demonstrate that SemanticFL surpasses existing
federated learning approaches, achieving accuracy gains of up to 5.49% over FedAvg, validating its effectiveness in learning robust representations for heterogeneous and
multimodal data for perception tasks.
Read the full abstract ▾
All authors Jing Liu, Zhengliang Guo, Yan Wang, Xiaoguang Zhu, Yao Du, Zehua Wang, Victor C. M. Leung
Computer Vision
Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
Xianjin Wu, Dingkang Liang, Tianrui Feng et al. (+5 more)
While
Multimodal Large
Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained
geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex
geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale
video generation models. We posit that to synthesize temporally coherent
videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (
Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained
video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a
token-level adaptive gated fusion mechanism, we enrich MLLMs with dense
geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial
reasoning, and
embodied manipulation
benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at https://github.com/H-EmbodVis/VEGA-3D.
Read the full abstract ▾
All authors Xianjin Wu, Dingkang Liang, Tianrui Feng, Kui Xia, Yumeng Zhang, Xiaofan Li, Xiao Tan, Xiang Bai
Computer Vision
Matryoshka Gaussian Splatting
Zhilin Guo, Boqiao Zhang, Hakan Aktas et al. (+11 more)
The ability to render scenes at adjustable fidelity from a single model, known as level of detail (LoD), is crucial for practical deployment of 3D Gaussian Splatting (3DGS). Existing discrete LoD methods expose only a limited set of operating points, while concurrent continuous LoD approaches enable smoother scaling but often suffer noticeable quality degradation at full capacity, making LoD a costly design decision. We introduce Matryoshka Gaussian Splatting (MGS), a training framework that enables continuous LoD for standard 3DGS pipelines without sacrificing full-capacity rendering quality. MGS learns a single ordered set of Gaussians such that rendering any prefix, the first k splats, produces a coherent reconstruction whose fidelity improves smoothly with increasing budget. Our key idea is
stochastic budget training: each iteration samples a random splat budget and optimises both the corresponding prefix and the full set. This strategy requires only two forward passes and introduces no architectural modifications. Experiments across four
benchmarks and six baselines show that MGS matches the full-capacity performance of its backbone while enabling a continuous speed-quality trade-off from a single model. Extensive ablations on ordering strategies, training objectives, and model capacity further validate the designs.
Read the full abstract ▾
All authors Zhilin Guo, Boqiao Zhang, Hakan Aktas, Kyle Fogarty, Jeffrey Hu, Nursena Koprucu Aslan, Wenzhao Li, Canberk Baykal, Albert Miao, Josef Bengtson, Chenliang Zhou, Weihao Xia, Cristina Nader Vasconcelos, Cengiz Oztireli
Computer Vision
Cubic Discrete Diffusion: Discrete Visual Generation on High-Dimensional Representation Tokens
Yuqing Wang, Chuofan Ma, Zhijie Lin et al. (+7 more)
Visual generation with discrete
tokens has gained significant
attention as it enables a unified
token prediction paradigm shared with
language models, promising seamless
multimodal architectures. However, current discrete generation methods remain limited to low-dimensional latent
tokens (typically 8-32 dims), sacrificing the semantic richness essential for understanding. While high-dimensional pretrained representations (768-1024 dims) could bridge this gap, their discrete generation poses fundamental challenges. In this paper, we present Cubic Discrete
Diffusion (CubiD), the first discrete generation model for high-dimensional representations. CubiD performs fine-grained masking throughout the high-dimensional discrete representation – any dimension at any position can be masked and predicted from partial observations. This enables the model to learn rich correlations both within and across spatial positions, with the number of generation steps fixed at $T$ regardless of feature dimensionality, where $T \ll hwd$. On ImageNet-256, CubiD achieves state-of-the-art discrete generation with strong scaling behavior from 900M to 3.7B parameters. Crucially, we validate that these discretized
tokens preserve original representation capabilities, demonstrating that the same discrete
tokens can effectively serve both understanding and generation tasks. We hope this work will inspire future research toward unified
multimodal architectures. Code is available at: https://github.com/YuqingWang1029/CubiD.
Read the full abstract ▾
All authors Yuqing Wang, Chuofan Ma, Zhijie Lin, Yao Teng, Lijun Yu, Shuai Wang, Jiaming Han, Jiashi Feng, Yi Jiang, Xihui Liu
Computer Vision
MonoArt: Progressive Structural Reasoning for Monocular Articulated 3D Reconstruction
Haitian Li, Haozhe Xie, Junxiang Xu et al. (+3 more)
Reconstructing articulated 3D objects from a single image requires jointly inferring object geometry, part structure, and motion parameters from limited visual evidence. A key difficulty lies in the
entanglement between motion cues and object structure, which makes direct articulation regression unstable. Existing methods address this challenge through multi-view supervision,
retrieval-based assembly, or auxiliary
video generation, often sacrificing scalability or efficiency. We present MonoArt, a unified framework grounded in progressive structural
reasoning. Rather than predicting articulation directly from image features, MonoArt progressively transforms visual observations into canonical geometry, structured part representations, and motion-aware
embeddings within a single architecture. This structured
reasoning process enables stable and interpretable articulation inference without external motion templates or multi-stage pipelines. Extensive experiments on PartNet-Mobility demonstrate that OM achieves state-of-the-art performance in both reconstruction accuracy and inference speed. The framework further generalizes to robotic manipulation and articulated scene reconstruction.
Read the full abstract ▾
All authors Haitian Li, Haozhe Xie, Junxiang Xu, Beichen Wen, Fangzhou Hong, Ziwei Liu
Computer Vision · Cross-listed from cs.RO
NavTrust: Benchmarking Trustworthiness for Embodied Navigation
Huaide Jiang, Yash Chaudhary, Yuping Wang et al. (+8 more)
There are two major categories of
embodied navigation:
Vision-
Language Navigation (VLN), where
agents navigate by following natural
language instructions; and Object-Goal
Navigation (OGN), where
agents navigate to a specified target object. However, existing work primarily evaluates model performance under nominal conditions, overlooking the potential corruptions that arise in real-world settings. To address this gap, we present NavTrust, a unified
benchmark that systematically corrupts input modalities, including RGB, depth, and instructions, in realistic scenarios and evaluates their impact on
navigation performance. To our best knowledge, NavTrust is the first
benchmark that exposes
embodied navigation agents to diverse RGB-Depth corruptions and instruction variations in a unified framework. Our extensive evaluation of seven state-of-the-art approaches reveals substantial performance degradation under realistic corruptions, which highlights critical robustness gaps and provides a roadmap toward more trustworthy
embodied navigation systems. Furthermore, we systematically evaluate four distinct mitigation strategies to enhance robustness against RGB-Depth and instruction corruptions. Our base models include Uni-NaVid and ETPNav. We deployed them on a real mobile
robot and observed improved robustness to corruptions. The project website is: https://navtrust.github.io/.
Read the full abstract ▾
All authors Huaide Jiang, Yash Chaudhary, Yuping Wang, Zehao Wang, Raghav Sharma, Manan Mehta, Yang Zhou, Lichao Sun, Zhiwen Fan, Zhengzhong Tu, Jiachen Li
Computer Vision
SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing
Xinyao Zhang, Wenkai Dong, Yuxin Song et al. (+10 more)
Current instruction-guided
video editing models struggle to simultaneously balance precise semantic modifications with faithful motion preservation. While existing approaches rely on injecting explicit external priors (e.g., VLM features or structural conditions) to mitigate these issues, this reliance severely bottlenecks model robustness and generalization. To overcome this limitation, we present SAMA (factorized Semantic Anchoring and Motion
Alignment), a framework that factorizes
video editing into semantic anchoring and motion modeling. First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic
tokens and
video latents at sparse anchor frames, enabling purely instruction-aware structural planning. Second, Motion
Alignment pre-trains the same backbone on motion-centric
video restoration pretext tasks (cube inpainting, speed perturbation, and tube shuffle), enabling the model to internalize temporal dynamics directly from raw
videos. SAMA is optimized with a two-stage pipeline: a factorized pre-training stage that learns inherent semantic-motion representations without paired
video-instruction editing data, followed by supervised fine-tuning on paired editing data. Remarkably, the factorized pre-training alone already yields strong zero-shot
video editing ability, validating the proposed factorization. SAMA achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems (e.g., Kling-Omni). Code, models, and
datasets will be released.
Read the full abstract ▾
All authors Xinyao Zhang, Wenkai Dong, Yuxin Song, Bo Fang, Qi Zhang, Jing Wang, Fan Chen, Hui Zhang, Haocheng Feng, Yu Lu, Hang Zhou, Chun Yuan, Jingdong Wang
Computer Vision
Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer
Chenyang Gu, Mingyuan Zhang, Haozhe Xie et al. (+3 more)
Prior motion generation largely follows two paradigms: continuous
diffusion models that excel at kinematic control, and discrete
token-based generators that are effective for semantic conditioning. To combine their strengths, we propose a three-stage framework comprising condition feature extraction (Perception), discrete
token generation (Planning), and
diffusion-based motion synthesis (Control). Central to this framework is MoTok, a
diffusion-based discrete motion tokenizer that decouples semantic abstraction from fine-grained reconstruction by delegating motion recovery to a
diffusion decoder, enabling compact single-layer
tokens while preserving motion fidelity. For kinematic conditions, coarse constraints guide
token generation during planning, while fine-grained constraints are enforced during control through
diffusion-based optimization. This design prevents kinematic details from disrupting semantic
token planning. On HumanML3D, our method significantly improves controllability and fidelity over MaskControl while using only one-sixth of the
tokens, reducing trajectory error from 0.72 cm to 0.08 cm and FID from 0.083 to 0.029. Unlike prior methods that degrade under stronger kinematic constraints, ours improves fidelity, reducing FID from 0.033 to 0.014.
Read the full abstract ▾
All authors Chenyang Gu, Mingyuan Zhang, Haozhe Xie, Zhongang Cai, Lei Yang, Ziwei Liu
Computer Vision
Under One Sun: Multi-Object Generative Perception of Materials and Illumination
Nobuo Yoshii, Xinran Nicole Han, Ryo Kawahara et al. (+2 more)
We introduce Multi-Object Generative Perception (MultiGP), a generative inverse rendering method for
stochastic sampling of all radiometric constituents – reflectance, texture, and illumination – underlying object appearance from a single image. Our key idea to solve this inherently ambiguous radiometric disentanglement is to leverage the fact that while their texture and reflectance may differ, objects in the same scene are all lit by the same illumination. MultiGP exploits this consensus to produce samples of reflectance, texture, and illumination from a single image of known shapes based on four key technical contributions: a cascaded end-to-end architecture that combines image-space and angular-space disentanglement; Coordinated Scheduling for
diffusion convergence to a single consistent illumination estimate; Axial
Attention applied to facilitate “cross-talk” between objects of different reflectance; and a Texture Extraction ControlNet to preserve high-frequency texture details while ensuring decoupling from estimated lighting. Experimental results demonstrate that MultiGP effectively leverages the complementary spatial and frequency characteristics of multiple object appearances to recover individual texture and reflectance as well as the common illumination.
Read the full abstract ▾
All authors Nobuo Yoshii, Xinran Nicole Han, Ryo Kawahara, Todd Zickler, Ko Nishino
Computer Vision
EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing
Yang Fu, Yike Zheng, Ziyun Dai et al. (+1 more)
Video object removal aims to eliminate dynamic target objects and their visual effects, such as deformation, shadows, and reflections, while restoring seamless backgrounds. Recent
diffusion-based
video inpainting and object removal methods can remove the objects but often struggle to erase these effects and to synthesize coherent backgrounds. Beyond method limitations, progress is further hampered by the lack of a comprehensive
dataset that systematically captures common object effects across varied environments for training and evaluation. To address this, we introduce VOR (
Video Object Removal), a large-scale
dataset that provides diverse paired
videos, each consisting of one
video where the target object is present with its effects and a counterpart where the object and effects are absent, with corresponding object masks. VOR contains 60K high-quality
video pairs from captured and synthetic sources, covers five effects types, and spans a wide range of object categories as well as complex, dynamic multi-object scenes. Building on VOR, we propose EffectErase, an effect-aware
video object removal method that treats
video object insertion as the inverse auxiliary task within a reciprocal learning scheme. The model includes task-aware region guidance that focuses learning on affected areas and enables flexible task switching. Then, an insertion-removal consistency objective that encourages complementary behaviors and shared localization of effect regions and structural cues. Trained on VOR, EffectErase achieves superior performance in extensive experiments, delivering high-quality
video object effect erasing across diverse scenarios.
Read the full abstract ▾
All authors Yang Fu, Yike Zheng, Ziyun Dai, Henghui Ding
Computer Vision
Spectrally-Guided Diffusion Noise Schedules
Carlos Esteves and Ameesh Makadia
Denoising
diffusion models are widely used for high-quality image and
video generation. Their performance depends on noise schedules, which define the distribution of noise levels applied during training and the sequence of noise levels traversed during sampling. Noise schedules are typically handcrafted and require manual tuning across different resolutions. In this work, we propose a principled way to design per-instance noise schedules for pixel
diffusion, based on the image's spectral properties. By deriving theoretical bounds on the efficacy of minimum and maximum noise levels, we design “tight” noise schedules that eliminate redundant steps. During inference, we propose to conditionally sample such noise schedules. Experiments show that our noise schedules improve generative quality of single-stage pixel
diffusion models, particularly in the low-step regime.
Read the full abstract ▾