Computer Vision
Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMs
Paul Jonas Kurz, Tobias Jan Wieczorek, Mohamed A. Abdelsalam et al. (+2 more)
Multimodal Large
Language Models (MLLM) are increasingly deployed in domains where both reliability and efficiency are critical. However, current models remain overconfident, producing highly certain but incorrect answers. At the same time, their large size limits deployment on edge devices, necessitating compression. We study the intersection of these two challenges by analyzing how Post-Training Quantization (PTQ) compression affects both accuracy and reliability in Visual Question Answering (VQA). We evaluate two MLLMs, Qwen2-VL-7B and Idefics3-8B, quantized with data-free (HQQ) and data-aware (MBQ) methods across multiple bit widths. To counteract the reduction in reliability caused by quantization, we adapt the Selector confidence
estimator for quantized
multimodal settings and test its robustness across various quantization levels and out-of-distribution (OOD) scenarios. We find that PTQ degrades both accuracy and reliability. Data-aware methods soften the effect thereof. The Selector substantially mitigates the reliability impact. The combination of int4 MBQ and the Selector achieves the best efficiency-reliability trade-off, closing in on uncompressed performance at approx. 75% less
memory demand. Overall, we present the first systematic study linking quantization and reliability in
multimodal settings.
Read the full abstract ▾
All authors Paul Jonas Kurz, Tobias Jan Wieczorek, Mohamed A. Abdelsalam, Rahaf Aljundi, Marcus Rohrbach
Computer Vision
Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs – Evolution, Limitations, and Cognitive Enhancement
Zhihang Yi, Jian Zhao, Jiancheng Lv et al. (+1 more)
Chart understanding is a quintessential information fusion task, requiring the seamless integration of graphical and textual data to extract meaning. The advent of
Multimodal Large
Language Models (MLLMs) has revolutionized this domain, yet the landscape of MLLM-based chart analysis remains fragmented and lacks systematic organization. This survey provides a comprehensive roadmap of this nascent frontier by structuring the domain's core components. We begin by analyzing the fundamental challenges of fusing visual and linguistic information in charts. We then categorize downstream tasks and
datasets, introducing a novel taxonomy of canonical and non-canonical
benchmarks to highlight the field's expanding scope. Subsequently, we present a comprehensive
evolution of methodologies, tracing the progression from classic deep learning techniques to state-of-the-art MLLM paradigms that leverage sophisticated fusion strategies. By critically examining the limitations of current models, particularly their perceptual and
reasoning deficits, we identify promising future directions, including advanced
alignment techniques and reinforcement learning for cognitive enhancement. This survey aims to equip researchers and practitioners with a structured understanding of how MLLMs are transforming chart information fusion and to catalyze progress toward more robust and reliable systems.
Read the full abstract ▾
All authors Zhihang Yi, Jian Zhao, Jiancheng Lv, Tao Wang
Computer Vision
Multi-encoder ConvNeXt Network with Smooth Attentional Feature Fusion for Multispectral Semantic Segmentation
Leo Thomas Ramos and Angel D. Sappa
This work proposes MeCSAFNet, a multi-branch encoder-decoder architecture for land cover
segmentation in multispectral imagery. The model separately processes visible and non-visible channels through dual ConvNeXt encoders, followed by individual decoders that reconstruct spatial information. A dedicated fusion decoder integrates intermediate features at multiple scales, combining fine spatial cues with high-level spectral representations. The feature fusion is further enhanced with CBAM
attention, and the ASAU activation function contributes to stable and efficient optimization. The model is designed to process different spectral configurations, including a 4-channel (4c) input combining RGB and NIR bands, as well as a 6-channel (6c) input incorporating NDVI and NDWI indices. Experiments on the Five-Billion-Pixels (FBP) and Potsdam
datasets demonstrate significant performance gains. On FBP, MeCSAFNet-base (6c) surpasses U-Net (4c) by +19.21%, U-Net (6c) by +14.72%, SegFormer (4c) by +19.62%, and SegFormer (6c) by +14.74% in mIoU. On Potsdam, MeCSAFNet-large (4c) improves over DeepLabV3+ (4c) by +6.48%, DeepLabV3+ (6c) by +5.85%, SegFormer (4c) by +9.11%, and SegFormer (6c) by +4.80% in mIoU. The model also achieves consistent gains over several recent state-of-the-art approaches. Moreover, compact variants of MeCSAFNet deliver notable performance with lower training time and reduced inference cost, supporting their deployment in resource-constrained environments. Model code is available at: https://github.com/Leo-Thomas/mecsafnet
Read the full abstract ▾
Computer Vision
Robustness of Vision Language Models Against Split-Image Harmful Input Attacks
Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu et al. (+1 more)
Vision-
Language Models (VLMs) are now a core part of modern AI. Recent work proposed several visual jailbreak attacks using single/ holistic images. However, contemporary VLMs demonstrate strong robustness against such attacks due to extensive safety
alignment through preference optimization, e.g., reinforcement learning from human feedback (RLHF). In this work, we identify a new vulnerability: while VLM pretraining and instruction tuning generalize well to split-image inputs, safety
alignment is typically performed only on holistic images and does not account for harmful semantics distributed across multiple image fragments. Consequently, VLMs often fail to detect and reject harmful split-image inputs, in which unsafe cues emerge only upon combining images. We introduce novel split-image visual jailbreak attacks (SIVA) that exploit this misalignment. Unlike prior optimization-based attacks, which exhibit poor black-box transferability due to architectural and prior mismatches across models, our attacks evolve in progressive phases from naive splitting to an adaptive white-box attack, culminating in a black-box transfer attack. Our strongest strategy leverages a novel
adversarial knowledge
distillation (Adv-KD)
algorithm to substantially improve cross-model transferability. Evaluations on four state-of-the-art modern VLMs and three jailbreak
datasets demonstrate that our strongest attack achieves up to 44% higher transfer success than existing baselines. Lastly, we propose efficient ways to address this critical vulnerability in the current VLM safety
alignment.
Read the full abstract ▾
All authors Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu, Shagufta Mehnaz
Computer Vision
Fields of The World: A Field Guide for Extracting Agricultural Field Boundaries
Isaac Corley, Hannah Kerner, Caleb Robinson et al. (+1 more)
Field boundary maps are a building block for agricultural data products and support crop monitoring, yield estimation, and disease estimation. This tutorial presents the Fields of The World (FTW) ecosystem: a
benchmark of 1.6M field polygons across 24 countries, pre-trained
segmentation models, and command-line inference tools. We provide two notebooks that cover (1) local-scale field boundary extraction with crop classification and forest loss attribution, and (2) country-scale inference using cloud-optimized data. We use MOSAIKS random convolutional features and FTW derived field boundaries to map crop type at the field level and report macro F1 scores of 0.65–0.75 for crop type classification with limited labels. Finally, we show how to explore pre-computed predictions over five countries (4.76M km\textsuperscript{2}), with median predicted field areas from 0.06 ha (Rwanda) to 0.28 ha (Switzerland).
Read the full abstract ▾
All authors Isaac Corley, Hannah Kerner, Caleb Robinson, Jennifer Marcus
Computer Vision
MambaFusion: Adaptive State-Space Fusion for Multimodal 3D Object Detection
Venkatraman Narayanan, Bala Sai, Rahul Ahuja et al. (+3 more)
Reliable 3D object detection is fundamental to autonomous driving, and
multimodal fusion
algorithms using cameras and
LiDAR remain a persistent challenge. Cameras provide dense visual cues but ill posed depth;
LiDAR provides a precise 3D structure but sparse coverage. Existing BEV-based fusion frameworks have made good progress, but they have difficulties including inefficient context modeling, spatially invariant fusion, and
reasoning under uncertainty. We introduce MambaFusion, a unified multi-modal detection framework that achieves efficient, adaptive, and physically grounded 3D perception. MambaFusion interleaves selective state-space models (SSMs) with windowed
transformers to propagate the global context in linear time while preserving local
geometric fidelity. A multi-modal
token alignment (MTA) module and reliability-aware fusion gates dynamically re-weight camera-
LiDAR features based on spatial confidence and calibration consistency. Finally, a structure-conditioned
diffusion head integrates graph-based
reasoning with uncertainty-aware denoising, enforcing physical plausibility, and calibrated confidence. MambaFusion establishes new state-of-the-art performance on nuScenes
benchmarks while operating with linear-time complexity. The framework demonstrates that coupling SSM-based efficiency with reliability-driven fusion yields robust, temporally stable, and interpretable 3D perception for real-world autonomous driving systems.
Read the full abstract ▾
All authors Venkatraman Narayanan, Bala Sai, Rahul Ahuja, Pratik Likhar, Varun Ravi Kumar, Senthil Yogamani
Computer Vision
Building Damage Detection using Satellite Images and Patch-Based Transformer Methods
Smriti Siva and Jan Cross-Zamirski
Rapid building damage assessment is critical for post-disaster response. Damage classification models built on satellite imagery provide a scalable means of obtaining situational awareness. However, label noise and severe class imbalance in satellite data create major challenges. The xBD
dataset offers a standardized
benchmark for building-level damage across diverse geographic regions. In this study, we evaluate
Vision Transformer (ViT) model performance on the xBD
dataset, specifically investigating how these models distinguish between types of structural damage when training on noisy, imbalanced data. In this study, we specifically evaluate DINOv2-small and DeiT for multi-class damage classification. We propose a targeted patch-based pre-processing pipeline to isolate structural features and minimize background noise in training. We adopt a frozen-head fine-tuning strategy to keep computational requirements manageable. Model performance is evaluated through accuracy, precision, recall, and macro-averaged F1 scores. We show that small ViT architectures with our novel training method achieves competitive macro-averaged F1 relative to prior CNN baselines for disaster classification.
Read the full abstract ▾
Computer Vision
MMLSv2: A Multimodal Dataset for Martian Landslide Detection in Remote Sensing Imagery
Sidike Paheding, Abel Reyes-Angulo, Leo Thomas Ramos et al. (+5 more)
We present MMLSv2, a
dataset for landslide
segmentation on Martian surfaces. MMLSv2 consists of
multimodal imagery with seven bands: RGB, digital elevation model, slope, thermal inertia, and grayscale channels. MMLSv2 comprises 664 images distributed across training, validation, and test splits. In addition, an isolated test set of 276 images from a geographically disjoint region from the base
dataset is released to evaluate spatial generalization. Experiments conducted with multiple
segmentation models show that the
dataset supports stable training and achieves competitive performance, while still posing challenges in fragmented, elongated, and small-scale landslide regions. Evaluation on the isolated test set leads to a noticeable performance drop, indicating increased difficulty and highlighting its value for assessing model robustness and generalization beyond standard in-distribution settings.
Dataset will be available at: https://github.com/MAIN-Lab/MMLS_v2
Read the full abstract ▾
All authors Sidike Paheding, Abel Reyes-Angulo, Leo Thomas Ramos, Angel D. Sappa, Rajaneesh A., Hiral P. B., Sajin Kumar K. S., Thomas Oommen
Computer Vision
Issar Tzachor, Dvir Samuel and Rami Ben-Ari
Recent studies have adapted generative
Multimodal Large
Language Models (MLLMs) into
embedding extractors for text-
vision tasks, typically by fine-tuning them to produce universal representations. However, their performance on
video-text
retrieval often remains below that of dedicated
Video Foundation Models. In this paper, we revisit the
video-text
retrieval capabilities of MLLMs. We first analyze the zero-shot
retrieval capabilities of pretrained MLLMs and show that they already encode substantial
retrieval-relevant information: combining intermediate-layer
embeddings with a calibrated MLLM head yields strong performance without any training. To further exploit these capabilities, we introduce a lightweight text-based optimization strategy that requires no paired
multimodal data. Our method uses dense
video captions as rich textual surrogates for
video semantics and short summaries as compact, query-like views, showing that a carefully designed text-
alignment objective can improve
multimodal alignment through the shared MLLM backbone. We backup these findings through in-depth analysis. Remarkably, our lightweight text-only optimization outperforms its counterpart trained with paired
video-text supervision, which is substantially more computationally expensive. Furthermore, our method outperforms existing approaches, often by a substantial margin, and achieves state-of-the-art performance across common
video-text
retrieval benchmarks. More broadly, our findings highlight the untapped
retrieval capabilities of pretrained MLLMs and establish text-only optimization as a more efficient and effective substitute for large-scale visual supervision.
Read the full abstract ▾
Computer Vision
Feng Wang, Sucheng Ren, Tiezheng Zhang et al. (+4 more)
This work presents a systematic investigation into modernizing
Vision Transformer backbones by leveraging architectural advancements from the past five years. While preserving the canonical
Attention-FFN structure, we conduct a component-wise refinement involving normalization, activation functions, positional encoding, gating mechanisms, and learnable
tokens. These updates form a new generation of
Vision Transformers, which we call ViT-5. Extensive experiments demonstrate that ViT-5 consistently outperforms state-of-the-art plain
Vision Transformers across both understanding and generation
benchmarks. On ImageNet-1k classification, ViT-5-Base reaches 84.2% top-1 accuracy under comparable compute, exceeding DeiT-III-Base at 83.8%. ViT-5 also serves as a stronger backbone for generative modeling: when plugged into an SiT
diffusion framework, it achieves 1.84 FID versus 2.06 with a vanilla ViT backbone. Beyond headline metrics, ViT-5 exhibits improved representation learning and favorable spatial
reasoning behavior, and transfers reliably across tasks. With a design aligned with contemporary foundation-model practices, ViT-5 offers a simple drop-in upgrade over vanilla ViT for mid-2020s
vision backbones.
Read the full abstract ▾
All authors Feng Wang, Sucheng Ren, Tiezheng Zhang, Predrag Neskovic, Anand Bhattad, Cihang Xie, Alan Yuille
Computer Vision
ReRoPE: Repurposing RoPE for Relative Camera Control
Chunyang Li, Yuanbo Yang, Jiahao Shao et al. (+3 more)
Video generation with controllable camera viewpoints is essential for applications such as interactive content creation, gaming, and simulation. Existing methods typically adapt pre-trained
video models using camera poses relative to a fixed reference, e.g., the first frame. However, these encodings lack shift-invariance, often leading to poor generalization and accumulated drift. While relative camera pose
embeddings defined between arbitrary view pairs offer a more robust alternative, integrating them into pre-trained
video diffusion models without prohibitive training costs or architectural changes remains challenging. We introduce ReRoPE, a plug-and-play framework that incorporates relative camera information into pre-trained
video diffusion models without compromising their generation capability. Our approach is based on the insight that Rotary Positional
Embeddings (RoPE) in existing models underutilize their full spectral bandwidth, particularly in the low-frequency components. By seamlessly injecting relative camera pose information into these underutilized bands, ReRoPE achieves precise control while preserving strong pre-trained generative priors. We evaluate our method on both image-to-
video (I2V) and
video-to-
video (V2V) tasks in terms of camera control accuracy and visual fidelity. Our results demonstrate that ReRoPE offers a training-efficient path toward controllable, high-fidelity
video generation. See project page for more results: https://sisyphe-lee.github.io/ReRoPE/
Read the full abstract ▾
All authors Chunyang Li, Yuanbo Yang, Jiahao Shao, Hongyu Zhou, Katja Schwarz, Yiyi Liao
Computer Vision
From Concept Erasure to Style Purification: Contrastive Eigenbases for Artist Style Protection
Tong Zhang, Ru Zhang and Jianyi Liu
Text-to-image
diffusion models can reproduce specific artists visual styles at extremely low cost, raising copyright and deployment safety concerns about unauthorized style mimicry. Existing model-side protection methods generally follow ordinary concept erasure, emphasizing aggressive deletion or redirection of target styles. However, our causal intervention analysis shows that the central issue is not insufficient erasure strength, but a mismatch between artist styles and this paradigm: unlike ordinary object concepts, artist styles do not form compact, localized editable semantic units. Consequently, sparse editing and fixed retain lists struggle to suppress target styles while preserving generation utility. We therefore reformulate artist style protection as style purification, suppressing target style expression during inference while preserving the requested content and visual structure. We propose CAPE (Contrastive Artist Style Purification with Eigenbases), a training-free framework against artist style mimicry. CAPE constructs contrastive triplets around the target request and formulates style direction estimation as a generalized
eigenvalue problem, capturing style-related directions that remain stable across content variations and are less affected by shared content. During inference, CAPE employs the Adaptive Suppression Controller to assign suppression strengths to different
tokens based on Q, K, and V responses, and performs target style suppression on the K and V paths of self-
attention. Experimental results show that CAPE effectively weakens target artist characteristics, including brushstrokes, textures, and local color processing, while better preserving major semantic entities, scene composition, and visual structures.
Read the full abstract ▾
Computer Vision
Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling
Xihang Yu, Rajat Talak, Lorenzo Shaikewitz et al. (+1 more)
In the presence of occlusions and measurement noise, geometrically accurate scene reconstructions – which fit the sensor data – can still be physically incorrect. For instance, when estimating the poses and shapes of objects in the scene and importing the resulting estimates into a simulator, small errors might translate to implausible configurations including object interpenetration or unstable
equilibrium. This makes it difficult to predict the dynamic behavior of the scene using a digital twin, an important step in simulation-based planning and control of contact-rich behaviors. In this paper, we posit that object pose and shape estimation requires
reasoning holistically over the scene (instead of
reasoning about each object in isolation), accounting for object interactions and physical plausibility. Towards this goal, our first contribution is Picasso, a physics-constrained reconstruction pipeline that builds multi-object scene reconstructions by considering geometry, non-penetration, and physics. Picasso relies on a fast rejection sampling method that reasons over multi-object interactions, leveraging an inferred object contact graph to guide samples. Second, we propose the Picasso
dataset, a collection of 10 contact-rich real-world scenes with ground truth annotations, as well as a metric to quantify physical plausibility, which we open-source as part of our
benchmark. Finally, we provide an extensive evaluation of Picasso on our newly introduced
dataset and on the YCB-V
dataset, and show it largely outperforms the state of the art while providing reconstructions that are both physically plausible and more aligned with human intuition.
Read the full abstract ▾
All authors Xihang Yu, Rajat Talak, Lorenzo Shaikewitz, Luca Carlone
Computer Vision
Weak to Strong: VLM-Based Pseudo-Labeling as a Weakly Supervised Training Strategy in Multimodal Video-based Hidden Emotion Understanding Tasks
Yufei Wang, Haixu Liu, Tianxiang Xu et al. (+2 more)
To tackle the automatic recognition of "concealed
emotions" in
videos, this paper proposes a
multimodal weak-supervision framework and achieves state-of-the-art results on the iMiGUE tennis-interview
dataset. First, YOLO 11x detects and crops human portraits frame-by-frame, and DINOv2-Base extracts visual features from the cropped regions. Next, by integrating Chain-of-Thought and Reflection prompting (CoT + Reflection), Gemini 2.5 Pro automatically generates pseudo-labels and
reasoning texts that serve as weak supervision for downstream models. Subsequently, OpenPose produces 137-dimensional key-point sequences, augmented with inter-frame offset features; the usual graph neural network backbone is simplified to an MLP to efficiently model the spatiotemporal relationships of the three key-point streams. An ultra-long-sequence
Transformer independently encodes both the image and key-point sequences, and their representations are concatenated with BERT-encoded interview transcripts. Each modality is first pre-trained in isolation, then fine-tuned jointly, with pseudo-labeled samples merged into the training set for further gains. Experiments demonstrate that, despite severe class imbalance, the proposed approach lifts accuracy from under 0.6 in prior work to over 0.69, establishing a new public
benchmark. The study also validates that an "MLP-ified" key-point backbone can match - or even surpass - GCN-based counterparts in this task.
Read the full abstract ▾
All authors Yufei Wang, Haixu Liu, Tianxiang Xu, Chuancheng Shi, Hongsheng Xing
Computer Vision
Vanilla Group Equivariant Vision Transformer: Simple and Effective
Jiahong Fu, Qi Xie, Deyu Meng et al. (+1 more)
Incorporating
symmetry priors as inductive biases to design equivariant
Vision Transformers (ViTs) has emerged as a promising avenue for enhancing their performance. However, existing equivariant ViTs often struggle to balance performance with equivariance, primarily due to the challenge of achieving holistic equivariant modifications across the diverse modules in ViTs-particularly in harmonizing the Self-
Attention mechanism with Patch
Embedding. To address this, we propose a straightforward framework that systematically renders key ViT components, including patch
embedding, self-
attention, positional encodings, and Down/Up-Sampling, equivariant, thereby constructing ViTs with guaranteed equivariance. The resulting architecture serves as a plug-and-play replacement that is both theoretically grounded and practically versatile, scaling seamlessly even to Swin
Transformers. Extensive experiments demonstrate that our equivariant ViTs consistently improve performance and data efficiency across a wide spectrum of
vision tasks.
Read the full abstract ▾
All authors Jiahong Fu, Qi Xie, Deyu Meng, Zongben Xu
Computer Vision
Enhanced Mixture 3D CGAN for Completion and Generation of 3D Objects
Yahia Hamdi, Nicolas Andrialovanirina, Kélig Mahé et al. (+1 more)
The generation and completion of 3D objects represent a transformative challenge in computer
vision. Generative
Adversarial Networks (GANs) have recently demonstrated strong potential in synthesizing realistic visual data. However, they often struggle to capture complex and diverse data distributions, particularly in scenarios involving incomplete inputs or significant missing regions. These challenges arise mainly from the high computational requirements and the difficulty of modeling heterogeneous and structurally intricate data, which restrict their applicability in real-world settings. Mixture of Experts (MoE) models have emerged as a promising solution to these limitations. By dynamically selecting and activating the most relevant expert sub-networks for a given input, MoEs improve both performance and efficiency. In this paper, we investigate the integration of Deep 3D Convolutional GANs (CGANs) with a MoE framework to generate high-quality 3D models and reconstruct incomplete or damaged objects. The proposed architecture incorporates multiple generators, each specialized to capture distinct modalities within the
dataset. Furthermore, an auxiliary loss-free dynamic capacity constraint (DCC) mechanism is introduced to guide the selection of categorical generators, ensuring a balance between specialization, training stability, and computational efficiency, which is critical for 3D voxel processing. We evaluated the model's ability to generate and complete shapes with missing regions of varying sizes and compared its performance with state-of-the-art approaches. Both quantitative and qualitative results confirm the effectiveness of the proposed MoE-DCGAN in handling complex 3D data.
Read the full abstract ▾
All authors Yahia Hamdi, Nicolas Andrialovanirina, Kélig Mahé, Emilie Poisson Caillault
Computer Vision · Cross-listed from gr-qc
Dynamic Black-hole Emission Tomography with Physics-informed Neural Fields
Berthy T. Feng, Andrew A. Chael, David Bromley et al. (+3 more)
With the success of static black-hole imaging, the next frontier is the dynamic and 3D imaging of black holes. Recovering the dynamic 3D gas near a black hole would reveal previously-unseen parts of the universe and inform new physics models. However, only sparse
radio measurements from a single viewpoint are possible, making the dynamic 3D reconstruction problem significantly ill-posed. Previously, BH-NeRF addressed the ill-posed problem by assuming Keplerian dynamics of the gas, but this assumption breaks down near the black hole, where the strong gravitational pull of the black hole and increased electromagnetic activity complicate fluid dynamics. To overcome the restrictive assumptions of BH-NeRF, we propose PI-DEF, a physics-informed approach that uses differentiable neural rendering to fit a 4D (time + 3D) emissivity field given EHT measurements. Our approach jointly reconstructs the 3D velocity field with the 4D emissivity field and enforces the velocity as a soft constraint on the dynamics of the emissivity. In experiments on simulated data, we find significantly improved reconstruction accuracy over both BH-NeRF and a physics-agnostic approach. We demonstrate how our method may be used to estimate other physics parameters of the black hole, such as its
spin.
Read the full abstract ▾
All authors Berthy T. Feng, Andrew A. Chael, David Bromley, Aviad Levis, William T. Freeman, Katherine L. Bouman
Computer Vision
MIND: Benchmarking Memory Consistency and Action Control in World Models
Yixuan Ye, Xuanyu Lu, Yuxin Jiang et al. (+7 more)
World models aim to understand, remember, and predict dynamic visual environments, yet a unified
benchmark for evaluating their fundamental abilities remains lacking. To address this gap, we introduce MIND, the first open-domain closed-loop revisited
benchmark for evaluating
Memory consIstency and action coNtrol in worlD models. MIND contains 250 high-quality
videos at 1080p and 24 FPS, including 100 (first-person) + 100 (third-person)
video clips under a shared action space and 25 + 25 clips across varied action spaces covering eight diverse scenes. We design an efficient evaluation framework to measure two core abilities:
memory consistency and action control, capturing temporal stability and contextual coherence across viewpoints. Furthermore, we design various action spaces, including different character movement speeds and camera rotation angles, to evaluate the action generalization capability across different action spaces under shared scenes. To facilitate future performance benchmarking on MIND, we introduce MIND-World, a novel interactive
Video-to-World baseline. Extensive experiments demonstrate the completeness of MIND and reveal key challenges in current world models, including the difficulty of maintaining long-term
memory consistency and generalizing across action spaces. Code: https://github.com/CSU-JPG/MIND.
Read the full abstract ▾
All authors Yixuan Ye, Xuanyu Lu, Yuxin Jiang, Yuchao Gu, Rui Zhao, Qiwei Liang, Jiachun Pan, Fengda Zhang, Weijia Wu, Alex Jinpeng Wang
Computer Vision
FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging
Ziyang Fan, Keyu Chen, Ruilong Xing et al. (+3 more)
Although
Video Large
Language Models (VLLMs) have shown remarkable capabilities in
video understanding, they are required to process high volumes of visual
tokens, causing significant computational inefficiency. Existing VLLMs acceleration frameworks usually compress spatial and temporal redundancy independently, which overlooks the spatiotemporal relationships, thereby leading to suboptimal spatiotemporal compression. The highly correlated visual features are likely to change in spatial position, scale, orientation, and other attributes over time due to the dynamic nature of
video. Building on this insight, we introduce FlashVID, a training-free inference acceleration framework for VLLMs. Specifically, FlashVID utilizes
Attention and Diversity-based
Token Selection (ADTS) to select the most representative
tokens for basic
video representation, then applies Tree-based Spatiotemporal
Token Merging (TSTM) for fine-grained spatiotemporal redundancy elimination. Extensive experiments conducted on three representative VLLMs across five
video understanding
benchmarks demonstrate the effectiveness and generalization of our method. Notably, by retaining only 10% of visual
tokens, FlashVID preserves 99.1% of the performance of LLaVA-OneVision. Consequently, FlashVID can serve as a training-free and plug-and-play module for extending long
video frames, which enables a 10x increase in
video frame input to Qwen2.5-VL, resulting in a relative improvement of 8.6% within the same computational budget. Code is available at https://github.com/Fanziyang-v/FlashVID.
Read the full abstract ▾
All authors Ziyang Fan, Keyu Chen, Ruilong Xing, Yulin Li, Li Jiang, Zhuotao Tian
Computer Vision
PhysDrape: Learning Explicit Forces and Collision Constraints for Physically Realistic Garment Draping
Minghai Chen, Mingyuan Liu, Ning Ma et al. (+2 more)
Deep learning-based garment draping has emerged as a promising alternative to traditional Physics-Based Simulation (PBS), yet robust collision handling remains a critical bottleneck. Most existing methods enforce physical validity through soft penalties, creating an intrinsic trade-off between
geometric feasibility and physical plausibility: penalizing collisions often distorts mesh structure, while preserving shape leads to interpenetration. To resolve this conflict, we present PhysDrape, a hybrid neural-physical solver for physically realistic garment draping driven by explicit forces and constraints. Unlike soft-constrained frameworks, PhysDrape integrates neural inference with explicit
geometric solvers in a fully differentiable pipeline. Specifically, we propose a Physics-Informed Graph Neural Network conditioned on a physics-enriched graph – encoding material parameters and body proximity – to predict residual displacements. Crucially, we integrate a differentiable two-stage solver: first, a learnable Force Solver iteratively resolves unbalanced forces derived from the Saint Venant-Kirchhoff (StVK) model to ensure quasi-static
equilibrium; second, a Differentiable Projection strictly enforces collision constraints against the body surface. This differentiable design guarantees physical validity through explicit constraints, while enabling end-to-end learning to optimize the network for physically consistent predictions. Extensive experiments demonstrate that PhysDrape achieves state-of-the-art performance, ensuring negligible interpenetration with significantly lower strain energy compared to existing baselines, achieving superior physical fidelity and robustness in real-time.
Read the full abstract ▾
All authors Minghai Chen, Mingyuan Liu, Ning Ma, Jianqing Li, Yuxiang Huan
Computer Vision
ForecastOcc: Vision-based Semantic Occupancy Forecasting
Riya Mohan, Juana Valeria Hurtado, Rohit Mohan et al. (+1 more)
Autonomous driving requires
forecasting both geometry and semantics over time to effectively reason about future environment states. Existing
vision-based occupancy
forecasting methods focus on motion-related categories such as static and dynamic objects, while semantic information remains largely absent. Recent semantic occupancy
forecasting approaches address this gap but rely on past occupancy predictions obtained from separate networks. This makes current methods sensitive to error accumulation and prevents learning spatio-temporal features directly from images. In this work, we present ForecastOcc, the first framework for
vision-based semantic occupancy
forecasting that jointly predicts future occupancy states and semantic categories. Our framework yields semantic occupancy forecasts for multiple horizons directly from past camera images, without relying on externally estimated maps. We evaluate ForecastOcc in two complementary settings: multi-view
forecasting on the Occ3D-nuScenes
dataset and
monocular forecasting on SemanticKITTI, where we establish the first
benchmark for this task. We introduce the first baselines by adapting two 2D
forecasting modules within our framework. Importantly, we propose a novel architecture that incorporates a temporal cross-
attention forecasting module, a 2D-to-3D view
transformer, a 3D encoder for occupancy prediction, and a semantic occupancy head for voxel-level forecasts across multiple horizons. Extensive experiments on both
datasets show that ForecastOcc consistently outperforms baselines, yielding semantically rich, future-aware predictions that capture scene dynamics and semantics critical for autonomous driving.
Read the full abstract ▾
All authors Riya Mohan, Juana Valeria Hurtado, Rohit Mohan, Abhinav Valada
Computer Vision
MCIE: Multimodal LLM-Driven Complex Instruction Image Editing with Spatial Guidance
Xuehai Bai, Xiaoling Gu, Akide Liu et al. (+3 more)
Recent advances in instruction-based image editing have shown remarkable progress. However, existing methods remain limited to relatively simple editing operations, hindering real-world applications that require complex and compositional instructions. In this work, we address these limitations from the perspectives of architectural design, data, and evaluation protocols. Specifically, we identify two key challenges in current models: insufficient instruction compliance and background inconsistency. To this end, we propose MCIE-E1, a
Multimodal Large
Language Model-Driven Complex Instruction Image Editing method that integrates two key modules: a spatial-aware cross-
attention module and a background-consistent cross-
attention module. The former enhances instruction-following capability by explicitly aligning semantic instructions with spatial regions through spatial guidance during the denoising process, while the latter preserves features in unedited regions to maintain background consistency. To enable effective training, we construct a dedicated data pipeline to mitigate the scarcity of complex instruction-based image editing
datasets, combining fine-grained automatic filtering via a powerful MLLM with rigorous human validation. Finally, to comprehensively evaluate complex instruction-based image editing, we introduce CIE-Bench, a new
benchmark with two new evaluation metrics. Experimental results on CIE-Bench demonstrate that MCIE-E1 consistently outperforms previous state-of-the-art methods in both quantitative and qualitative assessments, achieving a 23.96% improvement in instruction compliance.
Read the full abstract ▾
All authors Xuehai Bai, Xiaoling Gu, Akide Liu, Hangjie Yuan, YiFan Zhang, Jack Ma
Computer Vision
Deepfake Synthesis vs. Detection: An Uneven Contest
Md. Tarek Hasan, Sanjay Saha, Shaojing Fan et al. (+2 more)
The rapid advancement of deepfake technology has significantly elevated the realism and accessibility of synthetic media. Emerging techniques, such as
diffusion-based models and Neural Radiance Fields (NeRF), alongside enhancements in traditional Generative
Adversarial Networks (GANs), have contributed to the sophisticated generation of deepfake
videos. Concurrently, deepfake detection methods have seen notable progress, driven by innovations in
Transformer architectures, contrastive learning, and other machine learning approaches. In this study, we conduct a comprehensive empirical analysis of state-of-the-art deepfake detection techniques, including human evaluation experiments against cutting-edge synthesis methods. Our findings highlight a concerning trend: many state-of-the-art detection models exhibit markedly poor performance when challenged with deepfakes produced by modern synthesis techniques, including poor performance by human participants against the best quality deepfakes. Through extensive experimentation, we provide evidence that underscores the urgent need for continued refinement of detection models to keep pace with the evolving capabilities of deepfake generation technologies. This research emphasizes the critical gap between current detection methodologies and the sophistication of new generation techniques, calling for intensified efforts in this crucial area of study.
Read the full abstract ▾
All authors Md. Tarek Hasan, Sanjay Saha, Shaojing Fan, Swakkhar Shatabda, Terence Sim
Computer Vision
Continuity-driven Synergistic Diffusion with Neural Priors for Ultra-Sparse-View CBCT Reconstruction
Junlin Wang, Jiancheng Fang, Peng Peng et al. (+2 more)
The clinical application of cone-beam computed
tomography (CBCT) is constrained by the inherent trade-off between radiation exposure and image quality. Ultra-sparse angular sampling, employed to reduce dose, introduces severe undersampling artifacts and inter-slice inconsistencies, compromising diagnostic reliability. Existing reconstruction methods often struggle to balance angular continuity with spatial detail fidelity. To address these challenges, we propose a Continuity-driven Synergistic
Diffusion with Neural priors (CSDN) for ultra-sparse-view CBCT reconstruction. Neural priors are introduced as a structural foundation to encode a continuous threedimensional attenuation representation, enabling the synthesis of physically consistent dense projections from ultra-sparse measurements. Building upon this neural-prior-based initialization, a synergistic
diffusion strategy is developed, consisting of two collaborative refinement paths: a Sinogram Refinement
Diffusion (Sino-RD) process that restores angular continuity and a Digital Radiography Refinement
Diffusion (DR-RD) process that enforces inter-slice consistency from the projection image perspective. The outputs of the two
diffusion paths are adaptively fused by the Dual-Projection Reconstruction Fusion (DPRF) module to achieve coherent volumetric reconstruction. Extensive experiments demonstrate that the proposed CSDN effectively suppresses artifacts and recovers fine textures under ultra-sparse-view conditions, outperforming existing state-of-the-art techniques.
Read the full abstract ▾
All authors Junlin Wang, Jiancheng Fang, Peng Peng, Shaoyu Wang, Qiegen Liu
Computer Vision
FSP-Diff: Full-Spectrum Prior-Enhanced DualDomain Latent Diffusion for Ultra-Low-Dose Spectral CT Reconstruction
Peng Peng, Xinrui Zhang, Junlin Wang et al. (+3 more)
Spectral computed
tomography (CT) with
photon-counting detectors holds immense potential for material discrimination and tissue characterization. However, under ultra-low-dose conditions, the sharply degraded signal-to-noise ratio (SNR) in energy-specific projections poses a significant challenge, leading to severe artifacts and loss of structural details in reconstructed images. To address this, we propose FSP-Diff, a full-spectrum prior-enhanced dual-domain latent
diffusion framework for ultra-low-dose spectral CT reconstruction. Our framework integrates three core strategies: 1) Complementary Feature Construction: We integrate direct image reconstructions with projection-domain denoised results. While the former preserves latent textural nuances amidst heavy noise, the latter provides a stable structural scaffold to balance detail fidelity and noise suppression. 2) Full-Spectrum Prior Integration: By fusing multi-energy projections into a high-SNR full-spectrum image, we establish a unified structural reference that guides the reconstruction across all energy bins. 3) Efficient Latent
Diffusion Synthesis: To alleviate the high computational burden of high-dimensional spectral data, multi-path features are embedded into a compact latent space. This allows the
diffusion process to facilitate interactive feature fusion in a lower-dimensional
manifold, achieving accelerated reconstruction while maintaining fine-grained detail restoration. Extensive experiments on simulated and real-world
datasets demonstrate that FSP-Diff significantly outperforms state-of-the-art methods in both image quality and computational efficiency, underscoring its potential for clinically viable ultra-low-dose spectral CT imaging.
Read the full abstract ▾
All authors Peng Peng, Xinrui Zhang, Junlin Wang, Lei Li, Shaoyu Wang, Qiegen Liu
Computer Vision
EasyTune: Efficient Step-Aware Fine-Tuning for Diffusion-Based Motion Generation
Xiaofeng Tan, Wanjiang Weng, Haodong Lei et al. (+1 more)
In recent years, motion generative models have undergone significant advancement, yet pose challenges in aligning with downstream objectives. Recent studies have shown that using differentiable rewards to directly align the preference of
diffusion models yields promising results. However, these methods suffer from (1) inefficient and coarse-grained optimization with (2) high
memory consumption. In this work, we first theoretically and empirically identify the key reason of these limitations: the recursive dependence between different steps in the denoising trajectory. Inspired by this insight, we propose EasyTune, which fine-tunes
diffusion at each denoising step rather than over the entire trajectory. This decouples the recursive dependence, allowing us to perform (1) a dense and fine-grained, and (2)
memory-efficient optimization. Furthermore, the scarcity of preference motion pairs restricts the availability of motion reward model training. To this end, we further introduce a Self-refinement Preference Learning (SPL) mechanism that dynamically identifies preference pairs and conducts preference learning. Extensive experiments demonstrate that EasyTune outperforms DRaFT-50 by 8.2% in
alignment (MM-Dist) improvement while requiring only 31.16% of its additional
memory overhead and achieving a 7.3x training speedup. The project page is available at this link {https://xiaofeng-tan.github.io/projects/EasyTune/index.html}.
Read the full abstract ▾
All authors Xiaofeng Tan, Wanjiang Weng, Haodong Lei, Hongsong Wang
Computer Vision
D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioning
Changli Tang, Tianyi Wang, Fengyun Rao et al. (+2 more)
Spoken dialogue is a primary source of information in
videos; therefore, accurately identifying who spoke what and when is essential for deep
video understanding. We introduce D-ORCA, a dialogue-centric omni-modal large
language model optimized for robust audio-visual captioning. We further curate DVD, a large-scale, high-quality bilingual
dataset comprising nearly 40,000 multi-party dialogue
videos for training and 2000
videos for evaluation in English and Mandarin, addressing a critical gap in the open-source ecosystem. To ensure fine-grained captioning accuracy, we adopt group relative policy optimization with three novel reward functions that assess speaker attribution accuracy, global
speech content accuracy, and sentence-level temporal boundary
alignment. These rewards are derived from evaluation metrics widely used in
speech processing and, to our knowledge, are applied for the first time as reinforcement learning objectives for audio-visual captioning. Extensive experiments demonstrate that D-ORCA substantially outperforms existing open-source models in speaker identification,
speech recognition, and temporal grounding. Notably, despite having only 8 billion parameters, D-ORCA achieves performance competitive with Qwen3-Omni across several general-purpose audio-visual understanding
benchmarks. Demos are available at \href{https://d-orca-llm.github.io/}{https://d-orca-llm.github.io/}. Our code, data, and checkpoints will be available at \href{https://github.com/WeChatCV/D-ORCA/}{https://github.com/WeChatCV/D-ORCA/}.
Read the full abstract ▾
All authors Changli Tang, Tianyi Wang, Fengyun Rao, Jing Lyu, Chao Zhang
Computer Vision
One-Shot Crowd Counting With Density Guidance For Scene Adaptation
Jiwei Chen, Qi Wang, Junyu Gao et al. (+3 more)
Crowd scenes captured by cameras at different locations vary greatly, and existing crowd models have limited generalization for unseen surveillance scenes. To improve the generalization of the model, we regard different surveillance scenes as different category scenes, and introduce few-shot learning to make the model adapt to the unseen surveillance scene that belongs to the given exemplar category scene. To this end, we propose to leverage local and global density characteristics to guide the model of crowd counting for unseen surveillance scenes. Specifically, to enable the model to adapt to the varying density variations in the target scene, we propose the multiple local density learner to learn multi prototypes which represent different density distributions in the support scene. Subsequently, these multiple local density similarity matrixes are encoded. And they are utilized to guide the model in a local way. To further adapt to the global density in the target scene, the global density features are extracted from the support image, then it is used to guide the model in a global way. Experiments on three surveillance
datasets shows that proposed method can adapt to the unseen surveillance scene and outperform recent state-of-the-art methods in the few-shot crowd counting.
Read the full abstract ▾
All authors Jiwei Chen, Qi Wang, Junyu Gao, Jing Zhang, Dingyi Li, Jing-Jia Luo
Computer Vision
Integrating Specialized and Generic Agent Motion Prediction with Dynamic Occupancy Grid Maps
Rabbia Asghar, Lukas Rummelhard, Wenqian Liu et al. (+2 more)
Accurate prediction of driving scene is a challenging task due to uncertainty in sensor data, the complex behaviors of
agents, and the possibility of multiple feasible futures. Existing prediction methods using occupancy grid maps primarily focus on
agent-agnostic scene predictions, while
agent-specific predictions provide specialized behavior insights with the help of semantic information. However, both paradigms face distinct limitations:
agent-agnostic models struggle to capture the behavioral complexities of dynamic actors, whereas
agent-specific approaches fail to generalize to poorly perceived or unrecognized
agents; combining both enables robust and safer motion
forecasting. To address this, we propose a unified framework by leveraging Dynamic Occupancy Grid Maps within a streamlined temporal decoding pipeline to simultaneously predict future occupancy state grids, vehicle grids, and scene flow grids. Relying on a lightweight spatiotemporal backbone, our approach is centered on a tailored, interdependent loss function that captures inter-grid dependencies and enables diverse future predictions. By using occupancy state information to enforce flow-guided transitions, the loss function acts as a regularizer that directs occupancy
evolution while accounting for obstacles and occlusions. Consequently, the model not only predicts the specific behaviors of vehicle
agents, but also identifies other dynamic entities and anticipates their
evolution within the complex scene. Evaluations on real-world nuScenes and Woven
Planet datasets demonstrate superior prediction performances for dynamic vehicles and generic dynamic scene elements compared to baseline methods.
Read the full abstract ▾
All authors Rabbia Asghar, Lukas Rummelhard, Wenqian Liu, Anne Spalanzani, Christian Laugier
Computer Vision
Which private attributes do VLMs agree on and predict well?
Olena Hrynenko, Darya Baranouskaya, Alina Elena Baia et al. (+1 more)
Visual
Language Models (VLMs) are often used for zero-shot detection of visual attributes in the image. We present a zero-shot evaluation of open-source VLMs for
privacy-related attribute recognition. We identify the attributes for which VLMs exhibit strong inter-annotator agreement, and discuss the disagreement cases of human and VLM annotations. Our results show that when evaluated against human annotations, VLMs tend to predict the presence of
privacy attributes more often than human annotators. In addition to this, we find that in cases of high inter-annotator agreement between VLMs, they can complement human annotation by identifying attributes overlooked by human annotators. This highlights the potential of VLMs to support
privacy annotations in large-scale image
datasets.
Read the full abstract ▾
All authors Olena Hrynenko, Darya Baranouskaya, Alina Elena Baia, Andrea Cavallaro