Computer Vision
A Causally Grounded Taxonomy for Image Degradation Robustness Evaluation
Stefan Becker, Simon Weiss, Wolfgang Hübner et al. (+1 more)
Image degradations can occur during acquisition, processing, and transmission, altering visual appearance and affecting downstream
vision tasks. They are studied in several communities, including synthetic corruption
benchmarks for robustness evaluation, perceptual image quality assessment, and physically grounded analyses of imaging systems or real camera failures. Although these areas address closely related phenomena, they often use incompatible grouping schemes and backend specific severity definitions, making results difficult to compare across
datasets, degradation sources, and tasks. We propose a causally grounded framework for organizing and interpreting image degradations across these settings. Instead of introducing new degradations or redefining existing
benchmarks, we provide an interpretive representation and measurement layer that makes implicit assumptions explicit. Each degradation is described along two orthogonal axes: its dominant causal source in the imaging pipeline (environment, sensor/optics, ISP/renderer/codec, or transfer/system), and its resulting perceptual effect. This dual axis abstraction yields a compact taxonomy spanning algorithmic corruptions, perceptual distortions, and physically motivated imaging artifacts. To address inconsistent severity semantics without changing existing implementations, we introduce a lightweight severity measurement layer. For every degradation and each native severity level of a given backend, we quantify degradation strength using full reference image quality metrics: PSNR, SSIM, and LPIPS. This makes severity observable and comparable across sources while preserving native parameterizations. We demonstrate the framework through COCO Degradation, a taxonomy aligned
benchmark for evaluating object detector robustness under diverse imaging conditions.
Read the full abstract ▾
All authors Stefan Becker, Simon Weiss, Wolfgang Hübner, Michael Arens
Computer Vision · Cross-listed from eess.IV
Layer Selection in Feature-Based Losses Affects Image Quality and Microstructural Consistency in Deep Learning Super-Resolution of Brain Diffusion MRI
David Lohr and Rene Werner
Clinical application of high-resolution
diffusion MRI is hindered by hardware limitations and prohibitive scan times, motivating computational super-resolution. This study investigates the efficacy of a feature-based loss function in preserving
diffusion signal consistency in deep learning super-resolution. Using 7T data from the human connectome project to generate pairs of low- and high-resolution
diffusion weighted images (DWI), we trained UNets for 2D super-resolution. Ablation and isolation studies evaluated different VGG16-layers for feature-based losses against an image-based L1 baseline. Deeper layers and combinations thereof resulted in grid-like artifacts in super-resolution DWIs, which persisted in
diffusion parameters like quantitative and fractional anisotropy. No such artifacts were present when using the shallowest layer. Downstream analysis for this layer showed great consistency with the ground truth, even for 9-fold super-resolution. Image SNR and used VGG16-layer depths modulated artifact appearance and severity, mandating careful selection of contributing layers for application in and beyond
diffusion MRI.
Read the full abstract ▾
Computer Vision
Uncertainty-Aware Wildfire Smoke Density Classification from Satellite Imagery via CBAM-Augmented EfficientNet with Evidential Deep Learning
Ranjith Chodavarapu
Rapid and accurate wildfire smoke severity assessment from satellite images is essential for emergency response, air quality modeling, and human health risk management. Existing deep learning approaches treat smoke detection as a binary task, producing point estimates without any measure of prediction confidence. We propose a probabilistic framework to categorize a satellite patch into Light, Moderate, and Heavy severity classes and to provide decomposed epistemic and aleatoric uncertainty in a single forward pass. Our architecture uses the backbone of a pre-trained EfficientNet-B3 and a CBAM module with an evidential deep learning head that predicts Dirichlet concentration parameters, directly estimating vacuity (epistemic) and dissonance (aleatoric) without Monte Carlo sampling. Evaluated on 16,298 real satellite patches derived from the Wildfire Detection
dataset, our model achieves 93.8% weighted test accuracy (91.1% unweighted) with ECE=0.0274. Selective prediction retaining the most certain 50% of patches achieves 96.7% accuracy. As image quality degrades, uncertainty increases monotonically, and vacuity is a practical scan quality measure. The Moderate class represents transitional smoke conditions that exhibit the highest epistemic uncertainty (mean vacuity = 0.187), confirming the model correctly identifies ambiguous smoke boundary regions. CBAM spatial
attention maps localize to structurally distinctive scene regions, and t-SNE demonstrates the clear cluster separation of Light and Heavy smoke.
Read the full abstract ▾
Computer Vision
Unlocking Dense Metric Depth Estimation in VLMs
Hanxun Yu, Xuan Qu, Yuxin Wang et al. (+2 more)
Vision-
Language Models (VLMs) excel at 2D tasks such as grounding and captioning, yet remain limited in 3D understanding. A key limitation is their text-only supervision paradigm, which under-constrains fine-grained visual perception and prevents the recovery of dense geometry. Prior methods either distill geometry from external
vision models, introducing error accumulation, or enable direct prediction with inefficient per-pixel query or coarse
token-level outputs. In this paper, we propose DepthVLM, a simple yet effective framework that transforms a single VLM into a native dense geometry predictor while preserving its
multimodal capability. By attaching a lightweight depth head to the LLM backbone and training under a unified
vision-text supervision paradigm with a two-stage schedule, DepthVLM generates full-resolution depth maps alongside
language outputs in a single forward pass. We further introduce a unified indoor-outdoor metric depth
benchmark in a VLM-compatible format. Experiments show that DepthVLM significantly outperforms existing VLMs with higher inference efficiency, surpasses leading pure
vision models, and improves complex 3D spatial
reasoning, moving toward a truly unified
multimodal foundation model. The project page is available at https://depthvlm.github.io/
Read the full abstract ▾
All authors Hanxun Yu, Xuan Qu, Yuxin Wang, Jianke Zhu, Lei Ke
Computer Vision
Wenjie Yang, Hang Yu, Yuyu Guo et al. (+1 more)
In this work, we address the critical yet underexplored challenge of symmetric
multimodal-to-
multimodal (MM2MM)
retrieval, where queries and contexts are interchangeable. Existing universal
multimodal retrieval works struggle with this task, as they are constrained by the labeled asymmetric
datasets used. We produce
SOLAR (Self-supervised jOint LeArning for symmetric
multimodal Retrieval), a novel two-stage self-supervised framework that leverages readily available unlabeled web-scale image-text pairs. Based on the observation that both semantic
alignment and discrepancies exist between two modalities, in the first stage, we learn the intersection mask of image-text pair, allowing us to align intersection while preserving semantic of difference. In the second stage, the learned mask is further utilized to construct positive and hardnegative samples via masking different parts of image/text, which enable us to conduct self-supervised
multimodal embedding learning. Complementing this framework, we present a new
benchmark featuring high-quality human-verified positive and hard-negative pairs to evaluate symmetric MM2MM
retrieval under realistic conditions, as well as the corresponding pipeline. Extensive experiments against ten SOTA methods show
SOLAR surpasses the strongest supervised VLM by 7.08 points on this
benchmark, with over 50x fewer model parameters and a 5x smaller
embedding dimension. Code, model and
benchmark are available at https://github.com/codefuse-ai/
SOLAR.
Read the full abstract ▾
All authors Wenjie Yang, Hang Yu, Yuyu Guo, Peng Di
Computer Vision
Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
Chufan Shi, Cheng Yang, Yaokang Wu et al. (+4 more)
Vision-
Language Models (VLMs) often produce self-reflective statements like "let me check the figure again" during
reasoning. Do such statements trigger genuine visual re-examination, or are they merely learned textual patterns? We investigate this via VisualSwap, an image-swap probing framework: after a model reasons over an image, we replace it with a visually similar but semantically different one and test whether the model notices. We introduce VS-Bench, 800 image pairs curated from MathVista, MathVerse, MathVision, and MMMU-Pro. Experiments on Qwen3-VL, Kimi-VL, and ERNIE-VL reveal a striking failure: models overwhelmingly miss the swap, with accuracy dropping by up to 60%. Counterintuitively, thinking models are nearly 3x more vulnerable than their instructed counterparts, and scaling offers no mitigation. Multi-turn user instructions restore visual grounding, but self-generated reflective statements during continuous generation do not.
Attention analysis explains why: user instructions substantially elevate
attention to visual
tokens, whereas self-reflection does not. Current VLMs tend to say rather than actually see when claiming to perform visual re-examination. Our code and
dataset are available at the project page: https://visualswap.github.io
Read the full abstract ▾
All authors Chufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin, Bo Shui, Taylor Berg-Kirkpatrick, Xuezhe Ma
Computer Vision
On RGB-TIR Stereo Calibration under Extreme Resolution Asymmetry
Michał Król, Michał Salamonowicz, Władysław Skarbek et al. (+1 more)
Accurate
geometric calibration of RGB-thermal infrared (TIR) stereo camera systems is essential for
multimodal building envelope analysis, yet remains challenging when low-cost thermal sensors with very low spatial resolution are employed. This paper presents a practical stereo calibration framework for an RGB camera (2028 x 1520 px) paired with a TIR camera operating at only 80 x 62 px - a pixel-count ratio of approximately 1:625. An active OLED screen dynamically switches modality-specific patterns (checkerboard for TIR, ChArUco for RGB) on a single physical surface, providing controlled and repeatable thermal contrast. A dedicated corner detection
algorithm combining perspective rectification, Hessian saddle-point analysis, and Mean Shift localisation achieves reliable checkerboard detection at 80 x 62 px without per-frame parameter tuning. A baseline-constrained bundle adjustment enforces physically consistent rig geometry under the planar-calibration-object degeneracy, yielding a stereo baseline of 32.7 mm (nominal 30 mm) with an overall reprojection error of 0.382 px. The system is validated on a thermally active building mock-up using constant-depth and per-pixel depth estimation, demonstrating consistent TIR-to-RGB projection suitable for building energy performance assessment.
Read the full abstract ▾
All authors Michał Król, Michał Salamonowicz, Władysław Skarbek, Michał Tomaszewski
Computer Vision
Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?
Renye Yan, Jikang Cheng, Shikun Sun et al. (+7 more)
Despite strong image-generation performance,
diffusion models' reconstruction objectives limit
alignment with human preferences. RL enables such
alignment through explicit rewards. However, most studies apply RL to the full denoising trajectory, making it computationally costly and weakening preference
alignment, i.e., doing more but achieving less. We observe that the impact of RL fine-tuning varies significantly across denoising stages. In the early stage, image structures are unstable and distant from the final reward signal. Applying RL at this stage leads to delayed rewards and action-reward mismatching, resulting in high variance and inefficient updates. Conversely, in the later stage, reward gains saturate, and continued training tends to overfit local details, intensifying reward hacking. To tackle these challenges, we propose AdaScope, an RL-enhanced plug-in that improves generation quality while reducing computational cost. Specifically, AdaScope adaptively identifies the optimal intervention timing for RL by perceiving the structural
evolution and semantic consistency during denoising, and dynamically terminates training once the denoising converges and reward gains saturate. As a result, it achieves a rare 'dual benefit': a reduction in computational costs alongside a significant performance improvement. We offer theoretical grounds for the design of AdaScope. Compared with state-of-the-art methods, AdaScope improves performance by 66% while cutting computational cost by 59%.
Read the full abstract ▾
All authors Renye Yan, Jikang Cheng, Shikun Sun, Yi Sun, You Wu, Wei Peng, Zongwei Wang, Ling Liang, Junliang Xing, Yimao Cai
Computer Vision
GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction
Leyang Chen, Junyi Wu, Zhiteng Li et al. (+1 more)
Streaming 3D reconstruction from long
monocular video sequences requires maintaining a key-value (KV) cache that grows linearly with sequence length, creating a severe
memory bottleneck. Existing approaches either truncate the cache to a fixed set of anchor frames, leading to reconstruction quality degradation, or rely on
attention-score heuristics that are agnostic to 3D scene structure, failing to preserve geometrically valuable
tokens. To address these problems, we present GHOST (Geometry-Hierarchical Online Streaming
Token Eviction), a training-free KV cache management framework that exploits the model's own 3D geometry outputs to evict redundant
tokens online. GHOST introduces three mutually reinforcing innovations: a hierarchical dual-level importance scoring scheme, a privilege mechanism that protects special
tokens from eviction, and a cosine-similarity-guided layer-wise budget allocation. Experiments on various
benchmarks show that GHOST preserves excellent reconstruction quality while cutting the KV cache by nearly half and delivering 1.75x faster inference compared to state-of-the-art methods. Our code is available at https://github.com/lokiniuniu/GHOST.
Read the full abstract ▾
All authors Leyang Chen, Junyi Wu, Zhiteng Li, Yulun Zhang
Computer Vision
WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes
Jichen Hu, Jiawei Guo, Jiazhong Cen et al. (+3 more)
Recent 3D world modeling systems based on generative scene synthesis, such as Marble, can create coherent and explorable 3D environments, yet their outputs are typically static monolithic assets with limited editability and physical interaction. This restricts their use in immersive content creation and
embodied simulation, where generated worlds must be actively modified and manipulated. To tackle this challenge, we present WorldAct, a framework that converts static generated 3D worlds into editable and interaction-ready scenes. WorldAct uses a
multimodal agent to guide scene decomposition, identify actionable objects, reconstruct geometrically aligned object-level meshes for interaction, and restore the residual background via 3D inpainting. The resulting scenes support object-level editing, collision-aware manipulation, and
embodied task execution while preserving global scene coherence. Experiments show that WorldAct enables richer interaction scenarios than the original generated scenes, suggesting a practical path toward editable and interactive 3D world models. Project page: https://sjtu-deepvisionlab.github.io/WorldAct
Read the full abstract ▾
All authors Jichen Hu, Jiawei Guo, Jiazhong Cen, Chen Yang, Sikuang Li, Wei Shen
Computer Vision
Community-aware evaluation and threshold calibration for open-set plankton image recognition
Xi Chen, Eryuan Huang, Yingjun Xiao et al. (+1 more)
Automated plankton image recognition is increasingly used in aquatic ecosystem monitoring, but deployed classifiers inevitably encounter unseen taxa and non-target particles. Open-set recognition methods are usually evaluated with sample-level metrics such as AUROC, AUPR, and FPR@95% unknown-recall operating points, whereas ecological monitoring depends on community-level estimates of taxon abundance and diversity. This study examines the mismatch between these objectives using controlled pseudo-communities and three
datasets spanning marine zooplankton imaged by ZooScan, marine phytoplankton imaged by IFCB, and freshwater plankton imaged by an in-situ camera. We define Open-Set Community Distortion (OSCD), a Bray-Curtis-style error over known taxa plus an unknown bin, with directional components distinguishing known-taxon overestimation from underestimation. Closed-set classifiers achieved high known-class accuracy, but unknown samples were often absorbed with high confidence and in structured ways. Sample-level OOD metrics were not sufficient to select ecological operating points: for MSP, FPR@95% unknown-recall thresholds produced large test-community OSCD on all three
datasets mainly because true known taxa were over-rejected into the unknown bin. Community-aware threshold calibration reduced MSP OSCD relative to fixed 95% known recall on SYKE-ZooScan 2024 and SYKE-IFCB 2022; on ZooLake the fixed-recall baseline was already close to the community-aware threshold, and the best community-level method was a prototype-distance variant rather than MSP. The benefit of community-aware calibration therefore depends on validation-community representativeness and the gap between fixed recall and the community optimum. These results show that open-set plankton recognition should be evaluated as an ecological measurement problem, not only as a sample-level detection task.
Read the full abstract ▾
All authors Xi Chen, Eryuan Huang, Yingjun Xiao, Gang Fang
Computer Vision
Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer
Yipu Zhang, Jintao Cheng, Weilun Feng et al. (+5 more)
Feed-forward 3D reconstruction models, represented by Visual Geometry Grounded
Transformer (VGGT), jointly predict multiple visual geometry tasks such as depth estimation, camera pose prediction, and point cloud reconstruction in a single forward pass. They have been widely adopted in 3D
vision applications, but their billion-scale parameters bring substantial
memory and computation overhead, posing challenges for on-device deployment. Post-Training Quantization (PTQ) is an effective technique to reduce this overhead. Existing PTQ methods for feed-forward 3D models mainly focus on handling heavy-tailed activation distributions and constructing diverse calibration
datasets. However, we observe that feed-forward 3D models predict multiple
geometric attributes through a shared backbone, where different
transformer blocks and hidden channels contribute distinctly to each task, resulting in substantially different sensitivities to quantization errors across tasks, blocks, and channels. Consequently, treating all tasks equally over-emphasizes insensitive tasks and causes significant accuracy loss on the sensitive ones. To address this issue, we propose Fisher-Guided Quantization (FGQ) for feed-forward 3D reconstruction models. Specifically, FGQ uses the diagonal Fisher information
matrix to quantify the different sensitivities across tasks, blocks, and channels, and incorporates these sensitivities into the Learnable Affine Transformation during calibration to better preserve the channels and blocks most critical to each task. Extensive experiments across camera pose estimation, point map reconstruction, and depth estimation show that FGQ consistently outperforms state-of-the-art quantization baselines on VGGT, achieving up to 39% relative improvement under the 4-bit quantization. Code is available at https://github.com/ypzhng/FGQ.
Read the full abstract ▾
All authors Yipu Zhang, Jintao Cheng, Weilun Feng, Jiehao Luo, Chuanguang Yang, Zhulin An, Yongjun Xu, Wei Zhang
Computer Vision
FashionChameleon: Towards Real-Time and Interactive Human-Garment Video Customization
Quanjian Song, Yefeng Shen, Mengting Chen et al. (+5 more)
Human-centric
video customization, particularly at the garment level, has shown significant commercial value. However, existing approaches cannot support low-latency and interactive garment control, which is crucial for applications such as e-commerce and content creation. This paper studies how to achieve interactive multi-garment
video customization while preserving motion coherence using only single-garment
video data. We present FashionChameleon, a real-time and interactive framework for human-garment customization in autoregressive
video generation, where users can interactively switch garment during generation. FashionChameleon consists of three key techniques: (i) Instead of training on multi-garment
video data, we train a Teacher Model with In-Context Learning on a single reference-garment pair. By retaining the image-to-
video training paradigm while enforcing a mismatch between the reference and garment image, the model is encouraged to implicitly preserve coherence during single-garment switching. (ii) To achieve consistency and efficiency during generation, we introduce Streaming
Distillation with In-Context Learning, which fine-tunes the model with in-context teacher forcing and improves extrapolation consistency via gradient-reweighted distribution matching
distillation. (iii) To extend the model for interactive multi-garment
video customization, we propose Training-Free KV Cache Rescheduling, which includes garment KV refresh, historical KV withdraw, and reference KV disentangle to achieve garment switching while preserving motion coherence. Our FashionChameleon uniquely supports interactive customization and consistent long-
video extrapolation, while achieving real-time generation at 23.8 FPS on a single GPU, 30-180$\times$ faster than existing baselines.
Read the full abstract ▾
All authors Quanjian Song, Yefeng Shen, Mengting Chen, Hao Sun, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Liujuan Cao
Computer Vision · Cross-listed from cs.GR
StippleDiffusion: Capacity-Constrained Stippling using Controlled Diffusion
Ofir Gilad, Aleksander Plocharski, Przemyslaw Musialski et al. (+1 more)
Stipple patterns, point sets whose local density tracks a target image, are traditionally produced by per-density iterative optimizers, which are slow, non-differentiable, and must be re-run from scratch for each new target. Learned alternatives have so far addressed only unconditional point generation; capacity-constrained, image-conditioned stippling has remained out of reach. We present the first
diffusion-based sampler that simultaneously satisfies a learned local point-distribution prior and a continuous, image-defined capacity constraint at inference. The method is a ControlNet branch built on top of an optimal-transport-grid point-set
diffusion baseline, conditioned on the target density map and a high-resolution image. Two design choices make the combination tractable: training and inference are restricted to the late-stage denoising regime, initialized from a density-weighted rejection sample, and the standard zero-convolution injection is replaced with a sigmoid-gated 1x1 projection that preserves the base model's blue-noise structure under hard density signals. A single trained checkpoint accepts arbitrary target densities at inference, generalizes to point budgets that were not seen during training, and produces stipples in time nearly independent of the output point count. On the Icons-50
benchmark, our learned sampler reaches parity with per-density-optimized baselines on every reported metric while remaining differentiable end-to-end.
Read the full abstract ▾
All authors Ofir Gilad, Aleksander Plocharski, Przemyslaw Musialski, Andrei Sharf
Computer Vision
Embedding-perturbed Exploration Preference Optimization for Flow Models
Sujie Hu, Chubin Chen, Jiashu Zhu et al. (+3 more)
Recent advancements have established Reinforcement Learning (RL) as a pivotal paradigm for aligning generative models with human intent. However, group-based optimization frameworks (e.g., GRPO) face a critical limitation: the rapid decay of intra-group variance. As the distinctiveness among samples within a group diminishes, the variance approaches zero. This eliminates the very learning signal required for optimization, rendering the process unstable and forcing the policy into premature stagnation or reward hacking. Existing strategies, such as varying the initial noise or increasing group sizes, often fail to address this fundamental issue, resulting in training instability or diminishing returns. To overcome these challenges, we propose $\textbf{Embedding-perturbed Exploration Preference Optimization (}E^2\textbf{PO)}$, a novel framework that sustains optimization through
embedding-level perturbation. Our method introduces structured,
embedding-level perturbations within sample groups, guaranteeing a robust variance that preserves the discriminative signal throughout the training process. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art baselines, achieving a more faithful
alignment with human preference.
Read the full abstract ▾
All authors Sujie Hu, Chubin Chen, Jiashu Zhu, Jiahong Wu, Xiangxiang Chu, Xiu Li
Computer Vision
Cross-Modal Registration Between 3D and 2D Fingerprints via Pose-Aware Unwrapping and Point-Cloud Fusion
Xiongjun Guan, Jianjiang Feng and Jie Zhou
Three-dimensional (3D) fingerprints preserve global finger geometry and local ridge structure while avoiding contact-induced deformation, but they remain difficult to integrate with legacy two-dimensional (2D) fingerprint systems. This paper addresses the intermediate stage between 3D acquisition and cross-modal matching, and presents a unified framework for 3D fingerprint preprocessing and registration across contactless and contact-based 2D modalities. The framework combines four components: 1) a nonparametric visualization and unwrapping method that converts a 3D fingerprint point cloud into a rolled-equivalent 2D representation without relying on a global finger-shape model; 2) a point-cloud fusion pipeline that registers and mosaics multiple partial 3D captures into a more complete fingerprint model; 3) an ellipse-based pose normalization method for canonical finger
alignment; and 4) a pose-aware cross-modal registration strategy that improves compatibility between 3D fingerprints and both contactless and contact-based 2D fingerprints. Experiments on a self-collected
multimodal fingerprint database containing 150 fingers show that the proposed framework achieves ridge-level 3D registration accuracy, robust pose estimation, and consistent gains in 2D compatibility. In particular, the 3D fusion error is concentrated around 0.09 mm, contactless 2D–3D registration reaches ridge-scale projection accuracy, and pose-aware unwrapping improves genuine matching scores relative to generic 3D unwrapping. These results support the use of 3D fingerprints as an effective
geometric bridge across heterogeneous fingerprint modalities. The baseline implementation has been publicly released at https://github.com/XiongjunGuan/3DFpVisual.
Read the full abstract ▾
Computer Vision
Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models
Yujun Tong, Dongliang Chang, Zijin Yin et al. (+3 more)
The long-standing goal of
multimodal AI is to build unified models in which visual understanding and visual generation mutually enhance one another. Despite recent works such as BAGEL, BLIP3o achieves remarkable progress; In practice, however, this unification remains one-directional: understanding routinely guides generation, yet how and why generation can support understanding is rarely investigated. We revisit this asymmetry and propose Generation-to-Understanding (G2U) synergy, where visual generation becomes an explicit intermediate
reasoning step. Our framework enables a model to perform controlled generative acts, such as detail enhancement, context expansion or structural visualisation, to produce self-generated visual thoughts, which are then fed back into the model to refine perception without retraining or external tools. Through a comprehensive evaluation on twelve
benchmarks, this reversed information flow consistently improves
multimodal understanding. We show that generative fidelity bounds perceptual gain and that distinct families of edit
prompts govern transfer efficiency. We further analyse whether models can decide what to imagine. While they can produce plausible edits, these self-generated visual thoughts lack stable task
alignment, revealing that current large
multimodal models fall short of true self-reflection. This work exposes a missing mechanism in unified cognition and suggests that imagination is not the end of understanding but its beginning.
Read the full abstract ▾
All authors Yujun Tong, Dongliang Chang, Zijin Yin, Xintong Liu, Yuanchen Fang, Zhanyu Ma
Computer Vision
GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions
Junho Kim, Xu Cao, Houze Yang et al. (+6 more)
Understanding social interactions requires
reasoning over subtle non-verbal cues, yet current
multimodal large
language models (MLLMs) often fail to identify who interacts with whom in multi-person
videos. We introduce GRASP, a large-scale social
reasoning dataset that connects high-level social QA with fine-grained gaze and deictic gesture events. GRASP contains 290K question–answer pairs over 46K
videos totaling 749 hours, organized by a 16-category taxonomy spanning gaze, gesture, and joint gaze–gesture
reasoning, together with GRASP-Bench for evaluation. Unlike prior resources that focus on either isolated cues or high-level social QA, GRASP builds questions from identity-consistent gaze trajectories, deictic gestures, and their joint compositions into social events. Moreover, we propose Social Grounding Reward (SGR), a learning signal that uses these social events to encourage models to reason about the participants involved in each interaction. Experiments show that SGR improves performance on GRASP-Bench while maintaining zero-shot performance on related social
video QA
benchmarks.
Read the full abstract ▾
All authors Junho Kim, Xu Cao, Houze Yang, Bikram Boote, Ana Jojic, Fiona Ryan, Bolin Lai, Sangmin Lee, James M. Rehg
Computer Vision
Learn2Splat: Extending the Horizon of Learned 3DGS Optimization
Naama Pearl, Stefano Esposito, Haofei Xu et al. (+6 more)
3D Gaussian Splatting (3DGS) optimization is most commonly performed using general-purpose first-order optimizers such as Adam or SGD. Although robust across scenes, they update each parameter independently without exploiting the structural and spatial relationships among Gaussians, which slows convergence. Recent works introduced learned optimizers that predict correlated updates informed by inter-parameter and inter-Gaussian dependencies. However, those are trained for a fixed number of optimization iterations and rely on manually scheduled learning rates to avoid degradation. In this paper, we introduce Learn2Splat, a learned optimizer for 3DGS that avoids degradation over extended optimization horizons without auxiliary mechanisms. To enable this, we propose a meta-learning scheme that extends the optimization horizon via a checkpoint buffer and an optimizer rollout strategy, combined with an architecture that encodes gradient scale information in its latent states. Results show higher novel view synthesis quality at equal wall-clock time, while remaining stable over long horizons, with zero-shot generalization to unseen
datasets and settings. To support our findings, we build a unified framework to train and evaluate learned and standard optimizers across sparse and dense view settings. Code and models will be released publicly. Our project page is available at https://autonomousvision.github.io/learn2splat .
Read the full abstract ▾
All authors Naama Pearl, Stefano Esposito, Haofei Xu, Amit Peleg, Patricia Gschossmann, Lorenzo Porzi, Peter Kontschieder, Gerard Pons-Moll, Andreas Geiger
Computer Vision
Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Models
Cheng Zhang, Yuer Liu, Zhiyu Zhou et al. (+2 more)
Multimodal large
language models (MLLMs) can produce fluent artwork
emotion explanations, but they often suffer from attribute flooding: they enumerate many visible formal attributes without identifying which cues actually support the affective judgment. We therefore formulate artwork
emotion understanding as Attribute-Grounded Selective
Reasoning (AGSR), where predefined formal attributes serve as evidence units and only emotionally operative attributes should enter the final interpretation. To make this problem measurable, we extend EmoArt, originally introduced at ACM MM 2025 as a 132,664-artwork resource with content, formal-attribute, valence-arousal, and
emotion annotations, by adding a 1,400-artwork human salience extension annotated by 15 art-trained annotators. This extension provides instance-level supervision for distinguishing attributes that are merely present from those that are emotionally salient. We further propose FAB-G (Formal-Attribute Bottleneck-Guided
reasoning), a supervised multi-
agent framework that first predicts attribute-level salience and then constrains downstream emotional analysis to the retained cues. Experiments show that FAB-G yields consistent gains in
emotion, arousal, and valence prediction, achieves stronger agreement with human-marked salient attributes under Dice and Tversky metrics, and produces substantially more compact final explanations than prompting-based baselines. Cross-
dataset evaluation further suggests that attribute-grounded salience selection transfers beyond the source distribution of EmoArt, while also revealing attribute-specific boundary cases. The
dataset and project page are available at https://zhiliangzhang.github.io/EmoArt-130k/
Read the full abstract ▾
All authors Cheng Zhang, Yuer Liu, Zhiyu Zhou, Hongxia Xie, Wen-Huang Cheng
Computer Vision · Cross-listed from cs.RO
Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spaces
Xinggang Hu, Chenyangguang Zhang, Alexandros Delitzas et al. (+4 more)
Functional 3D scene graphs offer a versatile and flexible representation for 3D scene understanding and robotic manipulation, defined by object nodes, interactive elements, and functional relationship edges. However, their potential remains underexplored due to the limited coverage of existing
benchmarks and the overly straightforward design of previous pipelines, which primarily focus on large-scale furniture but lack of hierarchical structures. Therefore, in this work, we extend the
benchmark coverage by introducing dense tabletop objects and explicit multi-level functional relationships. This expansion introduces critical challenges involving small-scale, dense, and similar instances, with lack of visual anchoring in relational
reasoning, instance confusion during cross-frame fusion, and attribution uncertainty under dynamic viewpoints. To address these issues, we propose an open-vocabulary pipeline based on 2D visual grounding and 3D graph optimization. Specifically, we anchor fine-grained functional edges from 2D visual evidence, and associate nodes across frames in 3D using multiple cues. Furthermore, edge association is formulated as temporal graph optimization, integrating evidence accumulation,
entropy regularization, and temporal smoothing to robustly determine the functional connections of each node. Finally, global hierarchy shaping is performed to recover the hierarchical graph structure. Extensive experiments demonstrate that the proposed method can reliably infer functional 3D scene graphs in challenging real-world scenes, thereby further unlocking their potential for practical applications.
Read the full abstract ▾
All authors Xinggang Hu, Chenyangguang Zhang, Alexandros Delitzas, Xiangkui Zhang, Marc Pollefeys, Francis Engelmann, Xiangyang Ji
Computer Vision
HyperDiT: Hyper-Connected Transformers for High-Fidelity Pixel-Space Diffusion
Yu He, Lichen Ma, Zipeng Guo et al. (+5 more)
Pixel-space
diffusion models bypass the reconstruction bottleneck of Variational Autoencoders (VAEs) but face a fundamental "granularity dilemma": capturing global semantics favors large patch scales, while generating high-fidelity details demands fine-grained inputs. To address this issue, we propose HyperDiT, a unified framework establishing Hyper-Connected Cross-Scale Interactions to bridge the semantic and pixel
manifold. Diverging from injecting semantics by AdaLN, HyperDiT utilizes Cross-
Attention mechanisms, enabling fine-grained
tokens to query multi-level semantic anchors globally. To resolve the spatial mismatch during multi-scale interactions, we introduce Scale-Aware Rotary Position
Embedding (SA-RoPE) to ensure precise
geometric alignment among
tokens of varying patch sizes. Furthermore, we incorporate Registers to learn the dense semantics from a pretrained Visual Foundation Model (VFM), effectively reducing generation
hallucination and artifacts. Extensive experiments demonstrate that HyperDiT achieves state-of-the-art (SoTA) FID of $\mathbf{1.56}$ on ImageNet $256\times256$ directly within the pixel space. By combining the fine-grained stream with semantic guidance, HyperDiT offers a superior paradigm for high-fidelity pixel generation.
Read the full abstract ▾
All authors Yu He, Lichen Ma, Zipeng Guo, Xinyuan Shan, Jingling Fu, Dong Chen, Junshi Huang, Yan Li
Computer Vision
BARRIER: Bounded Activation Regions for Robust Information Erasure
Jan Miksa, Patryk Krukowski, Przemysław Spurek et al. (+2 more)
Machine unlearning aims to remove targeted concepts from a trained model while preserving the rest of its knowledge. Central challenge of this setting is that effective and robust erasure requires extensive parameter updates, which can unintentionally alter representations that should be retained. As a result, existing methods often trade erasure strength for preservation, due to the lack of formal guarantees on the protection of neutral concepts. To address this, we propose BARRIER (Bounded Activation Regions for Robust Information Erasure), a method that enables more intensive unlearning by driving updates within an identified activation space control region, where target erasure can be performed with limited collateral degradation. Using interval arithmetic, we obtain a closed-form bound on the worst-case representation change over protected regions and use it as a knowledge preservation objective. We provide a formal analysis of this protection and its effect on the functional drift. BARRIER is principled, architecture-agnostic, and compatible with existing erasure objectives. Empirical evaluations demonstrate that BARRIER achieves competitive performance across classification and generative settings, including notable gains in some cases, while maintaining strong robustness against
adversarial recovery attacks. Our code is available at https://github.com/OneAndZero24/BARRIER.
Read the full abstract ▾
All authors Jan Miksa, Patryk Krukowski, Przemysław Spurek, Dawid Damian Rymarczyk, Marcin Sendera
Computer Vision
BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
Huanyang Tong, Kai Liu, Fangjun Kuang et al. (+1 more)
Biomedical
Vision–
Language Models (VLMs) have shown remarkable promise in few-shot medical diagnosis but face a critical bottleneck: fragility to
prompt variations.Existing adaptation frameworks typically optimize visual and textual
prompts as independent streams, relying on ideal “Golden
Prompts”. In clinical reality, where descriptions are often noisy and heterogeneous, this modality isolation leads to unstable cross-modal
alignment. To address this, we propose BiomedAP, a
vision-informed dual-anchor framework with gated cross-modal fusion.BiomedAP enforces synergistic
alignment through two mechanisms: (1) Gated Cross-Modal Fusion, which enables layer-wise interaction between modalities, acting as a dynamic noise regulator to suppress irrelevant textual cues; and (2) a Dual-Anchor Constraint that regularizes learnable
prompts toward stable semantic centroids derived from both expert templates (High Anchors) and few-shot visual prototypes (Low Anchors). Extensive experiments across 11
benchmarks demonstrate that BiomedAP consistently surpasses baselines, achieving competitive few-shot accuracy and markedly enhanced robustness under
prompt perturbations. Our code is available at: https://github.com/tongdiedie/BiomedAP. Keywords:
Vision-
Language Models;
Prompt Learning; Parameter-Efficient Fine-Tuning; Few-shot Learning
Read the full abstract ▾
All authors Huanyang Tong, Kai Liu, Fangjun Kuang, Huiling Chen
Computer Vision
UAM: A Dual-Stream Perspective on Forgetting in VLA Training
Jianke Zhang, Yuanfei Luo, Yucheng Hu et al. (+6 more)
Vision–
language–action (VLA) models are typically built by fine-tuning a pretrained
vision–
language model (VLM) on action data. However, we show that this standard recipe systematically erodes the VLM's
multimodal competence, a side effect we call the embodiment tax. But do VLAs have to forget? Inspired by the two-stream organization of biological
vision, we trace this degradation to a structural bottleneck: current VLAs ask a single encoder to support both
language-grounded semantics and control-relevant visual features, whereas biological
vision separates recognition and visuomotor control into distinct pathways. Building on this view, we propose the Unified Action Model (UAM), which adds a parallel Dorsal Expert, an analog of the
brain's dorsal pathway. To make the Dorsal Expert an effective second pathway and reduce the control-learning burden on the VLM, we initialize it from a pretrained generative model and train it with a mid-level
reasoning objective that predicts visual dynamics. This design allows us to train the whole VLA end-to-end on action data alone: with no parameter freezing, no gradient stopping, and no auxiliary VL co-training, UAM retains over $95\%$ of the underlying VLM's
multimodal capability and at the same time achieves the highest average success rate among baselines on a variety of manipulation tasks that probe out-of-distribution generalization, including unseen objects, novel object–target compositions, and instruction variation. Together, these results suggest that semantic preservation in VLAs can emerge from architectural separation itself, rather than being enforced by frozen weights or auxiliary data replay, and that this preserved semantic capability can naturally transfer from VLMs to semantic generalization in actions.
Read the full abstract ▾
All authors Jianke Zhang, Yuanfei Luo, Yucheng Hu, Xiaoyu Chen, Yanjiang Guo, Ziyang Liu, Hongbin Xu, Tian Lan, Jianyu Chen
Computer Vision · Cross-listed from cs.NE
Structure Abstraction and Generalization in a Hippocampal-Entorhinal Inspired World Model
Tianqiu Zhang, Muyang Lyu, Xiao Liu et al. (+1 more)
Humans abstract experiences into structured representations to facilitate pattern inference and knowledge transfer. While the hippocampal-entorhinal (HPC-MEC)
circuit is known to represent both spatial and conceptual spaces, the mechanisms for concurrently extracting abstract structures from continuous, high-dimensional dynamics remain poorly understood. We propose a
brain-inspired hierarchical model that simultaneously infers latent transitions and constructs a predictive visual world model. Our architecture employs an inverse model for structural extraction alongside an HPC-MEC coupling model that dissociates relational structures (MEC) from integrated episodic scenes (HPC). Using primitive transformation dynamics as a
benchmark, we demonstrate the model's capacity for structural abstraction. By leveraging velocity-driven path integration, the framework enables robust prediction and structural reuse across diverse contexts, thereby achieving structural generalization. This work provides a novel computational framework for understanding how
brain-inspired, self-supervised learning of world models facilitates the acquisition of reusable abstract knowledge.
Read the full abstract ▾
All authors Tianqiu Zhang, Muyang Lyu, Xiao Liu, Si Wu
Computer Vision
DecomPose: Disentangling Cross-Category Optimization Contention for Category-Level 6D Object Pose Estimation
Yifan Gao, Lu Zou, Zhangjin Huang et al. (+1 more)
Category-level 6D object pose estimation is typically formulated as a multi-category joint learning problem with fully shared model parameters. However, pronounced
geometric heterogeneity across categories entangles incompatible optimization signals in shared modules, resulting in gradient conflicts and negative transfer during training. To address this challenge, we first introduce gradient-based diagnostics to quantify module-level cross-category contention. Building on results of diagnostics, we propose DecomPose, a difficulty-aware decomposition framework that mitigates optimization contention via: (1) difficulty-aware gradient decoupling, which groups categories using a data-driven difficulty proxy and routes each instance to a group-specific correspondence branch to isolate incompatible updates; and (2) stability-driven asymmetric branching, which assigns higher-capacity branches to structurally simple categories as stable optimization anchors while constraining complex categories with lightweight branches to suppress noisy updates and alleviate negative transfer. Extensive experiments on REAL275, CAMERA25, and HouseCat6D demonstrate that DecomPose effectively reduces cross-category optimization contention and delivers superior pose estimation performance across multiple
benchmarks.
Read the full abstract ▾
All authors Yifan Gao, Lu Zou, Zhangjin Huang, Guoping Wang
Computer Vision
DiLA: Disentangled Latent Action World Models
Tianqiu Zhang, Muyang Lyu, Yufan Zhang et al. (+2 more)
Latent Action Models (LAMs) enable the learning of world models from unlabeled
video by inferring abstract actions between consecutive frames. However, LAMs face a fundamental trade-off between action abstraction and generation fidelity. Existing methods typically circumvent this issue by using two-stage training with pre-trained world models or by limiting predictions to optical flow. In this paper, we introduce DiLA, a novel Disentangled Latent Action world model that aims to resolve this trade-off via content-structure disentanglement. Our key insight is that disentanglement and latent action learning are co-evolving: the predictive bottleneck inherent in latent action learning serves as a driving force for disentanglement, compelling the model to distill spatial layouts into the structure pathway while offloading visual details to a separate content pathway for generation. This synergy yields a continuous, semantically structured latent action space without compromising generative quality. DiLA achieves superior results in
video generation quality, action transfer, visual planning, and
manifold interpretability. These findings establish DiLA as a unified framework that simultaneously achieves high-level action abstraction and high-fidelity generation, advancing the frontier of self-supervised world model learning.
Read the full abstract ▾
All authors Tianqiu Zhang, Muyang Lyu, Yufan Zhang, Fang Fang, Si Wu
Computer Vision · Cross-listed from cs.LG
GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspective
Xu Wang, Xunkai Li, Yinlin Zhu et al. (+2 more)
Multimodal retrieval uses images, text, and object relationships to answer different questions about the same collection. A query may seek an object's paired description, another object in the same category, or an object connected by an observed relationship. These goals rely on different notions of relevance. Paired matching requires object-specific distinctions, whereas cross-object
retrieval benefits from relational agreement. Existing methods learn strong cross-modal correspondence or propagate information over a graph, but a shared output regularized toward neighbors can weaken identity distinctions. Moreover, gains from graph regularization can diminish after uniform graph propagation. We introduce Graph-Optimized
Multimodal Alignment (GOMA), which assigns these roles to two connected
embeddings. Each modality produces a content
embedding directly supervised for paired identity and a semantic
embedding jointly trained with cross-modal pairs and observed relationships. For complete records, both
embeddings form an initial fused representation, semantic agreement sets positive weights on observed edges, and restart graph propagation reinjects this initial signal. This design lets a jointly trained model support single-modality and dual-attribute
retrieval through a task-specific readout. Across six
datasets and four tasks, GOMA achieves state-of-the-art performance on all 14 primary measures against 14 external methods. Controlled comparisons further show how separate supervision, graph regularization, and semantic-guided graph propagation shape the final representation and align the learned signal with each
retrieval target.
Read the full abstract ▾
All authors Xu Wang, Xunkai Li, Yinlin Zhu, Rong-Hua Li, Guoren Wang
Computer Vision · Cross-listed from cs.LG
Bidirectional Fusion Guided by Cardiac Patterns for Semi-Supervised ECG Segmentation
Jeonghwa Lim, Minje Park and Sunghoon Joo
Accurate delineation of electrocardiogram (ECG), the
segmentation of meaningful waveform features, is crucial for cardiovascular diagnostics. However, the scarcity of annotated data poses a significant challenge for training deep learning models. Conventional semi-supervised semantic
segmentation (SemiSeg) methods primarily focus on consistency from unlabeled data, underutilizing the information exchange possible between labeled and unlabeled sets. To address this, we introduce CardioMix, a framework built on a bidirectional CutMix strategy guided by cardiac patterns for ECG
segmentation. This approach enriches the labeled set with realistic variations from unlabeled data while simultaneously applying stronger supervisory signals to the unlabeled set, as the cardiac pattern-guided mixing ensures all augmented samples remain physiologically meaningful. Our framework is designed as a plug-and-play module, demonstrating high compatibility with various SemiSeg
algorithms. Extensive experiments on SemiSegECG, a public multi-
dataset benchmark for ECG delineation, demonstrate that CardioMix consistently outperforms existing CutMix-based fusion strategies across diverse
datasets and labeled ratios as a plug-and-play module compatible with various SemiSeg
algorithms.
Read the full abstract ▾