Computer Vision
Envisioning Beyond the Few: Disentangled Semantics and Primitives for Few-Shot Atypical Layout-to-Image Generation
Nan Bao, Yifan Zhao, Wenzhuang Wang et al. (+1 more)
The layout-to-image (L2I) task enables fine-grained control over image generation via object categories and spatial layouts. However, existing L2I methods yield fragmented and distorted generations under few-shot atypical settings. We term this failure as representation fragmentation, arising from a granularity mismatch that entangles semantic identity with visual details. To address this issue, we propose a representation-driven framework that disentangles semantics from primitives for robust few-shot adaptation. Specifically, Semantic Anchoring aggregates categorical semantics into anchors for stable identity, while Primitive Imbuing models recomposable primitives for robust local detail modeling. Conceptual Steering further regulates optimization with a saliency-aware objective to preserve foreground semantic consistency. Extensive experiments demonstrate consistent improvements in the 5-shot regime over state-of-the-art L2I methods in both visual fidelity and
alignment across diverse atypical domains. The source code is publicly available at https://github.com/iCVTEAM/DSP.
Read the full abstract ▾
All authors Nan Bao, Yifan Zhao, Wenzhuang Wang, Jia Li
Computer Vision
Kaiwen Xue, Tao Wei, Guoxin Zhang et al. (+5 more)
Multimodal large
language models (MLLMs) have shown strong potential as
embodied agents, yet
embodied geo-localization remains underexplored due to the lack of fine-grained evaluation. We introduce ERGeoBench, a diagnostic
benchmark for
vision-driven
embodied geo-localization. ERGeoBench evaluates models under three progressive settings – single-view, panorama-view, and
embodied-view – where
agents may actively acquire observations through sequential changes in yaw, pitch, and zoom. The
benchmark contains 2,207 globally distributed street-view panoramas and measures four complementary capabilities: foundational perception, spatial awareness, common sense
reasoning, and geo-localization
reasoning. Evaluations of leading proprietary and open-source MLLMs show that current models can infer high-level geographic semantics, but still struggle with fine-grained perceptual operations, metric localization, and spatial consistency across views. We further observe that geo-localization is strongly correlated with the other capability dimensions, suggesting that accurate localization depends on integrated perception, spatial
reasoning, and commonsense inference rather than isolated visual recognition. Overall, ERGeoBench provides a unified framework for diagnosing and advancing human-like
embodied geo-localization. Project Page: https://kaixuewen.github.io/ERGeoBench/
Read the full abstract ▾
All authors Kaiwen Xue, Tao Wei, Guoxin Zhang, Zhonghong Ou, Kaoyan Lu, Yu Feng, Yifan Zhu, Haoran Luo
Computer Vision · Cross-listed from cs.CR
BadBone: Backdoor Attacks Against Backbone Models in Visual Prompt Learning
Ziqing Yang, Rui Wen, Xinlei He et al. (+3 more)
Prompt learning is a new machine learning paradigm that has attracted ample
attention due to its simplicity and proven efficacy. Despite its growing adoption, the security vulnerabilities associated with this paradigm remain underexplored. In this work, we take the first step to propose BadBone, a stealthy and adaptive
backdoor attack against
prompt learning using bi-level optimization. Instead of backdooring the
prompt learning process, we aim to compromise a backbone model such that only target downstream tasks employing
prompt learning inherit the
backdoor vulnerability. Extensive experiments on three different models and three
datasets from various domains show that our targeted/untargeted backdoored models achieve high attack performance while maintaining utility on both pre-training and downstream tasks. Moreover, we evaluate our approach against six state-of-the-art model-level defenses, including Neural Cleanse, ABS, MNTD, NAD, CLP, and D-BR. The results demonstrate that these defenses are largely ineffective against our backdoored models and thus leave the effective defense as an important direction for future work.
Read the full abstract ▾
All authors Ziqing Yang, Rui Wen, Xinlei He, Yun Shen, Michael Backes, Yang Zhang
Computer Vision
Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval
Alicja Dobrzeniecka, Filip Szatkowski, Sebastian Cygert et al. (+2 more)
While
retrieval is a core function of
vision-
language models, continually updating these models for
retrieval tasks remains critically underexplored. Existing work often approaches continual
retrieval through the lens of class-incremental learning (CIL), evaluating both standard CIL methods and
retrieval-oriented adaptations in settings that may not fully capture the
retrieval-specific dynamics. To address this, we introduce a new, principled evaluation framework for continual
multimodal retrieval (CMR) spanning diverse visual domains, and systematically evaluate common approaches within this setting. Our empirical analysis shows that standard CIL methods fail to yield meaningful gains in our more challenging scenario. Therefore, we propose Dynamic Adapter Routing (DAR), a novel approach based on adapters selected through prototype-based routing and combined via model merging.DAR achieves superior performance over the previous baselines and demonstrates strong generalization under out-of-distribution evaluation. Our results highlights the unique challenges of CMR and encourages further research in this direction.
Read the full abstract ▾
All authors Alicja Dobrzeniecka, Filip Szatkowski, Sebastian Cygert, Szymon Lukasik, Bartlomiej Twardowski
Computer Vision
HiERO-StepG @ Ego4D Step Grounding Challenge: hierarchical activity understanding enables zero-shot step grounding
Andrea Zenotto, Simone Alberto Peirone, Francesca Pistilli et al. (+1 more)
Procedural activities follow well-defined structures: whether we consider a cooking recipe or a mechanic repairing a car, these activities naturally decompose in a hierarchy of steps and sub-steps. Traditional approaches for step grounding require extensive annotations and scale poorly. Instead, we argue that such hierarchical structure can emerge naturally from uncurated
videos of human activities through recurring patterns of co-occurring actions and activities. Our approach builds on HiERO, a weakly-supervised representation learning approach that maps close in the feature space actions that are functionally related to each other, leveraging only fine-grained action-level narrations. In this feature space, procedure steps can be detected by a simple clustering, with no additional task-specific fine-tuning. For the Ego4D Step Grounding challenge, we augment this approach by ensuring fine and coarse level agreement in step assignments, enforcing strict temporal monotonicity of the grounded steps and post-processing the detected steps to reduce the impact of noisy predictions. We call this approach HiERO-StepG and it achieves 56.27 % on the R@1 (IoU = 0.3) metric on the global leaderboard at submission time, ranking second while being completely zero-shot and not requiring procedure-specific annotations. Project page: https://github.com/andreazenotto/HiERO-StepG.
Read the full abstract ▾
All authors Andrea Zenotto, Simone Alberto Peirone, Francesca Pistilli, Giuseppe Averta
Computer Vision
Latent Geometric Chords for Query-Efficient Decision-Based Adversarial Attacks
Ei Hmue Khine, Yao Li, Jiebao Sun et al. (+3 more)
While decision-based black-box
adversarial attacks present a severe security threat, current methodologies suffer from fundamental limitations. Pixel-wise attacks frequently introduce unnatural, high-frequency visual artifacts, while latent-space frameworks are confined by the limited search space of low-dimensional
manifolds and inherent reconstruction flaws. To resolve these limitations, we propose Latent
Geometric Chords (LGC) for Query-Efficient Decision-Based
Adversarial Attacks alongside a variant, LGC-H. At its core, LGC navigates decision boundaries by executing a
curvature-aware
geometric search within a compressed semantic
manifold. To guarantee high visual fidelity and circumvent dimensionality bottlenecks, we introduce a Residual-based
Adversarial Generation (RAG) mechanism. RAG isolates semantic perturbations as
geometric chords and superimposes them directly onto the original source image. RAG substantially resolves baseline reconstruction flaws and effectively doubles the permissible search space dimensions. Experimental results demonstrate that LGC achieves robust cross-
dataset transferability and substantially outperforms state-of-the-art baselines. Notably, our method, LGC, minimizes perturbation magnitudes while achieving state-of-the-art visual fidelity–with a Structural Similarity Index Measure (SSIM) exceeding 0.99 and a Learned Perceptual Image Patch Similarity (LPIPS) below 0.01 at 5000 queries–and sustaining high attack success rates under stringent perceptual constraints, successfully compromising adversarially trained robust models. The source code is available at: https://github.com/eihmuekhine/Latent-
Geometric-Chords.
Read the full abstract ▾
All authors Ei Hmue Khine, Yao Li, Jiebao Sun, Shengzhu Shi, Zhichang Guo, Boying Wu
Computer Vision
TALON: Token-Aligned Lightweight Adapters for 6-DoF Spacecraft Pose Estimation
Abid Ali, Nidhal Eddine Chenni, Arunkumar Rathinam et al. (+1 more)
Monocular 6-DoF spacecraft pose estimation methods predominantly process individual frames, discarding the temporal information present in an image sequence acquired during spacecraft manoeuvres. Few temporal approaches require full backbone fine-tuning or auxiliary optical flow networks, risking catastrophic forgetting or increasing computational cost, respectively. We propose TALON (
Token-Aligned Lightweight adapters for Orbital
Navigation): spatiotemporal 3D adapters injected before the self-
attention layers of a frozen ViT
vision transformer, combined with a patch-
token alignment loss that geometrically grounds the adapted features to keypoint structure through a prototype-conditioned KL-divergence objective. Pre-
attention placement allows the frozen
attention to reason over temporally enriched
tokens, achieving stronger performance with a single adapter per block than post-
attention alternatives. The
alignment loss shapes the intermediate representations so that each keypoint induces a spatially precise activation in the
token field, while the framework adds less than 5% parameters to the frozen backbone. On SPADES
dataset, TALON reduces the pose error by 50% over the prior state-of-the-art, and on SwissCube
dataset it surpasses the prior best by 21.8% in ADD-0.1d accuracy. Zero-shot cross-domain evaluation from sim-to-real on SPARK real data reduces pose error by 4.7x, and ablations characterise the role of adapter depth across in-domain and cross-domain settings.
Read the full abstract ▾
All authors Abid Ali, Nidhal Eddine Chenni, Arunkumar Rathinam, Djamila Aouada
Computer Vision · Cross-listed from cs.LG
Fixed-Point Masked Generative Modeling
Andrea Miele, Yiming Qin, Alba Carballo-Castro et al. (+2 more)
Masked Generative Models (MGMs) enable parallel decoding and achieve strong performance across modalities, but require full-sequence bidirectional
transformers at every step, making training costly and degrading quality under low sampling budgets. Existing work improves efficiency via better samplers or cheaper fixed-depth denoisers, but they still allocate a fixed amount of denoiser computation to each refinement step. We introduce Fixed-Point Masked Generative Models (FP-MGMs), which replace part of the denoiser with a fixed-point solver over shared
attention layers to enable adaptive depth with fewer parameters. To make it more effective for masked generation, we first introduce a cross-step consistency loss, which aligns hidden representations at neighboring denoising steps and, second, three-state reuse (3SR) which warm-starts the solver using the previous solution by treating differently unchanged, still-masked, and newly revealed
tokens respectively. Together, these components define our complete training-to-inference framework for fixed-point masked generation, CoFRe. We also show that pre-trained MGMs can be converted into FP-MGMs with short fine-tuning, avoiding full retraining. Across modalities, CoFRe improves the quality and cost trade-off. On OpenWebText, CoFRe reduces parameters by 38.8%, training time by 11.5%, and VRAM by 16.9%, while improving generative perplexity from 830.8 to 101.8 at a budget of $96$
transformer-block forward passes, compared to MDLM. In ImageNette, CoFRe reduces training time by 48.6% and VRAM by 50.7%, while improving FID in all sample budgets tested. Overall, CoFRe offers a practical framework for cheaper training and stronger low-budget masked generation.
Read the full abstract ▾
All authors Andrea Miele, Yiming Qin, Alba Carballo-Castro, Justin Deschenaux, Pascal Frossard
Computer Vision
Benchmarking and Enhancing Text-to-Image Models for Generating Visual Representations in Early Arithmetic Education
Junling Wang, Boqi Chen, Heejin Do et al. (+3 more)
AI systems are increasingly used to support educational content creation, yet it remains unclear whether they can generate outputs that faithfully represent the pedagogical concepts they are intended to teach. Thus, we introduce equation-to-visual generation, a task that, in contrast to conventional image generation, requires producing pedagogically meaningful visuals from arithmetic equations while precisely preserving their numerical and relational structure. Informed by interviews with teachers and an analysis of educational materials, we construct E2V-Bench, a
benchmark spanning four pedagogically grounded visual types, along with automatic metrics for evaluating visual correctness. Our evaluation reveals that recent text-to-image (T2I) models frequently fail on this task, with errors dominated by incorrect object counts and broken relational structure. Building on this, we explore
benchmark-guided enhancement strategies. These strategies improve representative models, while the remaining gap calls for stronger numerical and relational grounding in future T2I models.
Read the full abstract ▾
All authors Junling Wang, Boqi Chen, Heejin Do, Mubashara Akhtar, April Yi Wang, Mrinmaya Sachan
Computer Vision
Probabilistic Precipitation Nowcasting with Rectified Flow Transformers
Johannes Schusterbauer, Jannik Wiese, Nick Stracke et al. (+2 more)
Accurate weather forecasts are essential across various domains and are safety-critical in extreme weather conditions. Compared to simulation-based
forecasting, data-driven approaches show greater efficiency, enabling short-term, high-resolution nowcasting. In particular,
diffusion models proved effective in weather nowcasting due to their strong probabilistic foundation. However, existing methods rely on deterministic compression to reduce the complexity of high-dimensional weather data, limiting their ability to capture uncertainty in the decoding process. In this work, we introduce $\textbf{FREUD}$, a $\textbf{Fr}$ame-wise $\textbf{E}$ncoder and $\textbf{U}$nited $\textbf{D}$ecoder model based on rectified flow
transformers for efficient compression of spatio-temporal weather data. Frame-wise encoding enables continuous forecast updates, while the unified
video decoder ensures temporal consistency. Our uncertainty-preserving first stage allows us to capture aleatoric uncertainty via ensembling, which is particularly beneficial for extreme weather events with high decoding variability. We achieve state-of-the-art performance in precipitation nowcasting with a compact latent-space rectified flow
transformer on the SEVIR
benchmark and show further performance gains by model and test-time scaling. Code available here: https://github.com/CompVis/weather-rf
Read the full abstract ▾
All authors Johannes Schusterbauer, Jannik Wiese, Nick Stracke, Timy Phan, Björn Ommer
Computer Vision
Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration
Jun Wang, Xiaohao Xu and Xiaonan Huang
Safe human–
robot collaboration requires more than visual description: a monitor must determine whether the
robot body is safely separated, already colliding with the scene or a person, or about to collide. We call this capability collision grounding: binding visual observations to
robot body geometry, camera viewpoint, scene layout, human proximity, and temporal motion in order to infer present and imminent contact. We introduce TouchSafeBench, a physics-grounded
benchmark for evaluating collision grounding in
vision-
language models (VLMs). Built in Habitat~3.0, TouchSafeBench contains 2,940 simulated indoor co-presence episodes across social
navigation and social rearrangement, with synchronized multi-view RGB-D observations, top-down trajectory maps, calibrated camera metadata, and simulator-derived contact labels. We study two deployment-facing tasks: classifying the current safety state and warning about imminent collision before contact. Across three frontier or robotics-oriented VLMs and nine visual representations, current models remain far from reliable: the best average Macro-F1 stays below 50%, explicit depth is not automatically transformed into
robot-body collision evidence, and
robot–scene contact is consistently harder than human-contact risk. TouchSafeBench reveals a central limitation of
embodied VLMs: visual fluency does not imply physical accountability. Reliable
robot safety monitors will need representations that explicitly bind viewpoint,
robot morphology, metric geometry, and future collision. The
benchmark is available at https://jun25336-bit.github.io/TouchSafeBench/ .
Read the full abstract ▾
Computer Vision
Specialist-Generalist Fusion with Outcome-Supervised Rationales for Deepfake Detection
Benedikt Hopf, Zongwei Wu and Radu Timofte
Generalizable deepfake detection requires complementary forensic and semantic visual evidence. Specialist encoders capture subtle manipulation traces but can overfit to source-specific statistics, whereas MLLMs provide broader visual-semantic representations but may overlook fine forensic artifacts. We propose a two-stage detector in which an MLLM directly fuses patch-level features from a frozen forensic encoder with those from its native
vision encoder and produces the authenticity decision itself. This specialist–generalist
alignment provides the main cross-domain performance gain. We subsequently introduce outcome-supervised rationale tuning. The model generates a free-form visual rationale before its decision and is optimized using only the binary authenticity label and a format constraint, without task-specific rationale annotations. Rationale generation is optional at inference, so the tuned model can still return a direct binary score. On DF40, specialist–generalist
alignment improves average cross-domain AUC from $89.85$ to $93.32\pm0.44$. Across six paired runs, rationale tuning obtains $93.50\pm0.42$ and improves five of six paired models; the mean difference is small and not statistically conclusive. Results on SID-Set and legacy
benchmarks, together with fusion, output-order, and continued-training controls, demonstrate the value of complementary visual representations and show that label-only rationale tuning can add an optional explanation mode while approximately preserving direct detection performance.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10
Umut Onur Yasar
We investigate how teacher-student capacity relationships modulate knowledge
distillation (KD) effectiveness in ResNet-based image classification on CIFAR-10. Across four teacher-student pairs (R50->R18, R34->R18, R50->R34, and R101->R34) we compare Logit-KD and Feature-KD under a strict evaluation protocol: hyperparameters and checkpoints are selected on a held-out validation split, selected configurations are re-run with five seeds, and the test set is used exclusively for final reporting. Beyond accuracy, we measure
distillation fidelity directly via teacher-student agreement and KL divergence. We report four findings. First, the student-capacity pattern survives the corrected protocol at reduced magnitude: the only statistically significant gains occur for R34 students under Feature-KD (+0.19 and +0.21 pp, p<0.05), in two pairs whose teachers differ two-fold in parameters but not in accuracy, localizing the moderating variable on the student side, while no KD gain for R18 students is distinguishable from zero. Second, Feature-KD matches or outperforms Logit-KD in all four pairs, and its students land closer to the teacher's output distribution (KL at T=1) than Logit-KD students despite never observing teacher logits. Third, top-1 teacher-student agreement is flat across all pairs, decoupling fidelity from accuracy gains. Fourth, architecture dominates KD: correcting the ResNet stem for 32x32 inputs is worth +5.5 to +7.2 pp, more than 25x the largest KD gain. We also retract an attribution made in v1: a controlled re-run shows the reported gradient-clipping bug had no measurable effect, and v1's larger gains are explained by test-set selection. Code and results: github.com/umutonuryasar/kd-capacity-gap (tag v2.0).
Read the full abstract ▾
Computer Vision
From Local Geometry to Global Pseudo Labeling for Robust Positive Unlabeled Learning under Covariate Shift
Firas Gabetni, Alexandre Rocchi, Nacim Belkhir et al. (+2 more)
Detecting
covariate shift is critical for building reliable
vision systems. While most prior work focuses on improving robustness to shift, explicitly detecting
covariate shift remains underexplored. Existing approaches typically rely on fully supervised training, requiring labeled examples from both original and shifted distributions, which is often impractical. In this paper, we show that
covariate shift detection can be effectively addressed with weaker supervision using Positive Unlabeled (PU) learning. However, under
covariate shift, in distribution and shifted data overlap significantly, making classical PU methods unstable and sensitive to noise. To overcome this challenge, we introduce Spectral PU Neighborhood Annotation (SPUNA), a geometry aware framework that progressively discovers shifted data by leveraging the local
manifold structure of visual features. Extensive experiments show that SPUNA achieves state of the art performance in PU settings and remarkably matches the performances of fully supervised methods. Moreover, our approach transfers robustly across different types of shifts, demonstrating strong generalization capabilities. Code is available at https://github.com/fira7s/S-PUNA
Read the full abstract ▾
All authors Firas Gabetni, Alexandre Rocchi, Nacim Belkhir, Ziyi Liu, Gianni Franchi
Computer Vision
VaViT: Vanilla ViT for Automotive Point Cloud Semantic Segmentation
Gilles Puy, Nermin Samet, Alexandre Boulch et al. (+3 more)
Plain
Transformers have become the de-facto architecture for processing text, audio, image, and
video, offering a unified backbone for
multimodal learning. However, state-of-the-art architectures for point cloud semantic
segmentation remain dominated by U-Nets architectures where convolutions are interleaved with local or windowed attentions. In this work, we show how to effectively leverage vanilla, non-hierarchical ViTs for the
segmentation of large-scale automotive
lidar scenes. We bridge the performance gap thanks to a carefully designed tokenizer, a lightweight decoder
segmentation head, and tailored data augmentations. Our approach, VaViT, for Vanilla ViT, matches or exceeds the performance of state-of-the-art methods while maintaining the simplicity of ViT architecture. We provide extensive evaluations on nuScenes, SemanticKITTI, and Waymo Open
Dataset to validate the efficiency of our method. Code and pre-trained models are available at https://github.com/valeoai/VaViT.
Read the full abstract ▾
All authors Gilles Puy, Nermin Samet, Alexandre Boulch, Spyros Gidaris, Tuan-Hung VU, Renaud Marlet
Computer Vision
Detect in Any Scene: An Agentic Framework for Object Detection with Experience-Aware Reasoning
Wenlun Zhang, Jun Yin and Kentaro Yoshioka
Object detection in real-world scenarios remains challenging due to diverse image degradations and heterogeneous object distributions, which significantly hinder the generalization of existing detectors. Conventional approaches, including scene-specific representation learning and end-to-end pipeline design, are inherently limited by their reliance on predefined conditions and lack adaptability to dynamic environments. In this paper, we propose DetAS, an agentic detection framework that formulates object detection as a dynamic decision process. Instead of relying on static pipelines, DetAS leverages a
Multimodal Large
Language Model (MLLM) as a central
agent to adaptively compose detection workflows by selecting from a toolbox of restoration modules and specialized detectors. Specifically, DetAS consists of two key components: Self-Adaptive Image Restoration, which dynamically determines whether and how to enhance images for downstream detection, and Multi-Expertise Detection, which integrates multiple domain-specialized detectors and resolves their predictions through instance-level
reasoning. To further improve decision quality under fine-grained conditions, we introduce Self-Evolving Experience Harvesting and extend the framework to DetAS-X, which accumulates node-level decision experience from a small set of annotated data and enables experience-aware
reasoning during inference. This mechanism allows the system to progressively refine its decision policy and adapt to diverse real-world scenarios. Extensive experiments on six challenging
benchmarks demonstrate that DetAS-X significantly outperforms existing MLLM-based detectors, achieving an average improvement of 28.36% in F1 score, with up to 37.01% gain on DarkFace. These results demonstrate the promise of agentic detection and establish a solid foundation for its application in complex and dynamic environments.
Read the full abstract ▾
Computer Vision
Guidance for Low-Level Perceptual Editing in Unconditional Diffusion Models
Shreyansh Modi, Akshat Tomar and Aarush Aggarwal
Unconditional
diffusion models offer powerful generative priors, yet steering them toward aesthetically enhanced outputs remains largely unexplored. We show that h-space patching, the dominant paradigm for training-free
diffusion editing, systematically fails for global, low-level transformations required for aesthetic and perceptual refinement. We introduce a novel, generalized framework for image-editing in unconditional
diffusion models without explicit training. This inference-time mechanism operates on low-level features by extracting degradation concept vectors and combining bottleneck patching with classifier-free guidance to guide sampling away from the degraded
manifold, producing consistently improved images without any model retraining.
Read the full abstract ▾
Computer Vision
Light Interaction: Training-Free Inference Acceleration for Interactive Video World Models
Jiacheng Lu, Haoyi Zhu, Sipei Yi et al. (+3 more)
Interactive
video world models generate
video chunk by chunk in response to user-controlled camera movements, enabling applications such as real-time game simulation, virtual scene
navigation, and
embodied AI training. However, scaling to long interactive trajectories is prohibitively expensive due to growing context
memory, quadratic
attention complexity, and repeated denoising steps. We present Light Interaction, a training-free inference acceleration framework for interactive
video world models. Our key insight is that interaction naturally enables trajectory-dependent adaptive computation: retrieved spatial
memory can be discarded during novel exploration, temporal context can be adjusted according to local latent dynamics, and early-step model outputs can be reused when the camera revisits familiar regions. Based on this insight, Light Interaction combines adaptive context management, denoising cache acceleration, and hardware-software co-designed 3D block sparse
attention with fused Triton kernels. Evaluated on HY-WorldPlay and
Matrix-Game-3.0, Light Interaction achieves up to 2.59x speedup without model retraining while maintaining competitive visual quality.
Read the full abstract ▾
All authors Jiacheng Lu, Haoyi Zhu, Sipei Yi, Enze Xie, Yu Li, Cheng Zhuo
Computer Vision
BIAS-ID: A Framework for Analyzing Transformation Biases in AI-Generated Image Detectors
Jonas Ricker, Asja Fischer and Erwin Quiring
Given the surge of harmful AI-generated imagery online, reliably distinguishing authentic images from generated ones has become an urgent research topic. While many proposed detection methods perform well under controlled settings, they often collapse when tested on real-world data. A potential root cause are subtle biases in the detectors' training data. As a result, detectors may rely on spurious correlations instead of learning true forensic artifacts. While a recent line of work has identified the problem, there is not yet an established protocol to evaluate how biased a detector actually is. In this work, we therefore take a step back: First, we discuss what it means for a detector to be biased, and how this differs from a lack of robustness. Second, we propose BIAS-ID, a transparent framework for analyzing and quantifying the presence of transformation biases in AI-generated image detectors. We validate our framework by performing an evaluation of six detectors across two
datasets, revealing that several state-of-the-art detection methods are strongly affected by biases. Our results highlight the importance of bias-aware evaluation for developing reliable AI-generated image detectors.
Read the full abstract ▾
Computer Vision
SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
Tianhui Liu, Jie Feng, Zhiheng Zheng et al. (+6 more)
Humans can effortlessly perceive spatial layouts, form cognitive representations, reason about spatial relations, and translate such
reasoning into actions in everyday 3D environments. Although recent
vision-
language models (VLMs) have shown promising performance on observation-conditioned spatial perception and
reasoning tasks, it remains unclear whether they can build coherent spatial understanding, act upon it, and refine their actions through multi-turn feedback. To study this problem, we introduce SpatialAct, a simulator-grounded
benchmark for probing action-conditioned spatial
reasoning in 3D scenes. Starting from the most challenging setting, Multi-turn Interactive Refinement, we further design its decomposed counterpart, Single-step Error Detection and Fix, together with five fundamental spatial ability tasks to diagnose the underlying causes of model failures. Experiments reveal a clear
reasoning-to-action gap: current VLMs can perform well on isolated spatial
reasoning tasks, but struggle to maintain coherent spatial beliefs and produce reliable actions during multi-turn feedback, substantially underperforming humans. These results suggest that current VLM
agents still lack robust spatial state tracking under action-induced environment changes, even when low-level control is abstracted away.
Read the full abstract ▾
All authors Tianhui Liu, Jie Feng, Zhiheng Zheng, Shengyuan Wang, Yiming Guo, Yanxin Xi, Hangyu Fan, Yong Li, Pan Hui
Computer Vision
FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization
Mohammed Asad Karim and Vinay Kumar Verma
In-context localization (ICL) seeks to localize a target object specified by a small set of support examples in a query image, operating on the fly without training or parameter updates. Despite rapid advances in
vision-
language models (VLMs), achieving category-agnostic and visually grounded ICL remains an open problem, even though it is essential for applications such as image editing, personalized visual search, and
retrieval. Existing methods are fragile and rely on explicit category supervision, which not only limits applicability in realistic settings with unnamed or instance-specific objects but also introduces category bias that steers predictions toward semantic priors rather than visual evidence. We introduce a two-stage training framework that explicitly optimizes in-context
attention between support bounding boxes and query images without category supervision. We further refine localization via reinforcement learning using Group Relative Policy Optimization (GRPO) to directly minimize localization error. This formulation enforces visual correspondence over semantic priors, yielding robust instance-level localization. Empirically, a 7B-parameter model trained with our objectives outperforms models up to 72B parameters, demonstrating that context-aware localization objectives can surpass scaling alone. Comprehensive ablations validate the contribution of each component.
Read the full abstract ▾
Computer Vision
PolSAR Image Classification using a Hybrid Complex-Valued Network (HybridCVNet)
Mohammed Q. Alkhatib
Recently, convolutional neural networks (CNNs) have become popular for image classification due to their effectiveness in computer
vision tasks. Now, researchers are exploring the potential of
vision transformers (ViTs) in remote sensing and Earth observation. However, traditional Real-Valued networks often overlook important phase information in Complex-Valued (CV) data like polarimetric synthetic aperture radar (PolSAR) data. To address this, new CV deep architectures have emerged. HybridCVNet, a novel hybrid network, blends CV-CNN and CV
vision transformer (CV-ViT) techniques. It efficiently combines CV 3D and 2D CNNs as feature extractors, enhancing PolSAR image classification by extracting complementary information and effectively leveraging interdependencies within the data. Experimental results from widely-used PolSAR
datasets show HybridCVNet outperforms other methods, achieving an overall accuracy of 97.39% on the Flevoland
dataset and showing promise even with just a 1% sampling ratio, with a Kappa value of 0.972 on the San Francisco
dataset. Source code is accessible through https://github.com/mqalkhatib/HybridCVNet
Read the full abstract ▾
Computer Vision
QVGGT: Post-Training Quantized Visual Geometry Grounded Transformer
Zhizhen Pan, Hesong Wang and Huan Wang
Estimating 3D attributes directly from images has advanced rapidly with the Visual Geometry Grounded
Transformer (VGGT), which predicts camera parameters, depth maps, and point clouds in a single forward pass. However, its 1.2B-parameter scale severely limits deployment on resource-constrained platforms such as
UAVs and mobile AR devices. To address this limitation, we introduce QVGGT, a tailored quantization framework designed to compress VGGT. Our approach starts from the observation that
transformer blocks within VGGT exhibit heterogeneous sensitivity to quantization. We thus analyze per-block quantization sensitivity and propose a selective mixed-precision strategy that allocates higher precision to the most fragile
transformer blocks. To address the amplification of quantization error caused by high-variance camera and register
tokens, we further introduce
token filtering with camera information compensation, which removes these outliers from activation calibration and restores their
geometric cues using a PCA-derived global compensation
token. Finally, we develop a task-aware scale search mechanism that evaluates candidate quantization scales not only through layer reconstruction but also through multi-head supervision and cross-head
geometric consistency among camera poses, depth maps, and point maps. Extensive experiments on multiple geometry perception
benchmarks demonstrate that QVGGT achieves near-lossless W4A16 quantization, preserving the accuracy of all 3D prediction heads while delivering 3$\sim$4.9$\times$
memory reduction and up to 2.8$\times$ real hardware speedup over FP32. Our approach makes high-fidelity 3D perception feasible on edge devices, enabling practical deployment of feed-forward 3D reconstruction models in real-world constrained environments.
Read the full abstract ▾
Computer Vision
NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving
Jiahui Li, Jiawei Sun, Zixiang Ren et al. (+7 more)
Recent perception-free end-to-end (E2E) autonomous driving methods bypass explicit perception outputs by compressing dense image patch
tokens into compact scene
tokens for downstream trajectory generation and scoring. While these scene
tokens form a compact visual bottleneck for the planner, they receive supervision solely from the planning objective, providing limited constraints on the encoded visual information. To address this limitation, we introduce Neural
Token Reconstruction (NTR), a representation learning framework to directly constrain the compact scene-
token bottleneck in perception-free driving. NTR introduces a self-
distillation masked latent reconstruction objective that reconstructs masked patch-level latent features using only compact scene
tokens as reconstruction
memory. This forces reconstruction gradients to pass exclusively through the scene-
token bottleneck, encouraging scene
tokens to preserve richer and less redundant visual representations for planning. We further introduce semantic priors derived from foundation-model annotations as a weak semantic interface biasing reconstruction targets toward driving-related structures without introducing explicit perception heads. All auxiliary reconstruction components are removed at inference time, leaving the deployed planner unchanged. NTR achieves state-of-the-art performance on three public autonomous driving
benchmarks, including 8.0461 RFS on Waymo E2E and 94.1 PDMS / 90.9 EPDMS on NavSim1&2. The learned scene
tokens exhibit lower pairwise redundancy and higher effective rank, indicating that effective bottleneck supervision improves both compact visual representation learning and planning performance.
Read the full abstract ▾
All authors Jiahui Li, Jiawei Sun, Zixiang Ren, Ming Liu, Jiamin Shi, Ruiteng Zhao, Zhiyang Liu, Liying Liu, Zuoguan Wang, Kaidi Yang
Computer Vision
Polyphony: Diffusion-based Dual-Hand Action Segmentation with Alternating Vision Transformer and Semantic Conditioning
Hao Zheng, Hu Wang, Tiantian Zheng et al. (+2 more)
Dual-hand action
segmentation, densely predicting actions for both hands from untrimmed
videos, is essential for understanding complex bimanual activities. However, it poses several unique challenges: complex inter-hand dependencies, visual asymmetry between hands, representation conflicts where the dominant hand monopolizes gradients, and semantic ambiguity in fine-grained actions. We propose Polyphony, a three-stage method to address these challenges through: (1) an Alternating Dual-Hand
Vision Transformer that alternates training between left- and right-hand mini-batches to ensure balanced gradient contributions from both hands while sharing a spatio-temporal encoder; (2) Semantic Feature Conditioning that aligns visual features with structured, compositional action descriptions to enhance discrimination of semantically similar actions; and (3)
Diffusion-Based
Segmentation with cross-hand feature fusion for inter-hand coordination and adaptive loss weighting for balancing performance. Polyphony achieves state-of-the-art on both dual-hand
datasets (HA-ViD, ATTACH) with improvements up to 16.8 points, and on the single-stream Breakfast
dataset (82.5%), outperforming the prior best method that uses a 12x larger backbone. Notably, our unified model with a single shared backbone surpasses baselines requiring separate per-hand models. Code is at https://github.com/x-labs-xyz/Polyphony-Dual-hand-Action-
Segmentation.
Read the full abstract ▾
All authors Hao Zheng, Hu Wang, Tiantian Zheng, Prajjwal Bhattarai, Tuka Alhanai
Computer Vision
Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams
Jonathan Swinnen and Tinne Tuytelaars
In this work we introduce a novel approach to domain incremental learning, adapting models over time to evolving, non-stationary data. In contrast to other works, we do not attempt to avoid catastrophic forgetting, but rather allow it and exploit it. Our model combines a main task head with a self-supervised masked autoencoder (MAE) head. We then learn domain-specific LoRA adapters during incremental training. Each adapter specializes to its domain, naturally inducing forgetting on other domains in both heads. At inference, we perform online test-time training on the self-supervised MAE head to identify which LoRAs best matches the current input, so the model can `remember' the domain again. Our scheme is especially well-suited to real-world streaming data, such as
video, where consecutive samples are highly correlated and domain shifts are gradual. We demonstrate our method on domain-incremental action recognition and semantic
segmentation tasks.
Read the full abstract ▾
Computer Vision
iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning
Chang-Bin Zhang, Yujie Zhong, Qiang Zhang et al. (+1 more)
While visually grounded Chain-of-Thought (CoT) has emerged as a promising paradigm to enhance fine-grained perception in
multimodal large
language models (MLLMs), its efficacy during the inference phase remains underexplored. In this work, we empirically find that mandating explicit object boxes in visually grounded CoT during inference often degrades performance compared to standard textual CoT, which reasons without explicit visual grounding. We hypothesize that the visual localization capability can be internalized into the textual CoT and that the mandatory explicit grounding introduces unnecessary interference with the model's primary objective of answer prediction. To address this problem, we propose Internalizing Visually Grounded
Reasoning (iVGR), a novel reinforcement learning framework that transfers localization capabilities into the textual
reasoning process. We employ a dual-stream training strategy, where a textual stream is aligned with a high-quality visually grounded stream via a proposed consistency reward, enabling the model to localize accurately without explicit grounding during inference. Extensive experiments demonstrate that our method significantly outperforms existing baselines on fine-grained
benchmarks, while maintaining the flexibility to support tool-assisted inference workflows.
Read the full abstract ▾
All authors Chang-Bin Zhang, Yujie Zhong, Qiang Zhang, Kai Han
Computer Vision
Redefining Instance Matching: A Unified Framework for Part-Aware Matching in Panoptic Segmentation Evaluation
Erik Großkopf, Soumya Snigdha Kundu, Hendrik Möller et al. (+9 more)
The Panoptic Quality (PQ) metric is the standard for jointly evaluating instance and semantic
segmentation. However, its original definition relies on a One-to-One matching between predicted and ground truth segments, which is only straightforward when the IoU threshold exceeds 0.5. Below 0.5, multiple matching strategies emerge in a poorly explored problem space. We systematically elucidate this space by recasting segment matching as a constrained bipartite assignment problem. Independently bounding the prediction- and ground-truth-side degrees yields four matching strategies: One-to-One, Many-to-One, One-to-Many, and Many-to-Many. We show that the first three are well-defined within the PQ framework, while Many-to-Many falls outside it. These strategies become relevant when instances are fragmented, adjacent objects are difficult to delineate, or annotations are noisy. Central to our framework is a vertex-based accounting of TP, FN, and FP, anchored to ground truth and predicted segments rather than to matching edges. We further show that the framework extends naturally to part-aware panoptic
segmentation, and we explore part-aware evaluation on biomedical data. Across configurable case studies we report how different combinations of thresholds and matching strategies behave in practice. We release a unified open-source package built on Panoptica. It exposes Voronoi-based region-wise analysis, part-aware evaluation, and Area Under Threshold Curve computations as configurable options.
Read the full abstract ▾
All authors Erik Großkopf, Soumya Snigdha Kundu, Hendrik Möller, Nicolas Münster, Mehdi Astaraki, Paula Tamara Buzduga, Kerstin Ritter, Benedikt Wiestler, Jan Kirschke, Jonathan Shapey, Tom Vercauteren, Florian Kofler
Computer Vision
Cross-Modal Clinical Knowledge Integration for Mammography Report Generation
Jiayi Zhu, Fuxiang Huang, Yu Xie et al. (+7 more)
Breast cancer is a major global health concern, and mammography screening plays a central role in early detection. The large volume of screening examinations creates a substantial workload for radiologists, making accurate and consistent report generation a critical clinical challenge. Existing automated mammography report generation methods primarily focus on direct visual-to-text mapping, while overlooking the structured clinical
reasoning process followed by radiologists in real-world practice. To address this limitation, we propose MammoRG, a mammography report generation framework that explicitly simulates the clinical reporting workflow by following the BI-RADS guideline and incorporating prior clinical knowledge to produce diagnostic reports. Specifically, MammoRG adopts a two-stage training framework. In the first stage, the model learns to integrate clinically relevant prior knowledge from a patient's four-view mammograms through classification-based supervision. In the second stage, a terminology-aware supervised fine-tuning strategy is introduced to model mammography-specific clinical terms as atomic semantic units, enabling the generation of high-quality reports with improved clinical consistency. To facilitate clinical efficacy evaluation of generated reports, we further develop MammoRGTool, a dedicated mammography report parsing tool that extracts structured clinical information from free-text reports. Extensive experiments demonstrate that MammoRG consistently outperforms existing methods across multiple clinical efficacy metrics, particularly in diagnosis-related BI-RADS F1, where it surpasses the second-best model by 2.73%, 2.04%, 1.90%, and 3.27% on the internal, external 1, external 2, and VinDr-Mammo
datasets, respectively.
Read the full abstract ▾
All authors Jiayi Zhu, Fuxiang Huang, Yu Xie, Xi Wang, Zhixuan Chen, Yuan Guo, Qingcong Kong, Zhenhui Li, Qiong Luo, Hao Chen
Computer Vision
On Revisiting Entropy for Identifying Mislabeled Images
Chunlei Li, Zixuan Zheng, Yilei Shi et al. (+5 more)
Mislabeled samples in training
datasets severely degrade the performance of deep networks, as overparameterized models tend to memorize erroneous labels. We address this challenge by proposing a novel approach for mislabeled data detection that leverages training dynamics. Our method is grounded in the key observation that correctly labeled samples exhibit consistent
entropy decrease during training, while mislabeled samples maintain relatively high
entropy throughout the training process. Building on this insight, we introduce a signed
entropy integral (SEI) statistic that captures both the magnitude and temporal trend of prediction
entropy across training epochs. SEI is broadly applicable to classification networks and demonstrates particular effectiveness when integrated with contrastive
language-image pretraining (CLIP) architectures. Through extensive experiments on four medical imaging
datasets – a domain particularly susceptible to labeling errors due to diagnostic complexity – spanning diverse modalities and pathologies, we demonstrate that SEI achieves state-of-the-art performance in mislabeled data identification, outperforming existing methods while maintaining computational efficiency and implementation simplicity. Our code is available at https://github.com/MedAITech/SEI.
Read the full abstract ▾
All authors Chunlei Li, Zixuan Zheng, Yilei Shi, Guanglu Dong, Pengfei Li, Jingliang Hu, Xiao Xiang Zhu, Lichao Mou