Computer Vision · Cross-listed from econ.GN
Representation Risk in Pretrained Image Encoders
Ardyn Nordstrom, Morgan Nordstrom, Vamuyan Sesay et al. (+1 more)
Applied researchers increasingly convert images into features with pretrained encoders, then use those features in a downstream prediction model. The encoder is often treated as an implementation detail. We show that it can instead be a consequential source of model uncertainty. We call this uncertainty representation risk: plausible pretrained encoders map the same images into different feature spaces and can yield sharply different out-of-sample conclusions from predictive performance. We compare ten modern and legacy frozen encoders across applications involving house prices, racehorse performance, breast-cancer histology, chest radiographs, continuous facial age, and rice disease. With common dimension control, heads, and group-safe splits, validation selects SigLIP 2 for houses, raising test $R^2$ from 0.396 for ResNet50 to 0.629, and DINOv2 for horses, raising $R^2$ from 0.029 to 0.105. No encoder is best in every task. Candidate procedures are constructed using training data and compared on a separate validation partition. The selected procedure reaches 0.658 for houses and 0.979 accuracy for pneumonia. Fixed-split gains are small for horses and rice, while repeated partitions reveal instability in horse feature union. Continuous age selects SigLIP 2 at 4.786 years MAE. The principal representation gaps persist with neural heads, similarly sized DINOv2 and ViT models, and limited adaptation. These results support a simple workflow:
benchmark plausible representations, select on locked validation data, combine only when separate validation evidence justifies the additional cost, and report paired and split-level uncertainty. We implement this workflow in LOOKAGAIN-ML, the software package used to conduct the analyses in this paper.
Read the full abstract ▾
All authors Ardyn Nordstrom, Morgan Nordstrom, Vamuyan Sesay, Matthew D. Webb
Computer Vision
W2Rep: Learning Visual Representations by Watching the World Change
Wen Huang, Hang Guo, Jiarui Yang et al. (+3 more)
Images capture the world at one moment, whereas
video reveals how it changes. Image self-supervision learns spatial structure from a single moment, while
video methods commonly learn temporal relationships inside a representation computed jointly from several frames. We ask whether watching a scene change can instead improve features available from one image without sacrificing the ability to represent
video. We introduce W2Rep, a masked feature-prediction framework in which an independently encoded source image participates in prediction at the same or another moment. The predictor is conditioned on visible
video context, the queried location, and the signed time interval between source and target. This gives the cross-frame objective two complementary roles: the image path learns features that remain useful across time, while the
video path must gather evidence that is missing from the source image. Across model scales and downstream tasks, W2Rep improves frozen and fine-tuned recognition under our comparison protocol, while joint
video encoding provides further gains over frame-wise aggregation. Controlled experiments show that these gains depend on directly updating the source-image features and on using both
video context and temporal displacement. Overall, change across a
video can supervise a visual encoder whose representations remain useful at either image or
video granularity. Code is available at~\href{https://wenooi.github.io/W2Rep}{https://wenooi.github.io/W2Rep}.
Read the full abstract ▾
All authors Wen Huang, Hang Guo, Jiarui Yang, Zheng Liu, Tao Dai, Shu-tao Xia
Computer Vision
How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Lin Chen, Bolin Ni, Qi Yang et al. (+6 more)
Most modern
multimodal large
language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: (1) Removing the visual encoder shifts the compute-optimal allocation for the
multimodal objective toward larger models, while leaving that for text nearly unchanged. (2) The two architectures exhibit nearly overlapping loss–compute frontiers on the text objective, but diverge on the
multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around $10^{22}$ FLOPs, well within practical pretraining budgets. (3) Without a visual encoder, the
language model learns to take over its role via
vision-specific adaptation: bidirectional interactions among visual
tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual
tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for
multimodal pretraining.
Read the full abstract ▾
All authors Lin Chen, Bolin Ni, Qi Yang, Lan Jiang, Kun Ding, Xiaoran Fan, Hower Yang, Ying Wang, Shiming Xiang
Computer Vision
From internal representations to model improvement through prediction errors
Yushi Nakaya, Kenichi Higuchi and Shuichi Ishida
With limited annotation budgets, choosing which images to label determines how much a model improves. Data-selection methods that use features from a separately trained model, or scene descriptions written by
vision-
language models, have been successful, but those signals do not directly capture changes in the model being improved. The target model's own internal features reflect what it has learned so far and change with retraining, making them a natural cue for choosing the next training data. However, feature rarity alone does not reveal the errors that matter for performance. Here we link internal features to prediction errors and their expected impact on performance and select images for labeling and retraining without using labels for candidate images. We evaluated the method with an object detector on two
datasets and two pairs of random seeds. Adding internal features improved the identification of prediction errors in 15 of 16 conditions. When performance was averaged over successive labeling rounds, the method outperformed selection based only on feature rarity in all four evaluation settings and ranked among the top two of six methods. With other conditions held fixed, performance after retraining was again higher than with rarity-based selection, even though the latter collected more errors. With longer retraining, the proposed method ranked first among six methods. These results suggest that linking a model's internal features to its errors and their effects on performance may help select training images that improve performance, thereby allowing the model's current state to guide which images are labeled next.
Read the full abstract ▾
Computer Vision
DiMoP: Diffusion-Driven Motion Representation Learning With Frame-Level Pseudo-Classification for Skeleton-Based Action Recognition
Shanaka Ramesh Gunasekara, Wanqing Li, Nikalal Kaldera et al. (+2 more)
Robust skeleton-based action recognition requires representations that capture a wide spectrum of motions, from subtle to moderate and strong ones. Existing methods often focus on strong motions. This paper introduces DiMoP, a masking- and
diffusion-driven motion representation learning method with frame-level pseudo-classification to explicitly learn the distribution of joint motions rather than regressing deterministic coordinates, as existing methods often do. By diffusing masked joints with progressive noise and denoising them conditioned on visible joints, DiMoP learns through controllable noising and denoising processes, enabling uniform learning of weak, moderate, and strong dynamics. To enable the masking-based generative
diffusion learning with a discriminative capability, a pseudo-frame classifier is proposed that enforces the learning towards sequence-consistent and temporally coherent pseudo-labels without manual annotations. Together, these strategies provide a principled mechanism for joint generative and discriminative motion modeling. DiMoP achieves state-of-the-art performance across NTU RGB+D 60/120, and PKUMMD, including a 1.1 percentage point gain over prior works on NTU RGB+D 120 with the cross-subject protocol.
Read the full abstract ▾
All authors Shanaka Ramesh Gunasekara, Wanqing Li, Nikalal Kaldera, Philip Ogunbona, Jack Yang
Computer Vision · Cross-listed from cs.RO
Revision, Not Restart: Revisable Visual Plans for Closed-Loop World-Action Models
Pengyiang Liu, Junbo Niu, Wenhao Zheng et al. (+5 more)
World-action models use predicted visual futures to condition
robot actions, yet execution feedback can invalidate parts of a prediction while leaving its task structure useful. We propose Revisable Temporal Planning (RTP), which maintains the visual future as a persistent action condition and revises it after feedback. Its central mechanism is a learned revision bridge: it resumes an intermediate state saved during visual generation and adapts its continuation to current observations. Visual and action supervision connect this revision to subsequent control. Time-aware history supplies observed evidence, and an adaptive policy selects retention, bridge revision, or fresh replanning from new noise before decoding the next action. On RoboMME and RMBench, RTP achieves task-averaged success rates of 48.6% and 84.8%, respectively. Matched comparisons support learned continuation; estimated checkpoint-source and action-prefix effects are positive but less precisely resolved. These results connect feedback-driven visual-plan revision to closed-loop task performance. Project Page: https://PLACEHOLDER.github.io/RTP/
Read the full abstract ▾
All authors Pengyiang Liu, Junbo Niu, Wenhao Zheng, Xinchen Chen, Canyu Li, Zhongyue Shi, Jiahao Xie, Si Liu
Computer Vision · Cross-listed from q-bio.QM
An integrated geometric quantification and shape analysis framework for axillary lymph node metastasis in breast cancer patients
Zixi Yi, Limeng Qu and Gary P. T. Choi
Quantitative characterization of lymph node morphology is important for assessing axillary lymph node metastasis in breast cancer. However, surfaces reconstructed from computed
tomography (CT) segmentation may contain
geometric and topological defects that compromise subsequent analysis, while conventional shape descriptors predominantly characterize global morphology. To address these issues, we developed an integrated framework combining
topology-aware surface processing with multi-resolution spherical harmonic (SH) analysis of CT-derived axillary lymph nodes. The processing pipeline produced
topology-valid genus-0 surfaces with improved mesh quality, which were then represented at multiple SH degrees and characterized using 20 predefined
geometric feature families.
Geometric fidelity increased with SH degree, whereas predictive performance peaked at intermediate resolutions. Preferred SH degree also differed across feature families. A family-specific mixed-resolution model achieved an AUC of 0.918, compared with 0.884 for the conventional PyRadiomics Shape14 baseline, corresponding to an improvement of 0.0344. Controlled perturbation experiments showed that higher SH degrees transmitted more fine-scale
geometric variation and yielded lower stability of
curvature-based predictions. Representative
geometric descriptors provided interpretable characterization of metastasis-associated surface morphology. Independent validation further supported the framework's transportability: label-free replication in a multicenter lymph node cohort reproduced the family-specific resolution effects, while a labeled LIDC-IDRI lung-nodule experiment reproduced the resolution-dependent relationship between SH degree and predictive performance. Altogether, the framework provides a
topology-valid basis for quantitative characterization of lymph node morphology and metastasis-associated imaging phenotypes.
Read the full abstract ▾
Computer Vision
Automated Species Identification in Camera Trap Images for Wildlife Conservation
Nowshin Amin, Nafisa Tabassum Oyshi, Tahmid Abrar Zidan et al. (+2 more)
Wildlife conservation involves protecting, preserving, and managing wildlife species and their habitats. With today's rapid pace of human development,
climate change, and other unsustainable practices, the need for wildlife conservation has heightened. Despite significant progress in species identification using deep-learning models, significant challenges still remain in effectively detecting small animals in low-contrast trap images due to limited feature extraction capabilities. This thesis presents a novel end-to-end framework integrating a self-
attention mechanism to address these limitations. The proposed architecture involves a Swin-BiFPN backbone integrated in a Faster RCNN detection network, coupled with a visual semantic extraction module driven by the LLaVA v1.5 (13B)
multimodal large
language model. The detection framework, capable of extracting crucial features in challenging trap images, demonstrates consistently high results and robust generalization capabilities. Furthermore, the visual semantic extraction module provides zero-shot detection capability, as well as providing valuable insights and emergent cues of the animal's behavior, further supporting the conservation effort. The MLLM evaluation was conducted using both traditional NLP metrics (precision, recall, F1, and SBERT similarity) and subjective scoring by LLM-based judges (GPT-4.1 and GROK 3.0), across five MLLMs, demonstrating the model's strong performance in visual description generation. The proposed framework improves detection accuracy across low-contrast trap images and small animals while also demonstrating zero-shot detection capability leveraging the MLLM.
Read the full abstract ▾
All authors Nowshin Amin, Nafisa Tabassum Oyshi, Tahmid Abrar Zidan, Miftaun Noor, Md. Abrar Rahman Shafin
Computer Vision
When Should the Count Change? Learning State Maintenance for Causal Video Counting
Pengyiang Liu, Dongyue Lyu, Junbo Niu et al. (+3 more)
Continuous
video counting requires distinguishing new observations from new objects or completed events. We introduce StaMina (State Maintenance), which learns to maintain counting state through state-conditioned updates. Recurrent visual context supports recognition; learned transitions maintain visibility, persistent identities, and completed-event records. A differentiable recurrence trains event transitions over legal paths constrained by count endpoints; visibility and association objectives train the object branch. A multi-source pipeline organizes 39.8K spatial queries and complementary event annotations into counting trajectories. On SVCBench, we evaluate counting adaptation with partial
video overlap and held-out groups of linked annotations. Under prefix replay (Full) and persistent streaming (Stream), 4B and 8B models reach 41.9/36.4 and 44.9/38.2 Gaussian Precision Accuracy, respectively. The 8B model gains 10.9/3.2 points over Counting-SFT on the same queries. Matched-graph comparisons isolate phase conditioning and trajectory supervision, assessing training objectives alongside hard decisions. Online
video benchmarks and count-conditioned decisions assess online understanding and task eligibility. Project Page: https://PLACEHOLDER.github.io/StaMina/
Read the full abstract ▾
All authors Pengyiang Liu, Dongyue Lyu, Junbo Niu, Zhongyue Shi, Jiahao Xie, Si Liu
Computer Vision · Cross-listed from cs.RO
Adaptive Safety Filtering for Frozen ACC Policies via Conformal Residual Calibration
Zhiruo Zhou, Rigaudiere Z. Li, Chen Xiwen et al. (+3 more)
Frozen adaptive cruise control (ACC) policies can violate constraints when deployment dynamics differ from their training conditions. We propose residual-aware conformal action filtering (RACF), which calibrates residuals of a fixed nominal predictor and converts their
quantile into an operating margin for finite-model action projection. Completed transitions update margins and candidate selection without retraining the policy. In a registered comparison over 2,400 controller-trial units, Adaptive RACF achieves 94.3% episode safety, improving by 19.9 percentage points over the evaluated nominal CBF-QP baseline while reducing projection frequency from 8.11% to 6.63%. A controlled study isolates a 4.54-point improvement from residual-margin injection. In a separate matched-hardware evaluation, Adaptive reduces mean amortized rollout time by 21.2% relative to Robust CBF-QP, with 161/180 versus 170/180 safe episodes. We characterize conditions linking one-step residual coverage to constraint satisfaction and quantify the observed safety-computation trade-offs.
Read the full abstract ▾
All authors Zhiruo Zhou, Rigaudiere Z. Li, Chen Xiwen, Yucheng Chen, Xiaojun Zhu, Houde Liu
Computer Vision
Spectral Super-Resolution using Spatial-Spectral Residual Operator Networks
Seokhyun Chin
Spectral super-resolution of multispectral satellite images can enable high temporal- and spatial-resolution hyperspectral satellite imagery at a modest cost, significantly increasing the applicability of hyperspectral remote sensing. This task is inherently ill-posed, making it well-suited for deep learning-based methods. In this study, the spectral super-resolution task is framed as an operator learning problem, and SSRON is proposed as a Deep Operator Network that effectively learns function-to-function mappings from downsampled spectra to continuous spectra. The model is trained to super-resolve Sentinel-2A-like multispectral imagery to EMIT images. Compared to baseline models, SSRON achieves superior performance across all metrics. The model also demonstrates zero-shot spectral super-resolution capability by predicting bands unseen during training. Furthermore, its continuous-output formulation suggests the potential to estimate spectra at finer wavelength intervals than the native sensor. These results suggest the potential of SSRON and establishes operator learning as a promising direction for spectral super-resolution.
Read the full abstract ▾
Computer Vision
BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion
Wanjiang Weng, Yongliang Wu, Xiaofeng Tan et al. (+3 more)
Text-to-motion generation and motion-to-text captioning are two fundamental tasks in human motion modeling, both grounded in the same underlying motion-text correspondence. Existing unified approaches mostly rely on autoregressive modeling, which imposes a fixed generation order and is therefore poorly suited to the bidirectional dependencies between
language and motion, allowing early prediction errors to persist as fixed context and degrade both temporal coherence and cross-modal consistency. Masked discrete
diffusion, which models sequences through iterative bidirectional prediction, offers a natural remedy. We therefore propose BiMoGen (Bidirectional Motion-text Generation), a unified masked discrete
diffusion framework for bidirectional motion-text modeling. To stabilize training, we design Decoupled Uni- and Cross-Modal Training, in which masked pretraining first establishes cross-modal correspondence on paired motion-text sequences, after which supervised fine-tuning specializes the model for bidirectional generation. Masked
diffusion nonetheless introduces its own source of error, as the model is trained on clean ground-truth context yet encounters self-generated and potentially erroneous context at inference, with errors committed under heavily masked states propagating through subsequent steps. We further introduce Generation-Aware Self-Correction that exposes the model to its own predictions during training and applies correction passes at early sampling steps to revise unreliably committed
tokens. Extensive experiments on HumanML3D and KIT-ML demonstrate competitive performance on both tasks, validating the effectiveness of the proposed two-stage training and self-correction designs. The project page is available at https://wengwanjiang.github.io/BiMoGen-Page.
Read the full abstract ▾
All authors Wanjiang Weng, Yongliang Wu, Xiaofeng Tan, Xingyu Zhu, Wenbo Zhu, Hongsong Wang
Computer Vision
Reduce, Then Encode: Multiscale Volumetric Reduction for 2D Foundation Models in Brain MRI
Dexuan Ding, Yuankai Qi, Bogong Wang et al. (+2 more)
Pretrained 2D foundation models offer a practical alternative to dedicated 3D pretraining for
brain structural
magnetic resonance imaging (sMRI), but their use on volumetric data requires bridging the mismatch between a 2D encoder and a 3D volume input. Existing methods typically encode slices independently and integrate their features afterwards. We introduce Multiscale Volumetric Reduction (MVR), a reduce-then-encode approach that compresses each anatomical view from (D) slices into (M << D) complementary 2D components before foundation-model encoding. MVR combines an uncentered-PCA base component derived from the original through-plane intensities with residual detail components constructed from multiscale spatial descriptors. The reduction is estimated from the training volumes without diagnostic labels or gradient-based optimization and remains fixed thereafter. The resulting components are independently processed by a shared frozen 2D foundation model and concatenated for linear probing. Under this frozen-encoder setting, MVR achieves strong overall performance across ADNI, OASIS, and ABIDE relative to the evaluated 2D-to-3D adaptation methods and simple input-reduction baselines, while also generalizing strongly from ADNI to AIBL.
Read the full abstract ▾
All authors Dexuan Ding, Yuankai Qi, Bogong Wang, Luping Zhou, Amin Beheshti
Computer Vision
Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline
Xiao Zhang, Wang Zeng, Sheng Jin et al. (+3 more)
Video Large
Language Models (
Video LLMs) have achieved remarkable progress in
video understanding, but their inference efficiency is constrained by the large number of visual
tokens produced by long
videos. Recent
video token compression methods increasingly introduce sophisticated strategies for
token selection,
pruning, and merging. This raises a fundamental question: how much of compression performance can be obtained by simply preserving the structure encoded in the visual representations? We investigate this question with SimpleCluster, a simple and training-free baseline that performs position-aware cross-frame clustering in the visual feature space and represents each cluster using the mean of its original visual features. Extensive experiments across four
video understanding
benchmarks and three representative
Video LLMs show that SimpleCluster achieves competitive or superior performance over recent compression methods across a wide range of
token retention ratios, with particularly strong robustness under extremely low retention rates (e.g., 1%). To understand this behavior, we analyze the feature space preserved by different compression methods in terms of local approximation fidelity and global coverage. The results show that stronger downstream performance is consistently associated with better preservation of the original visual feature distribution, especially its global coverage. These findings highlight feature-space preservation as an important consideration for
video token compression under highly constrained
token budgets. Our code is available at https://github.com/xiaozhang79/SimpleCluster.
Read the full abstract ▾
All authors Xiao Zhang, Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Shichao Kan
Computer Vision
Ego-Forge: Text and Geometric-Attention Free Exo-to-Egocentric Video Generation
Mohammad Mahdi, Luc Van Gool and Danda Pani Paudel
Exo-to-egocentric
video generation aims to synthesize what a person sees from their own viewpoint given third-person footage and a target head trajectory. The task requires transferring appearance and semantics across large viewpoint changes while hallucinating content never observed by the exocentric camera. Existing approaches either impose additional input requirements, such as a ground-truth initial egocentric frame or multiple synchronized exocentric views, or remain limited to category-specific settings. EgoX is the first to address cross-activity and in-the-wild generalization, but requires a human-provided caption of the non-existent egocentric view at inference and introduces a computationally expensive geometry-guided
attention bias that can propagate reconstruction errors and suppress textual and visual context. We therefore propose Ego-Forge, a caption-free and bias-free framework for exo-to-egocentric generation. It introduces Dynamic Captioning, which derives conditioning
tokens directly from the model's hidden states and adapts them to the
diffusion timestep and network depth, replacing external text conditioning. By scaling training by an order of magnitude and using all available exocentric viewpoints, Ego-Forge learns cross-view correspondence implicitly and eliminates the need for geometry-guided
attention, requiring only a lightweight depth prior. Ego-Forge achieves state-of-the-art performance on Ego-Exo4D, runs faster end-to-end, requires no external annotation at inference, and generalizes to in-the-wild scenes, including cases where over-reliance on geometry blocks appearance inference. Our model and source code will be made publicly available.
Read the full abstract ▾
Computer Vision
Generative Uncertainty as a Self-supervised Signal for Semantic Similarity Learning
Enrico Pallotta, Sina Raoufi, Lars Doorenbos et al. (+2 more)
Evaluating semantic similarity between
videos is a fundamental challenge in computer
vision, essential for tasks ranging from out-of-distribution (OOD) detection to
video retrieval. However, defining and labeling
video similarity is notoriously difficult and expensive due to the complex spatio-temporal nature. In this paper, we propose a novel self-supervised approach that leverages generative uncertainty from text-to-
video (T2V)
diffusion models to learn semantic similarity without human annotations. Our method is based on the observation that T2V models produce consistent outputs for familiar concepts but exhibit high variance and uncertainty when prompted with specialized concepts. We utilize this behavior to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA. Specifically, we learn a mask over these
embeddings using purely generated data, encouraging the model to retain features that remain consistent across generations of general concepts while discarding those associated with generative noise or uncertainty. Experimental results across three key tasks demonstrate that our learned feature subspaces consistently outperform original pretrained features and baseline feature selection methods.
Read the full abstract ▾
All authors Enrico Pallotta, Sina Raoufi, Lars Doorenbos, Gianni Franchi, Juergen Gall
Computer Vision · Cross-listed from cs.AI
TMCS: Tool-Grounded Multi-Agent Reasoning for Compositional Chemical Problem Solving
Shengqin Wang, Jie Jin, Yu Cheng et al. (+4 more)
Despite the promise of Large
Language Models (LLMs) in computational chemistry, rigorous combinatorial chemistry problems remain difficult because they require quantitatively constrained molecular modification, candidate validation, and systematic revision after failed attempts. Existing tool-augmented chemical
agents demonstrate useful planning and tool use, but they rarely provide a unified loop for property-driven molecular optimization and workflow-level composition. To bridge this gap, we propose Tool-Grounded Multi-
Agent Reasoning for Compositional Chemical Problem Solving (TMCS), a step-by-step multi-
agent framework that formalizes chemical problem solving as an interpretable, tool-augmented workflow. At the task level, specialized
agents leverage external tools, few-shot trajectory
memory, and structured reflection to iteratively refine solutions. At the workflow level, TMCS chains generation, understanding, editing, description, and optimization into a closed-loop pipeline. Evaluations across multiple chemical tasks demonstrate that TMCS consistently enhances chemical
reasoning across both open- and closed-source base models, achieving state-of-the-art performance.
Read the full abstract ▾
All authors Shengqin Wang, Jie Jin, Yu Cheng, Yihang Chen, Weilin Luo, Yuan Xie, Zhizhong Zhang
Computer Vision
RoGSW4RLD: Feed-Forward 4D Gaussian Lifting for Robot World Model Rollouts
Jin Hyun Kim, Min Young Kim, Soohwan Song et al. (+1 more)
Action-conditioned
video world models predict future
robot interactions from multiple cameras, yet their outputs remain disparate
video collections rather than a shared metric scene queryable across viewpoints and time. While existing 4D reconstruction methods offer a path to spatialize these predictions, independently reconstructing and merging each camera stream fails to enforce cross-view consistency. This limitation is particularly detrimental when combining moving
robot-mounted cameras with fixed external views. To address this, we introduce RoGSW4RLD, a feed-forward framework that lifts synchronized multi-camera rollouts into a unified, time-queryable metric 4D Gaussian field. Rather than learning a separate
geometric transition model, RoGSW4RLD directly reconstructs the visual future generated by existing world models. Its core innovation is a two-stage architecture: Stage 1 jointly forms the metric 4D field by fusing cross-view evidence with
robot-specific articulated geometry and kinematics, while Stage 2 refines the field's geometry and appearance while strictly preserving the initial temporal displacements. Evaluated on 256 held-out DROID episodes, RoGSW4RLD significantly outperforms camera-wise reconstruction with calibrated merging, improving novel-view PSNR by 2.15 dB, reducing depth AbsRel by 47%, and lowering
robot displacement error by 61%. These robust gains extend to action-conditioned Cosmos 3 rollouts, demonstrating that predicted
video futures can be successfully translated into consistent, spatially queryable 4D metric representations.
Read the full abstract ▾
All authors Jin Hyun Kim, Min Young Kim, Soohwan Song, Daekyum Kim
Computer Vision
PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents
Jiazhou Zhou, Hu Zhou, Yucheng Chen et al. (+3 more)
Reinforcement learning with verifiable rewards (RLVR) via Group-Relative Policy Optimization (GRPO) is widely used for multi-turn VLM
agent training, yet it suffers from zero-gradient silence on uniform failures and coarse episode-level credit assignment. While On-Policy
Distillation (OPD) and On-Policy Self-
Distillation (OPSD) mitigate sparse rewards using hindsight information, their underlying mechanisms remain poorly understood. Through controlled counterfactual rollback probes across five multi-turn VLM
agent benchmarks, we reveal that performance gains in OPSD/OPD are largely driven by physical state rollback at the pivot step, defined as the first unrecoverable action without remaining step budget. However, physical state rollbacks are computationally prohibitive and infeasible in real-world environments. To bridge this gap, we present Pivot-Aware Internalized Visual On-Policy Training (PIVOT), an RL framework that internalizes pivot localization and state restoration directly into
token-level parameter updates, eliminating environment rollbacks during RL training and additional skill hints at test time. PIVOT unifies three functional roles within a single architecture: a failure Analyzer non-invasively localizes the pivot step and diagnoses failure modes from visual trajectory collages and action logs; a detached Teacher re-scores failed
tokens under this privileged diagnostic context; and a Student optimizes joint GRPO and confidence-gated OPD objectives. At test time, both Teacher and Analyzer branches are stripped. Evaluated on five multi-turn VLM
agent tasks across cognitive grid puzzles, 3D embodied control and navigation, and generative
reasoning, PIVOT achieves 0.90 overall accuracy on Qwen2.5-VL-3B (+8% over SFT+GRPO baseline and +5% over previous SOTA) and scales to 0.92 on Qwen3-VL-2B (+12% over SFT+GRPO baseline).
Read the full abstract ▾
All authors Jiazhou Zhou, Hu Zhou, Yucheng Chen, Jinyuan Qu, Ying-Cong Chen, Lei Zhang
Computer Vision
Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Models
Xingming Long, Jie Zhang, Yuecong Min et al. (+2 more)
Object
hallucination remains a major challenge for large
vision-
language models. While off-policy preference optimization proves to be an effective solution, on-policy reinforcement learning provides a more promising direction as it directly targets a model's current failure modes. However, we find that without fine-grained reward formulation and allocation, on-policy optimization often falls into an easy shortcut: reducing
hallucinations merely by saying less—making fewer valid claims. To comprehensively resolve this, we propose a fine-grained
alignment framework that couples dense reward signals at the data level with precise credit assignment at the algorithmic level. Specifically, we first construct the Dense Object Presence and Absence (DOPA)
dataset to address sparse annotations that prevent valid object claims from being verified and rewarded. DOPA exhaustively annotates the deterministic presence and absence of every concept across an expanded vocabulary, significantly increasing the density of reliable reward signals during on-policy rollouts. Second, we propose Subsentence-level Credit Assignment for on-Policy Optimization (SCAPO) to prevent response-level shared advantages from allowing local
hallucinations to compromise all other valid outputs within the same response. By assigning credit to each subsentence independently based on its object claims, SCAPO can precisely reinforce faithful generations and penalize
hallucinations. Furthermore, we leverage the resulting faithful image descriptions as auxiliary context to transfer generative gains to discriminative tasks. Experiments demonstrate that our method produces highly informative, faithful descriptions in generative tasks while yielding clear performance gains on discriminative evaluation.
Read the full abstract ▾
All authors Xingming Long, Jie Zhang, Yuecong Min, Shiguang Shan, Xilin Chen
Computer Vision
Scaffold Then Internalize: Representation Injection for Diffusion Transformers
Han Fu, Jiacheng Chen, Baoquan Zhao et al. (+4 more)
Recent representation
alignment (REPA) methods accelerate
diffusion transformer training by aligning projections of the
transformer's hidden states with representations from pretrained visual encoders. In this work, we explore a reverse and complementary direction to REPA: rather than projecting
diffusion representations into the encoder's space, we inject encoder representations into the
diffusion transformer, allowing them to actively participate in the denoising process. To this end, we introduce REPresentation Injection (REPI), a training framework based on a scaffold-to-internalization strategy, in which projected encoder representations initially serve as a temporary scaffold and are then progressively internalized by the
diffusion transformer. REPI outperforms REPA across a wide range of backbones and is highly complementary to it: combining the two yields substantial gains over either alone. Notably, with only 160K training steps, REPI + REPA matches vanilla SiT trained for 7M steps, a speedup of over $43.5\times$. Code will be available at https://jeneveuxpas.github.io/REPI
Read the full abstract ▾
All authors Han Fu, Jiacheng Chen, Baoquan Zhao, Weidong Chen, Wei Liu, Qing Li, Xudong Mao
Computer Vision · Cross-listed from cs.LG
Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment
Shunchang Liu, Lukas Fluri, Xin Chen et al. (+1 more)
Modern AI models are aligned through post-training to adapt them to downstream tasks. Recent work shows that fine-tuning
language models on narrow tasks can induce emergent misalignment (EM), causing broadly harmful behaviors beyond the training task. However, EM has been studied almost entirely in text-only tasks, leaving its manifestation in
multimodal models unclear. In this paper, we define and analyze EM in the context of
vision-
language models. We first induce EM via fine-tuning on narrow
multimodal tasks targeting vulnerable code, careless household-object use, and conspiratorial interpretations of ordinary scenes. Across fifteen commercial and open-source models with different scales, we find that narrow
multimodal fine-tuning can induce coherent and broadly misaligned behavior that transfers to unrelated tasks, including misaligned opinions, visual factual dishonesty, unsafe image generation, vulnerability to visual jailbreaks, and risky agentic actions. We further find that
multimodal EM does not depend on the apparent harmfulness of training data but is sensitive to training-evaluation modality
alignment. EM can arise under both supervised fine-tuning and preference optimization and can propagate through intermediate
reasoning. Finally, we explore several mitigation strategies, including
prompt inoculation, benign continued training, and activation-level steering, which can partially reduce EM. Overall, our findings suggest that
multimodal EM reflects a behavioral shift rather than a general loss of capability, extending beyond text to the visual modality.
Read the full abstract ▾
All authors Shunchang Liu, Lukas Fluri, Xin Chen, Francesco Croce
Computer Vision
Domain-adaptive Zero-Shot Image Enhancement via Locality-Constrained Diffusion Guidance
Theresa Neubauer, Dimitrios Lenis, Astrid Berg et al. (+7 more)
Denoising
Diffusion Probabilistic Models have shown remarkable performance in unconditional image generation. In order to generate images with desired semantics, recent works have restricted the solution space by using guidance constraints in the
diffusion sampling process. However, for image enhancement across different domains, these methods struggle to balance two main requirements: looking realistic in the target domain (photorealistic images) and preserving relevant features of the source domain, e.g., low-quality renderings or art paintings. Here, small local changes can alter the fidelity of the image completely, while large changes in other regions might be insignificant. We introduce LocDiff, a locality-constrained guidance method for image enhancement, which serves as a zero-shot extension to pre-trained
diffusion models, ensuring the preservation of critical features during domain adaptation. In this way, we retain important local features, while allowing less critical regions to remain unconstrained and not interfere with the guidance process for relevant regions. We evaluate our method on two different domain-shift tasks: For art-to-photo translation, we apply the method in a fully zero-shot setting, preserving facial identity from paintings while generating photorealistic details. For enhancing low-quality fetal ultrasound renderings, we demonstrate zero-shot inference with auxiliary prior
alignment. Here, the objective is to artificially add high-resolution characteristics and produce photorealistic ultrasound renderings, a target domain for which no ground truth distribution exists. Our experimental results demonstrate that LocDiff achieves favorable realism-faithfulness trade-offs compared to state-of-the-art methods, enabling controllable cross-domain enhancement.
Read the full abstract ▾
All authors Theresa Neubauer, Dimitrios Lenis, Astrid Berg, Maria Wimmer, Gaia Romana De Paolis, Philip Matthias Winter, David Major, Johannes Novotny, Ariharasudhan Muthusami, Katja Bühler
Computer Vision · Cross-listed from cs.LG
$\lambda$-JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning
Berker Demirel, Clémentine Dominé, Valentino Maiorca et al. (+3 more)
Joint-
embedding self-supervised learning typically combines an invariance objective across augmented views with additional mechanisms to prevent representational collapse. These objectives are often applied after a projection head, while downstream tasks use the backbone representation before the projector. We find that this mismatch does not necessarily prevent dimensional collapse in the backbone, which can retain low effective rank and potentially limit downstream transfer. To address this, we introduce SACReg, a spectral anti-collapse regularizer motivated by an analysis of $\lambda$-balance, which captures the relative scale of weight
matrices across layers. In a two-layer linear network, we show that (i) $\lambda$-balance prevents collapse, and (ii) our regularizer applied to the backbone induces $\lambda$-balance. In the nonlinear case, this regularizer leads to anti-collapse as well and, in realistic architectures on ImageNet100, it empirically increases the representations' ranks. We apply SACReg to JEPA and propose $\lambda$-JEPA, which improves over LeJEPA and VISReg on ImageNet-1k classification and in average linear-probe transfer performance across eight downstream image
datasets. On
video self-supervised learning, $\lambda$-JEPA improves over LeVJEPA and V-JEPA 2 on the Something-Something-v2 and Kinetics-400
benchmarks. Code is available at https://github.com/berkerdemirel/lambda-jepa.
Read the full abstract ▾
All authors Berker Demirel, Clémentine Dominé, Valentino Maiorca, Marco Fumero, Marco Mondelli, Francesco Locatello
Computer Vision
Evaluating Hierarchy-Aware Deep Learning for the Recognition of Tironian Notes
Yule Kang, Thomas Gorges, Janne van der Loop et al. (+4 more)
Tironian notes are generally regarded as the first Latin shorthand system and are notable for their large, fine-grained symbol inventory. Their high visual similarity and large class set make manual reading time-consuming, leaving manuscripts that contain Tironian notes inaccessible to many researchers. Automatic recognition is also challenging because models must distinguish subtle differences in stroke shape and sign structure while realistic training data remain scarce. However, standard flat classifiers do not explicitly use visual or structural relations between related signs. This paper investigates whether structural relationships between Tironian notes can support automatic recognition. We use the Supertextus Notarum Tironianarum (SNT) by Martin Hellmann, which provides idealized sign forms and a hierarchical organization of Tironian notes. We compare flat ResNet18, ConvNeXt, Shifted Window
Transformer (Swin), and
Vision Transformer (ViT) classifiers with Hierarchical Deep Convolutional Neural Network (HD-CNN)-style coarse-to-fine models and hierarchy-aware routing models based on visual class cleaning and similarity-based re-clustering. The models are evaluated on handwritten samples and manuscript-domain samples from Vergilius Turonensis, both with and without limited few-shot adaptation to the manuscript domain. The results show that the relative performance of flat and hierarchical models depends on adaptation. On Vergilius Turonensis, HD-CNN achieves the best non-adapted Top-1 result with 45.43%, while flat classification reaches the best Top-1 result after few-shot adaptation with 82.09%. Overall, the results indicate that hierarchical structure can support Tironian note recognition, especially under non-adapted conditions.
Read the full abstract ▾
All authors Yule Kang, Thomas Gorges, Janne van der Loop, Franziska Marske, Nikolaus Weichselbaumer, Tino Licht, Vincent Christlein
Computer Vision · Cross-listed from cs.LG
eval-unlearn: Benchmarking unlearning in Text-to-Image Diffusion Models
Mansi, Nikhil Raghavan, Zixia Huang et al. (+4 more)
The rising number of concept unlearning techniques for text-to-image (T2I)
diffusion models has produced a fragmented evaluation landscape. Methods are assessed under heterogeneous experimental conditions making principled cross-method comparison difficult. We present eval-unlearn, an open-source Python library providing a unified, reproducible benchmarking framework for concept unlearning in T2I
Diffusion models. eval-unlearn integrates twelve published unlearning techniques spanning fine-tuning, closed-form model editing, and inference-time intervention, alongside nine complementary evaluation metrics covering erasure efficacy,
adversarial robustness, generative quality, and concept retention. Its plugin architecture lets third-party techniques and metrics self-register without modifying the core framework, and its streaming, batched pipeline supports efficient evaluation of both standard NSFW concepts and arbitrary general concepts. As a further contribution, we release a public leaderboard on HuggingFace along with an interactive tool for real-time evaluation of unlearning techniques. The leaderboard compares nudity concept erasure case study across all twelve techniques, exposing significant accuracy-quality trade-offs that are obscured by heterogeneous evaluation. eval-unlearn is released under the MIT license; the package, code, leaderboard, and documentation are all available at https://eval-unlearn.readthedocs.io.
Read the full abstract ▾
All authors Mansi, Nikhil Raghavan, Zixia Huang, Kai Sheng Ong, Ji Shen Lim, Brandon Siao Xiang Ling, Francesco Leofante
Computer Vision · Cross-listed from cs.RO
Spatial Grafting: Grounding 3D Features for Flow-Matching Robot Policies
Dingsheng Liu, Yangzheng Wu, Mahboubeh Asadi et al. (+5 more)
Pretrained
robot manipulation policies such as
vision-
language-action models (VLAs) or world-action models (WAMs) leave interaction-relevant metric geometry implicit. Recent breakthroughs in spatial reconstruction can supply the necessary geometry reliably, but their features describe local shape without stating where it lies with respect to the
robot. How best to deliver these features to a pretrained policy remains unresolved. We propose Spatial Grafting, a versatile, lightweight spatial module that binds frozen reconstruction features to metric,
robot-relative geometry. Spatial Grafting constructs metric-grounded spatial
tokens and injects them into the flow-matching action expert through cross-
attention, without modifying the host's perceptual pathway, so the host retains the full benefit of its pretraining. We evaluate it more broadly than any geometry-aware policy we compare against: one graft architecture, with no per-host redesign, on two VLAs and two WAMs, across four simulation
benchmarks that span short-horizon manipulation, visual robustness, clutter and long-horizon mobile manipulation, and on three real-
robot platforms with single- and dual-arm configurations. On RoboTwin 2.0, a dual-arm manipulation
benchmark, the graft improves every host across VLAs and WAMs. Grafted $\pi_{0.5}$ gains 11.3% and 15.6% on clean and randomized scenes, reaching 94.0% and 92.4%, above the strongest published 3D-conditioned policy, WAM4D (93.8% and 89.9%). The margin widens as the horizon lengthens: on tasks from BEHAVIOR-1K, a dual-arm mobile manipulation challenge scored by average task progress, it surpasses the 2025 challenge winner on five of six tasks,by up to 0.47 Q-score, and exceeds a map-conditioned spatial policy on average across the three tasks both report.
Read the full abstract ▾
All authors Dingsheng Liu, Yangzheng Wu, Mahboubeh Asadi, Zhiyuan Li, Jinbang Huang, Yixin Xiao, Tongtong Cao, Yingxue Zhang
Computer Vision
AnswerMap: Faithful Spatial Interpretability of VLMs from Answer Posteriors
Mohamed Eltahir, Fardows Adam, Duaa M. Tahir et al. (+6 more)
When a VLM answers a visual query, current interpretability tools rely on text rationales, which use a mismatched modality, or on internal read-outs, which originate too early to reflect the final output and require white-box access to the model. We introduce AnswerMap, a training-free, task-agnostic, black-box visual rationale constructed from the output head. The image is cut into K row and K column bands, each shown alone to the frozen model along with the query in the format of a yes/no relevance question. The outer product of the row and column “yes” posteriors gives the query-conditioned spatial map. Crucially, by defining a fixed read-out R (e.g., expectation, maximum) on top of AnswerMap, we can derive continuous outputs like location natively. This bypasses the reliance on discrete text
tokens for continuous-output tasks and guarantees an image-dependent answer by construction. However, a rationale can be confabulated, so we validate AnswerMap across four models and three query distributions with two tests: (a) agreement with the model's own generated point and (b) deletion of the map's region. The map lands where the model points (AUC 0.85 against 0.38 for
attention), and deleting its region flips 53% of correct answers (against 19% for
attention's). Beyond establishing faithfulness, we demonstrate the map's task-agnostic utility through three distinct read-outs: its maximum flags hallucinated objects without generation, its expectation localizes correctly when the model's own pointing fails, and its top-mass region, fed back as a crop, fixes half of the model's wrong answers. AnswerMap thus offers a new lens on VLM interpretability and, through its read-outs, a new output interface for visual tasks beyond text
tokens.
Read the full abstract ▾
All authors Mohamed Eltahir, Fardows Adam, Duaa M. Tahir, Lama Alamoudi, Sana Ammar, Atheer A. Alboloshi, Jory Albluey, Tanveer Hussain, Naeemullah Khan
Computer Vision
DrawingsDreamer: A Unified Multi-View Engineering Drawings Generation Model
Shurui Liu, Weide Chen, Changwang Yi et al. (+1 more)
Scalable Vector Graphics (SVG) are essential for modern industrial Computer-Aided Design (CAD). However, existing autoregressive SVG generation models are predominantly tailored for artistic creation and struggle to maintain the rigorous
geometric fidelity and cross-view spatial
alignment required for engineering drawings. To bridge this gap, we introduce DrawingsDreamer, a unified Large
Language Model (LLM)-driven framework for multi-view vector-based engineering drawings generation. By formulating the generation of multi-view engineering drawings purely as a sequence modeling task, we eliminate the need of raster image encoders. We propose a Streamlined Representation utilizing hierarchical postfix tokenization, which guides the model to establish local
geometric coordinates before assigning semantic boundaries. Optimized via a progressive task-aware curriculum schedule, DrawingsDreamer effectively transitions from localized structural repair to macroscopic generation in a unified model. Extensive experiments demonstrate that our unified model achieves strong performance in both
geometric fidelity and syntactic accuracy across diverse conditional and unconditional generation tasks.
Read the full abstract ▾
All authors Shurui Liu, Weide Chen, Changwang Yi, Ancong Wu
Computer Vision
Beyond Selection: Token Parameterization for Extreme Visual Token Compression
Rui Zhong, Yu Li, Zheyu Yan et al. (+1 more)
Visual-
token compression is effective for improving the efficiency of
vision-
language models, but under extreme compression budgets,
token pruning can break visual grounding while learned resamplers increase parameter count,
attention cost, and training complexity. We revisit compression through a
token parameterization lens, separating (i) basis transformation and structured truncation (retained subspace/compressibility) from (ii) coordinate organization (optimization and cross-modal
alignment). This view yields two coupled objectives, compressibility and learnability, which we formalize as unified functionals. Guided by these objectives, we design Braco, a lightweight four-step coder that combines transform-basis truncation, input-independent basis-coordinate
embeddings, budget-dependent orthogonal re-parameterization, and learned spatial residual
tokens from lightweight pooling. Experiments show that Braco forms the favorable empirical accuracy-efficiency frontier under $23\times$–$64\times$ compression and remains competitive at $144\times$, reaching 95.2% accuracy while reducing prefill FLOPs by 84.2%–86.7% relative to the uncompressed upper bound. Against prior methods, Braco matches or improves accuracy while achieving up to approximately 36% end-to-end speedup and using $16.6\times$/$78.8\times$ lower compressor latency/FLOPs.
Read the full abstract ▾
All authors Rui Zhong, Yu Li, Zheyu Yan, Cheng Zhuo