Computer Vision · Cross-listed from cs.LG
MAGNET: Manifold-Aware Graph Diffusion Network for Connectome Generation
Protyay Dey, Ayush Roy, Hyuna Cho et al. (+2 more)
Functional
brain connectome represents neural connectivity as a
matrix of pairwise interactions between
brain regions. Generation of functional connectomes is not only a question of validity; rather, having satisfied the constraints on the correlation
matrix, the next step is to recover the class-conditional geometry buried under coarse labels. We propose MAGNET, a
Manifold-Aware Graph
Diffusion Network which uses a normalized-Cholesky representation of the
manifold of correlation
matrices that guarantees validity. MAGNET lifts noisy latent states into ROI-level region
tokens and performs denoising with a relational inductive bias over
brain atlas regions. To deal with structural problems induced by coarse labels, MAGNET employs class-anchored conditioning, amortized structural bridge, and relevance-preserving corruption. Across ABIDE, ADNI, and OASIS-3, MAGNET consistently achieves favorable results compared to previous
manifold-aware approaches, demonstrating improvements of 7-21% in class-conditional fidelity ($\alpha,\beta$-F1) across three cohorts and better sampling efficiency. Moreover, while training only with strict binary labels, MAGNET is capable of maintaining clinical heterogeneity through fine substructure of the connectomes in a zero-shot setting, improving subclass covariance
alignment ($\lambda$-MSE) by over 30%. These results suggest that
geometric validity is a necessary but insufficient condition for clinical utility in connectome synthesis. Moreover, efforts in making the
diffusion denoising class-conditional
manifold aware finds utility beyond the highly curved
brain connectome generation as this is a critical problem in various general settings. The code repository can be found at https://github.com/protyayofficial/MAGNET.
Read the full abstract ▾
All authors Protyay Dey, Ayush Roy, Hyuna Cho, Won Hwa Kim, Vishnu Suresh Lokhande
Computer Vision
MaskSense: Confronting the Visual Exploration Trap in Masked Image Generation
Yawen Shao, Jie Xiao, Kai Zhu et al. (+6 more)
Reinforcement learning (RL) has shown strong potential for aligning generative models with human intent, but adapting RL to masked generative models (MGMs) remains largely underexplored. Existing approaches that directly adapt Group Relative Policy Optimization (GRPO) to MGMs leave a foundational conflict unaddressed: MGMs' exploitative decoding nature is inherently at odds with the exploratory diversity that GRPO presupposes. In this work, we reveal a structural decoding bias termed the visual exploration trap: confidence-based sampling trades the exploration of diverse subject realizations for the greedy resolution of low-uncertainty background regions, causing premature collapse of the generative exploration space and starving GRPO of the rollout diversity it depends on. To this end, we propose MaskSense, a novel RL framework for MGMs that confronts the visual exploration trap at both the sampling and optimization stages. Specifically, we introduce a semantic anchored routing sampling mechanism that leverages semantic priors to preserve high-
entropy subject
token exploration while amplifying intra-group reward variance through exploratory-exploitative routing to yield more discriminative advantage estimates. Furthermore, we design a global
entropy transition anchoring strategy to identify the most consequential decoding steps and concentrate gradient updates on them. Extensive experiments on multiple text-to-image
benchmarks demonstrate that MaskSense substantially improves upon the base model, with GenEval accuracy improving from 54% to 86% and HPS from 28.89 to 36.39, achieving state-of-the-art performance.
Read the full abstract ▾
All authors Yawen Shao, Jie Xiao, Kai Zhu, Yu Liu, Hongchen Luo, Xueyang Fu, Yang Cao, Wei Zhai, Zheng-Jun Zha
Computer Vision
Self-Supervised Learning of Semantically Consistent Visual Features via a Simple 3D Prior
Sebastian Day and Oisin Mac Aodha
Visual features extracted from
vision models like DINOv2 or Stable
Diffusion enable high quality semantic correspondence between images. However, such features exhibit documented limitations that hinder accurate correspondence estimation such as the inability to disambiguate the left from right for categories exhibiting
symmetries or repeated parts. Previous attempts to improve these features rely on using supervised data such as annotated keypoint pairs or camera pose information, or make restrictive topological assumptions, such as modelling the object of interest as a 3D sphere. We introduce a new self-supervised approach that allows us to learn features for topologically complex object categories using a simple prior. Instead of relying on an expensive instance-level analysis by synthesis pipeline, our approach makes use of contrastive learning and distribution matching at the global
dataset-level to learn the coarse shape and appearance of a category. Combined with additional
geometric losses, this enables us to replicate the global and local statistics of visual features which match those that are extracted from real images. Quantitative results are presented across several challenging object categories, demonstrating the effectiveness of our approach.
Read the full abstract ▾
Computer Vision
AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding
Yuqi Ye, Caigao Jiang, Zhuonan Wang et al. (+12 more)
Vision-
language-action (VLA) models show strong potential for end-to-end autonomous driving by leveraging world knowledge and understanding to excel in long-tail scenarios. However, current VLAs have two limitations: 1) Degeneration into
vision-action (VA) models. For low latency, regression/flow-based methods directly generate actions but underuse VLM understanding, causing unsafe, unexplainable actions. 2) Autoregressive high latency. Autoregressive methods sequentially generate chain-of-thought (CoT) and action
tokens, fully leveraging VLM understanding, but high latency hinders deployment. To enable low-latency deployment while using CoT to assist action generation, we propose AUGUstus, an Autonomous-driving VLA model with Unified Generation and Understanding. We unify
diffusion-based action generation and autoregressive CoT understanding in one model, where understanding only serves generation. Joint CoT supervision preserves VLM spatial understanding in shared representation for safer, more accurate trajectories. AUGUstus-mode orders action generation before CoT, so CoT supervises reliable actions during training while inference needs only action output, enabling low latency. AUGUstus-GRPO, a unified GRPO
algorithm, jointly optimizes action and CoT, improving action performance. Experiments show AUGUstus achieves 94.6 PDMS on NAVSIMv1 and 90.7 EPDMS on NAVSIMv2, SOTA planning without scorer module. On nuScenes, it achieves average L2 error 0.27 m and average collision rate 0.05%, best among compared methods. To validate AUGUstus practicality, we train on in-house data with 512 NVIDIA H800 GPUs for ~2 days, evaluate on an internal 1,000 km
dataset, and preliminarily deploy on a real vehicle, achieving 12 Hz operation. Project page: https://augustus-ad.github.io
Read the full abstract ▾
All authors Yuqi Ye, Caigao Jiang, Zhuonan Wang, Congyuan Pei, Chao Qu, Changhao Peng, Junhong Lin, Qize Yu, Zetian Song, Guangzhao Fu, Chongruo Wu, Shiyu Huang, Shuchang Zhou, Honggou Yang, Wei Gao
Computer Vision
The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors
Shu Zhao and Tan Yu
Visual in-context learning (ICL) enables
vision-
language models (VLMs) to adapt at test time via demonstration examples, but its effectiveness depends critically on the examples selected. Recent counterfactual selection methods improve ICL by constructing demonstrations that expose how individual attribute changes affect the answer. However, they require multi-stage pipelines involving attribute extraction, caption engineering, and composed image matching. We propose a simpler alternative grounded in a key observation: a model's own prediction errors are natural counterfactual demonstrations that directly encode its failure modes. Our method, SMILE (Self-Mined Visual In-Context Learning from Errors), builds a model-specific pool of hard negatives offline by recording the target VLM's own mistakes. At test time, the prediction serves as a direction signal that surfaces, in the semantic
embedding space, the past mistakes most likely to recur for the current query. This prediction-conditioned identification provides a unified mechanism for both closed-set classification and open-ended visual question answering without task-specific design. Across four
benchmarks and four VLMs, SMILE delivers an average task-score gain of +3.09 points over the SOTA baseline at 5x lower latency. Project page: https://tree-shu-zhao.github.io/SMILE/
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
Lei Yao, Qiao Feng, Mutian Tong et al. (+3 more)
Latent Action Models (LAMs) can capture manipulation priors from action-free
videos, but discrete latents usually impose a granularity bottleneck, limiting their ability to represent fine-grained action. Recent continuous alternatives mitigate this, but they are typically used as proxies for world modeling and require a further planning step to produce executable actions. We thus introduce UniLAP, a unified continuous latent-action policy that models the latent space via flow matching and bypasses reliance on future frames during inference. By distilling a LAM teacher with a flow-adaptive objective, UniLAP inherits the teacher's action representation while retaining the
multimodal distribution of feasible behaviors. To decouple latent representations from task-irrelevant appearance, we extract a label-free motion target to isolate relevant kinematic cues from spurious visual dynamics. With the established embodiment-agnostic latent
manifold, UniLAP can be easily adapted to novel platforms. Pretrained on multi-source
videos at a compact 1.1B parameter scale, UniLAP outperforms state-of-the-art latent-action and
Vision-
Language-Action (VLA) baselines across hand pose forecasting, single-arm and humanoid manipulation, and real-world deployment.
Read the full abstract ▾
All authors Lei Yao, Qiao Feng, Mutian Tong, Qihao Yun, Lap-Pui Chau, Lingjie Liu
Computer Vision
Proactive Physical AI through Continuous Observation and Timely Intervention
Zhenhua Yang, Jiafeng Liang, Xingyu Zhou et al. (+26 more)
Physical AI is commonly formulated around an explicitly specified task: given a goal or instruction, a system perceives, reasons, and acts. In many real-world settings, however, important tasks emerge from continuously changing world states before a person explicitly formulates them. We study this complementary problem as Proactive Physical AI, where an AI system continuously observes the physical world, identifies situations that warrant intervention, and initiates appropriate actions at the right time. We develop ProAgent, an end-to-end system built around two components: a Jev-style proactive timing model (dubbed StreamingJev) that determines whether and when intervention is needed, and a Response Model that generates responses for people or invokes available device capabilities for physical execution. To support learning at scale, we develop an automated data pipeline that converts
videos into supervision for proactive timing. Experiments demonstrate strong proactive performance under both explicit objectives and standing roles, combining more accurate intervention timing with fewer false triggers than the source-backbone baseline. Real-
robot evaluations further demonstrate ProAgent's ability to identify emerging needs and translate them into appropriate physical actions.
Read the full abstract ▾
All authors Zhenhua Yang, Jiafeng Liang, Xingyu Zhou, Yi Jiang, Xiayu Cao, Zui Chen, Luqi Wang, Haiyang Sun, Yufei Xiang, Chang Li, Rongchuan Mu, Bowen Chi, Zihe Tong, Mingyu Li, Huijie Ma, Yiqiao Yang, Baoqi Ren, Zhiming Ma, Zhihao Zhu, Jiahao Ji, Zhuoyao Wang, Qianyu Wang, Xiangyu Liu, Wang Zixiang, Shunliang Zhang, Ming Liu, Xiaocheng Feng, Duyu Tang, Bing Qin
Computer Vision
DDVR-Net: Degradation-Aware Decoupling and Visibility-Guided Restoration for Historical Document Recognition
Jiu Sha, Yu Weng and Mengxiao Zhu
Historical document analysis is crucial for cultural heritage preservation and low-resource
language digitization. However, real historical documents often suffer from severe degradation, such as ink erosion and structural damage, causing mixed-visibility character sequences. Conventional OCR systems struggle in such evidence-limited scenarios, while contextual
language models may over-correct visually clear characters. To address these challenges, we propose DDVR-Net, a degradation-aware decoupling and visibility-guided restoration framework for historical document recognition. DDVR-Net decouples intrinsic glyph structures from physical corruption at the feature level and predicts local visibility to estimate character reliability. A contextual decoder is selectively activated through a visibility-guided gate, restoring degraded characters while preserving clear visual recognition. Furthermore, we construct a low-resource multilingual
benchmark covering Tibetan, Mongolian, and Uyghur scripts with 187000 document pairs, combining large-scale synthetic data and progressive training with real-world annotations to reduce the synthetic-to-real gap. Extensive experiments demonstrate that DDVR-Net consistently outperforms existing methods across three
languages and diverse reading orders, providing an effective framework and
benchmark for low-resource historical document recognition. Code and
Benchmark: https://github.com/Shajiu/DDVR-Net
Read the full abstract ▾
Computer Vision
UniMetric: Universal Metric Scale Recovery for Visual Geometry Foundation Models
Lei Yang, Yuner Zhang, Ziyang Liu et al. (+5 more)
Recent advances in
multimodal foundation models, exemplified by GPT-6 Astra, raise the prospect of general-purpose embodied
reasoning and planning. Using Internet
videos for embodied pretraining calls for reliable physical-scale annotations. Yet visual geometry foundation models often recover scene structure and camera motion without reliable metric scale. We show that uncertain size priors of familiar objects can jointly constrain a reconstruction's shared scale. Building on this insight, we present UniMetric, a universal and training-free metric scale recovery framework for geometry foundation models, without additional sensors or task-specific metric supervision. Specifically, Metric Prior Acquisition elicits object height intervals from a
vision-
language model. Semantic-
Geometric Alignment measures reconstructed object heights along the estimated support-surface normal in each frame and converts the priors into scale intervals. Global Scale Recovery reconciles these intervals through a convex objective, yielding a jointly feasible scale when they overlap and a unique compromise otherwise. The estimated sequence-level scale is applied uniformly to reconstructed points and camera translations. Controlled experiments with real VLM priors show that combining observations improves scale accuracy with exact relative object heights. To evaluate metric accuracy on GFM reconstructions without ground-truth scale
alignment, we introduce MetricBench, covering twelve embodied, driving, and roadside subsets with sensor-derived ground truth. Across nine geometry foundation models, UniMetric reduces mean depth Abs Rel, reconstruction Chamfer distance, and trajectory error by 54.2%, 83.8%, and 65.6%, respectively, improving every model on all three tasks at the suite level. We release the UniMetric code and MetricBench data.
Read the full abstract ▾
All authors Lei Yang, Yuner Zhang, Ziyang Liu, Ziying Song, Kai WU, Yiyang Gao, Jiangtao Su, Yiming Li
Computer Vision
FOCUS: A Subspecialty Foundation Model for Intraoperative Central Nervous System Pathology
Xing Liu, Yifan Guo, Hongtai Lv et al. (+18 more)
Intraoperative frozen-section diagnosis of central nervous system (CNS) lesions guides surgical decisions but is challenged by artefacts, diagnostic complexity and time constraints. Most pathology foundation models have limited exposure to CNS frozen tissue. Here we present FOCUS, a subspecialty foundation model pretrained on 27,930 frozen-sections enriched for CNS tissue. FOCUS led 11 internal diagnostic and 12 biomarker-prediction tasks, with mean AUCs of 0.969 and 0.877, respectively, retaining leading performance in temporal and external cohorts. To provide more comprehensive diagnostic information, we further organized ten diagnostic tasks into a WHO-aligned diagnostic hierarchy and separately evaluated FOCUS for lymphoma detection, a critical task for intraoperative surgical decision-making. In reader assessments by six neuropathologists, FOCUS assistance enabled junior readers to approach or exceed unassisted senior readers in hierarchical diagnosis and raised junior readers' lymphoma sensitivity from 81.6% to 94.7%. FOCUS could improve diagnostic accuracy and granularity, inform lymphoma-related resection decisions and provide provisional molecular information. Such capabilities highlight the potential of FOCUS to provide robust, clinically actionable support for intraoperative CNS pathology.
Read the full abstract ▾
All authors Xing Liu, Yifan Guo, Hongtai Lv, Lei Lou, Yuyi Tan, Jiang Du, Xijia Zhang, Ming Zhang, Chao Pang, Minglei Li, Wei Liu, Xiaobo Zhu, Song Zhang, Rong Yu, Hanlin Shu, Wenzhuo Xu, Tao Jiang, Cheng Lu, Jie Tian, Qing Chang, Zhenyu Liu
Computer Vision
ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving
Shengzhen Li, Yongkang Li, Hao Gao et al. (+5 more)
End-to-end driving requires rich spatiotemporal representations to support scene understanding and trajectory planning under strict latency constraints. However, most systems initialize their visual encoders from generic image pretraining and densely process all patch
tokens for each camera view and input frame, limiting both representation quality and inference efficiency. To address these challenges, we present ConDenseDrive, a knowledge-dense and
token-sparse visual encoder for end-to-end autonomous driving. Our three-stage framework first makes the encoder knowledge-dense by distilling complementary visual, semantic, and
geometric knowledge from three
vision foundation models. It then adapts the distilled encoder to multi-frame end-to-end driving and integrates spatiotemporal context into register
tokens for planning. Finally, it makes the encoder
token-sparse by applying register-guided
token compression, reducing redundant patch computation and yielding a 2.0$\times$ speedup while improving planning performance. ConDenseDrive achieves 94.5 PDMS on NAVSIM v1 and 90.8 EPDMS on NAVSIM v2, and reaches 37.4 HD-Score in zero-shot evaluation on HUGSIM. Extensive experiments and qualitative analyses demonstrate that visual encoder design provides an effective route to reliable and efficient end-to-end driving.
Read the full abstract ▾
All authors Shengzhen Li, Yongkang Li, Hao Gao, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang
Computer Vision
Mining Cross-Modal Perception Gaps in Autonomous Driving: When Camera and LiDAR Disagree
Hongsen Cao, Mona Jaber, Shanxin Yuan et al. (+1 more)
Multi-modal fusion for 3D object detection assumes that cameras and
LiDAR are broadly complementary. In practice, how much each modality is trusted at a given location is decided by learned weights that are neither characterized nor auditable. We label every ground-truth object by which modalities detected it, then mine the resulting 0.4M-object database across nuScenes, KITTI, and Waymo. Because the patterns shift with distance from the ego vehicle, we read the mined model through Spatially Stratified Feature Attribution (SSFA), which conditions Shapley attribution on distance zones and exposes structure that global attribution averages away. Three assumptions built into the current fusion design do not hold. The modality-preference boundary tracks the object itself: a traffic cone is nearly six times as likely as a truck to be seen by the camera alone (16.8% vs. 2.9%), regardless of range. Camera-only detections peak at 10-30 m, inside the band usually ceded to
LiDAR, and fall away on either side. The drivers of failure are not additive either: the distance - density interaction alone outweighs the entire main effect of viewing direction. We further find that modality dominance is directionally non-stationary, with objects behind the vehicle being markedly more
LiDAR-dominant than those in front. We compile the mined structure into a Modality Reliability Map and inject it into four pretrained fusion detectors through a lightweight
attention layer. This improves performance by up to 1.3 mean Average Precision (mAP) while fine-tuning under 4% of their parameters in 2-3 GPU-hours. Our work lifts reliability out of the weights and treats it as a quantity that can be examined before deployment. That is what a safety measure for multi-modal perception must rest on.
Read the full abstract ▾
All authors Hongsen Cao, Mona Jaber, Shanxin Yuan, Ahmed Sayed
Computer Vision
Barycentric Guidance: Turning Foundation Image Editors into Continuous Affective Controllers
Harvey Mannering, Zhiwu Huang and Adam Prügel-Bennett
Emotions are expressed on a continuum and are naturally described in the valence-arousal (VA) space. Existing VA-based image editors are typically training-based and domain-specific, limiting them to VA regions seen during training and making them brittle across
datasets and domains. Meanwhile, despite producing high-quality edits, state-of-the-art foundation image editors are not continuous affective controllers: non-linear emotion representations in text encoders makes
prompt interpolation poorly calibrated, so accurate and continuous VA control remains unreliable. To fill this gap, we introduce Barycentric Guidance, a training-free framework that turns pretrained foundation image editors into continuous affective controllers over VA, requiring no finetuning, extra data, or model changes. Our method maps any target VA point to simplex-based guidance weights, enabling VA controllability in three aspects: (i) accurate targeting of VA states, (ii) broad coverage across the VA plane, and (iii) continuous control. Across three face
datasets, we improve VA target accuracy by 25-29%, expand VA-plane coverage by 28-50%, and increase expression diversity by 29-49%, over the strongest baseline. Unlike prior domain-specific methods, our training-free approach exploits the foundation model's inherent versatility, enabling generalization beyond human faces to animals, artworks, and complex scenes within a single framework. Code is available at https://github.com/harveymannering/BarycentricGuidance.
Read the full abstract ▾
Computer Vision
EviStress: Evidence Stress Learning for Generalizable Face Forgery Detection with Textual Explanations
Jiuyao Jing, Xuanyu Yi, Yu Zheng et al. (+1 more)
The rapid spread of face forgeries on the Web challenges the credibility of visual content. In practical Web environments, manipulated media originate from evolving generation pipelines and undergo diverse processing during dissemination, making reliable authenticity assessment under unseen manipulations and visual changes a critical challenge. We observe that a standard LoRA-adapted CLIP detector exhibits uneven patch-level decision dependence, concen trating a disproportionate share of class-specific relevance on a small fraction of visual patches. To improve robustness to variations in such highly dependent evidence, we propose EviStress, a two-stage framework for generalizable face forgery detection with textual explanations. In the first stage, Evidence-Targeted Patch Perturbation (ETPP) estimates patch dependence from global, relational, and semantic cues and assigns positively correlated perturbation probabilities, explicitly stress-testing highly dependent visual evidence during training. We further introduce Discriminative Variation Preservation (DVP), which constrains the residual between clean and perturbed representations to retain authenticity discriminative information. In the second stage, Intrinsic Evidence Instruction Tuning (IEIT) transfers the adapted CLIP encoder into LLaVA as its
vision tower and performs parameter-efficient
multimodal instruction tuning, enabling authenticity prediction and textual explanation directly from the learned forgery-sensitive representation without an external detector. Experiments demonstrate improved cross-
dataset and cross-method generalization and corruption robustness, with the
multimodal model providing authenticity predictions and human-readable explanations.
Read the full abstract ▾
All authors Jiuyao Jing, Xuanyu Yi, Yu Zheng, Chunlei Peng
Computer Vision
From Volpiano to Pixels: 30,000 Lines of OMR Ground Truth for Medieval Chant
Robert Klugseder
Optical music recognition (OMR) for medieval chant lacks training data linking manuscript images to musical symbols across notational families. Using existing Volpiano transcriptions from the Cantus Database, we created 30,567 image-target pairs from ten sources in eleven exports, including 339 verse fragments. Their target sequences contain 1,952,143
tokens. The pipeline locates staff systems, infers clefs and maps known pitches to staff positions. Six detector families handle the different layouts and notations. Where heuristics failed, two small U-Nets learned staff lines and clef letters from manual annotations. Reviews of sampled export lines found zero to four pitch or clef errors, but used different error definitions and do not provide a uniform audit of the corpus. A CRNN trained from scratch achieved a symbol error rate of 9.6 % on 3,086 test lines. The earlier model, trained mainly on Austrian sources of the 15th and 16th centuries, scored 45.3 %. In three separate training runs, each excluding one manuscript, error rates on the excluded manuscript were 21.5-31.0 %. Model-assisted
alignment recovered a further 2,686 lines.
Read the full abstract ▾
Computer Vision
Vision Models Capture Complementary Representations in Children That Converge in Adults
Domenic Bersch, Timothy Schaumlöffel, Michela Proietti et al. (+11 more)
How the relationship between image-text aligned and
vision-only models changes across development remains unknown. We applied time-resolved representational similarity analysis to
EEG data from 8-year-olds, 12-year-olds, and adults. CLIP showed significant model-
brain alignment already in children, whereas
vision-only models aligned more strongly later in development. Residual analyses revealed complementary structure, with CLIP retaining
alignment beyond an architecture-matched
vision-only model from age 8. Much of the model-
brain alignment reflected semantic structure, with
EEG representations of category and animacy strengthening toward adulthood, while
alignment with low-level image properties remained weak across age. Together, these findings suggest that image-text aligned and
vision-only models capture complementary aspects of developing visual representations and may provide useful constraints for AI systems that aim to learn and adapt over time.
Read the full abstract ▾
All authors Domenic Bersch, Timothy Schaumlöffel, Michela Proietti, Antonia Franaszek-Traczewska, Hannah Elisabeth Zwad, Siying Xie, Marlena Baldauf, Teresa Sylvester, Christina Maria Schätz, Bert Turtleton, Moritz Köster, Stefanie Höhl, Radoslaw Cichy, Gemma Roig
Computer Vision
Towards Cost-Efficient Video Dataset Construction with Large Language Models
Mohammed Shabot, Jonathan Adams, Jan Van den Stock et al. (+1 more)
Research on
video classification models has witnessed significant growth recently, accompanied by a significant increase in the demand for
datasets to train such models. Existing
video dataset curation approaches share a common bottleneck: candidate
videos must be downloaded before any relevance judgment is made, which results in storage, time and annotation costs. In this work, we present a filtering step, using Large
Language Models (LLMs) to pre-screen
videos based on their metadata, before actually downloading these
videos, thereby saving on these costs. We evaluate the LLMs against a strict human majority vote derived from four independent annotators and validate the reliability of the annotations through inter-annotator agreement analysis (Gwet's AC1). We compare the LLM pre-filter against two no-LLM baselines, namely two keyword heuristics. Via a Leave-One-Rater-Out (LOO) mechanism and an Add-One-Rater-In (AORI), we evaluate the performance of the LLMs as virtual annotators. Finally, we report cost comparisons relative to existing approaches. To ensure reproducability, we made our implementation available on github.
Read the full abstract ▾
All authors Mohammed Shabot, Jonathan Adams, Jan Van den Stock, Joost Vennekens
Computer Vision
A Multi-Task Deep Learning Framework for Pose-Invariant Face Recognition and Facial Landmark Localization
Ali Ashkeyev and Zukhra Abdiakhmetova
Face recognition systems often experience performance degradation under large pose variations due to changes in facial appearance and visibility of discriminative features. This study presents a multi-task deep learning framework that jointly addresses pose-invariant face recognition and facial landmark localization within a unified architecture. To enable simultaneous optimization of identity recognition and facial structure learning, the original Cross-Pose Labeled Faces in the Wild (CPLFW)
dataset was adapted into a multi-task learning
dataset by associating facial images with identity labels and five-point landmark annotations. The proposed framework consists of two task-specific branches operating on a common facial input: a ResNet50-based branch for identity recognition and an EfficientNetV2B0-based branch with spatial
attention for facial landmark localization. The two objectives are jointly optimized using a weighted multi-task loss, allowing the model to learn complementary semantic and spatial facial representations. The proposed framework was evaluated on the prepared CPLFW-based multi-task
dataset. Unlike the original CPLFW
benchmark, which focuses primarily on identity recognition, the proposed
dataset adaptation incorporates five-point facial landmark annotations to enable joint recognition and localization learning. Experimental results demonstrate that the model achieves a validation recognition accuracy of 81.58% and a facial landmark localization mean squared error of 96.38. The results indicate that combining identity classification and landmark regression within a single optimization framework provides an effective approach for learning robust facial representations under cross-pose conditions The proposed method demonstrates the potential of multi-task learning and
attention-based feature refinement for unconstrained facial analysis.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CR
Beyond User-Side Defenses: Image Protection in Open-Weight Diffusion Models
Jihye Kang, Jiwon Park and Namhyuk Ahn
Model providers can support image protection, but open-weight deployment introduces practical constraints. We propose four requirements: protection effectiveness, clean fidelity, compatibility, and tamper resistance. We incorporate long-horizon attack simulations to strengthen resistance to protection removal. Experiments with a single protection tool show improved resistance to re-
alignment at a cost to clean fidelity, motivating further research toward practical model-side protection.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.SD
Unsupervised Data Drift Monitoring for Deployed Deepfake Speech Detectors
Lukáš Prokeš, Vojtěch Staněk, Anton Firc et al. (+1 more)
Deepfake
speech detectors are typically evaluated as binary classifiers on fixed
benchmarks, but deployed systems face evolving attacks, codecs, speakers, and recording conditions, with labels unavailable during operation. This makes it difficult to know when a detector operates in familiar conditions and can be relied on. We propose an unsupervised drift monitor that operates on the internal
embeddings of a deployed detector. The monitor compares each input against bona fide and spoof references using regularized Mahalanobis distances. These distances define trusted, ambiguous, and novel regions of the detector
embedding space, allowing unfamiliar inputs to be routed for review. We evaluate the monitor on four modern detectors on three
datasets. The monitor raises alerts for
dataset mismatches and substantially reduces the security-critical case of accepting deepfakes at the cost of reduced automatic coverage. The results also show that drift affects both classes: bona fide recordings can move away from the training reference, increasing alert rates even in in-domain settings. Our findings suggest that deployed deepfake
speech detectors should be treated not only as a binary classification, but also as a reliability problem.
Read the full abstract ▾
All authors Lukáš Prokeš, Vojtěch Staněk, Anton Firc, Kamil Malinka
Computer Vision
Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers
Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi et al. (+4 more)
Progressive depth growth builds large networks by starting with shallow models and inserting layers during training. Recent advances in large
language models have renewed interest in this longstanding approach, demonstrating benefits for training efficiency and
reasoning alongside changes in internal computations. However, how growing affects representation quality and computational patterns in self-supervised
vision transformers remains unclear. In this work, we systematically investigate progressive depth growth in two popular self-supervised families, namely masked autoencoders (MAE) and DINOv3, and analyze how growing affects downstream performance, intermediate representations, and internal computations. Across global and dense tasks, we show that growing improves downstream performance while reducing training FLOPs by 13.9–24.8%, with ImageNet linear-probe accuracy gains of up to 3.36 percentage points. Growing also improves downstream performance at intermediate layers and reshapes internal computational patterns, introducing recurring alternations between global and localized
attention in MAE and earlier local-to-global
attention progression in self-distilled DINOv3. Together, these findings establish progressive depth growth as an effective and efficient pretraining strategy for improving downstream performance across global and dense visual tasks with less computation, and advance our understanding of how growing shapes visual computation.
Read the full abstract ▾
All authors Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi, Karl Henrik Johansson, Kaitlin Maile, Johannes von Oswald, Stefan Bauer
Computer Vision
Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models
Lotfi Abdelkrim Mecharbat, Abdelrahman Elsayed, Ibrahim Almakky et al. (+2 more)
Vision foundation models (VFMs) provide transferable visual representations, but their inference cost limits deployment in many settings. Fine-tuning specializes a VFM without shrinking it;
distillation shrinks it, but through expensive teacher-guided training. Structured
pruning offers a more direct route by extracting a compact subnetwork from the VFM. However, as compression gets aggressive, existing methods fail because they treat subnetwork selection and recovery as separate axes. Across layers, capacity must follow task sensitivity rather than spread uniformly. Within a layer, a removal cannot be judged by importance alone; its impact also depends on how well the lost contribution can be recovered from what remains. Selection and recovery must therefore be decided jointly. We introduce Carve VFM, which resolves this interdependence. Given a VFM, a calibration set, and a parameter budget, it (i) searches for task-specific capacity allocations across layers; (ii) guides the search via an adaptive recovery-aware fitness criterion; and (iii) reconstructs removed behavior through iterative closed-form low-rank corrections merged into the surviving weights. The result is a dense encoder produced in minutes on a single GPU without backbone retraining. Across twelve
benchmarks and four models from DINOv3 and MetaCLIP-2, Carve VFM achieves the highest reported mean accuracy in seven of eight backbone-budget configurations among the evaluated dense structured-
pruning methods. Further fine-tuning surpasses the strongest evaluated edge model (88.1% vs. 85.8%), and carved models cut compute and
memory while delivering 1.3-2.0x higher GPU throughput than the uncompressed one. These results show that the compact model you need is inside the VFM; it only has to be carved out.
Read the full abstract ▾
All authors Lotfi Abdelkrim Mecharbat, Abdelrahman Elsayed, Ibrahim Almakky, Martin Takáč, Mohammad Yaqub
Computer Vision
X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning
Yi Wang, Zhenlong Yuan, Yuchan Guo et al. (+9 more)
Chest X-ray (CXR) interpretation often requires integrating visual findings with task-specific evidence rather than relying on direct visual recognition alone. While tool-augmented medical
agents can supply such evidence via external modules, they often lack explicit mechanisms for learning when auxiliary evidence is needed and for preventing inapplicable quantitative evidence from entering diagnostic
reasoning. To address this, we propose X-Ragent, a post-training framework for tool-augmented reinforcement learning that decouples learned auxiliary-evidence seeking from validity-gated quantitative measurement acquisition for evidence-grounded CXR
reasoning. We construct Rad-CoT, a 7.7K-sample corpus combining causally ordered tool-augmented trajectories with report-style diagnostic
reasoning, and use it to initialize Qwen3-VL-8B via agentic supervised fine-tuning (SFT). We then optimize the policy with Group Relative Policy Optimization (GRPO) using an accuracy-gated reward that applies process-level terms for measurement consistency and auxiliary-tool cost only to correct trajectories, preventing plausible process behavior from compensating for an incorrect diagnosis. Across three CXR
benchmarks, X-Ragent achieves the strongest results, including an 8.6-point improvement on the fully held-out ChestAgentBench. By seeking evidence selectively and admitting only valid measurements, X-Ragent supports more faithful and evidence-grounded CXR
reasoning.
Read the full abstract ▾
All authors Yi Wang, Zhenlong Yuan, Yuchan Guo, Maolei Huang, Zijian Song, Dapeng Zhang, Rongbin Tan, Jiyuan Wang, Zhiyuan Wang, Fangfang Lin, Shuhan Song, Huawei Cao
Computer Vision
ViewStruct: View-Grounded Articulation and Connectivity Inference
Wenjing Bian, Ranran Huang and Andreas Geiger
Interacting with unfamiliar objects requires understanding how their parts connect and move. We introduce ViewStruct, a framework for jointly predicting connectivity and articulation in objects and scenes. Given an RGB image with aligned part segmentation and depth, optionally supplemented by images or
video, ViewStruct predicts rigid-body groupings, attachments, and joint parameters. Its view-grounded representation links predictions to observed regions and expresses joint geometry in a
gravity-aligned, view-relative frame, without requiring category-specific canonical orientations. A pretrained
vision–
language model is adapted to predict structure in seconds, without requiring observed articulation motion or per-instance optimisation. Experiments demonstrate improved structural prediction over evaluated baselines and transfer to unseen categories, asset sources, and real-world human-interaction
videos, despite task-specific training exclusively on synthetic data. Simulated manipulation experiments further demonstrate the utility of predicted joints for generating
robot motion trajectories.
Read the full abstract ▾
Computer Vision
InfoVis: Unified Slide Image Generation and Editing
Jiahao Wang, Xiyan Jiang, Chenyuan Wu et al. (+7 more)
Generating and editing presentation slides requires models to combine accurate text rendering, instruction following, visual grounding, aesthetic design, and precise editing within one canvas. Existing work covers only subsets of these capabilities and lacks large-scale slide-native supervision and diagnostic evaluation. We formulate four slide-centric tasks and build a scalable data engine with over 13M continual-pretraining records and 1.9M supervised fine-tuning records. We also introduce VisSlideBench, a
benchmark of 1,527 manually screened instances that covers all four tasks with task-specific, multidimensional evaluation. Its automatic rankings agree with blinded, dimension-wise human rankings. Using these resources, we train InfoVis, a unified 9B slide generation and editing model. On VisSlideBench, InfoVis leads every evaluated open-source model on all eleven metrics and scores 7.94 overall against 7.99 for Nano-Banana-2. Starting from FLUX.2-klein-base-9B, slide-specific post-training adds 1.6 points overall, with the largest generation gains in content faithfulness; the remaining gap to proprietary models is smallest in content faithfulness and largest in aesthetic quality. Built on InfoVis, our document-to-slide workflow achieves the second-highest Aesthetics score on the external SlidesGen-Bench. We will release the model, code, data, and
benchmark.
Read the full abstract ▾
All authors Jiahao Wang, Xiyan Jiang, Chenyuan Wu, Haoqiang Zhang, Xin Luo, Yueze Wang, Defu Lian, Bo Zhang, Jiajun Zhang, Zheng Liu
Computer Vision
Towards Complementary Keypoint Detection via Mixture of Detectors
Xiaoyong Lu, Guobao Xiao, Dong Liang et al. (+1 more)
Keypoint detection plays a central role in
geometric vision pipelines. Existing detectors exhibit complementary strengths across different scene structures and keypoint budgets, and reliance on a single detector often results in suboptimal keypoint distributions. Motivated by this observation, we propose Mixture of Detectors (MoD) for dense matching pipelines, a unified framework for complementary keypoint detection through the adaptive composition of heterogeneous detectors. MoD first distills heterogeneous detectors into a unified model, allowing diverse detection maps to be predicted efficiently within a single forward pass. Detector selection is subsequently formulated as a patch-wise, budget-conditioned decision process, in which a lightweight router assigns the most suitable detector to each local region. To avoid heuristic supervision, the router is trained with task-driven rewards derived from downstream matching quality, jointly capturing both keypoint quantity and precision. Extensive experiments on MegaDepth, ScanNet, and HPatches demonstrate that MoD consistently outperforms strong baselines across a wide range of keypoint budgets, demonstrating its effectiveness in improving keypoint distribution and downstream
geometric performance.
Read the full abstract ▾
All authors Xiaoyong Lu, Guobao Xiao, Dong Liang, Songlin Du
Computer Vision
Image Tokenizers Should Be Trained Between Images
Ole Beißwenger, Jan-Niklas Dihlmann and Hendrik Lensch
Latent
diffusion models generate images as codes of an image tokenizer, whose decoder turns them into the pixel image. Recent tokenizers improve how well these codes reconstruct an image and how easily a generator learns them. Yet a generator does not reproduce the codes of training images, and when a generated code lands outside what the decoder expects, the decoded image contains imperfections. Current methods make the decoder more robust to generated codes by adding noise during training. However, generated codes lie between the training codes and resemble their mixture more than one of them plus noise. To improve generated images, decoders should therefore be trained on codes in the space between training samples. We propose InterpTok, which fine-tunes MAETok to decode mixtures of image codes into natural images. Since no target image exists between two training samples, a discriminator judges whether each decoded mixture looks as natural as its two sources. A mixing ratio loss requires the decoded mixture to keep the mixing ratio of the two codes. Together, they prevent collapse to a trivial solution and yield a sharp and natural composition of both images. InterpTok keeps MAETok's reconstruction quality and improves interpolation quality more than threefold, the best of all tokenizers in a recent
benchmark. Beyond interpolation, InterpTok improves the generation FID of a matched
diffusion transformer from 11.61 to 8.48 without guidance, and even reaches 8.11 at an eighth of the sampling steps, where the generator on MAETok's codes worsens. The generator trained on our tokenizer also reaches the final result of the one trained on MAETok at half the training steps. Our findings show that the space between image codes can and should be explicitly supervised rather than ignored.
Read the full abstract ▾
Computer Vision
The Midas Touch for Metric Depth: Interpretable Metricization for Generalizable Depth Perception
Mengtan Zhang, Yu Ma, Shiqi Jiang et al. (+6 more)
Recent foundation models have drastically improved the generalization of monocular relative depth estimation, yet recovering accurate metric depth from their estimates for depth perception tasks remains challenging due to scale ambiguity and local metric-scale inconsistency. Existing approaches recover metric depth from sparse observations using either explicit global affine transformation
algorithms or implicit data-driven depth completion networks. The former cannot accommodate spatially varying relative-to-metric relationships, while the latter are sensitive to the distribution of sparse observations and may not fully exploit generalizable relative depth priors. To simultaneously address these issues, this study extends our previous work MTD and proposes MTDv2, a hybrid metric depth perception framework that combines explicit depth metricization with data-driven metric depth refinement. The explicit depth metricization process performs segment-wise affine transformations to recover spatially varying metric scales while preserving the
geometric structure of relative depth, followed by pixel-wise depth refinement that propagates metric observations along paths with minimal accumulated
geometric variation. The data-driven depth refinement process subsequently extends the explicit single-path depth propagation to a learnable multi-path formulation while preserving its interpretability. Extensive experiments conducted on 16 public
datasets demonstrate the robust and generalizable depth scale recovery capability of MTDv2 across diverse sources of sparse observations and different depth estimation paradigms, as well as its broad applicability to downstream 3D perception tasks. Code will be publicly available at https://mias.group/MTD upon publication.
Read the full abstract ▾
All authors Mengtan Zhang, Yu Ma, Shiqi Jiang, Yi Feng, Wei Ye, Xieyuanli Chen, Hanli Wang, Qijun Chen, Rui Fan
Computer Vision
FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Models
Zihan Meng, Weidong Chen, Dexiang Hong et al. (+2 more)
Video large
language models (
Video LLMs) encode each frame into hundreds of visual
tokens, making
video inference expensive. Existing methods reduce this cost through similarity-based merging or
attention-based selection but face two challenges: Hierarchy-blind
Pruning, which overlooks regional compressibility and can discard critical evidence, and
Attention Drift, where positional effects and
attention sinks shift text-to-
vision attention away from task relevance. We propose FoveaTree, a two-stage, training-free visual-
token pruning framework. First, Merge-Gain Density Estimation measures the feature distortion caused by regional merging and constructs a quadtree-derived density map to guide
token dropping, merging, and retention. Second,
Attention Tree
Pruning uses the surviving units and inherited tree
topology to refine text-to-
vision attention in intermediate LLM layers, preserving relevant regional evidence while removing low-relevance content. We evaluate FoveaTree on InternVL3.5-8B and Qwen3-VL-8B across five
video question-answering
benchmarks and one captioning
benchmark at 50%, 30%, and 10% visual-
token retention. FoveaTree consistently achieves overall state-of-the-art performance across the evaluated settings, offering a strong accuracy-efficiency trade-off.
Read the full abstract ▾
All authors Zihan Meng, Weidong Chen, Dexiang Hong, Xinyan Liu, Zhendong Mao
Computer Vision
VAttnPO: Visual Attention-Guided Policy Optimization
Lihao Zheng, Zhenwei Shao, Zhou Yu et al. (+5 more)
Multimodal reinforcement learning with verifiable rewards commonly assigns one sequence advantage to an entire response. This outcome-level signal distinguishes overall response quality but leaves changes in visual-evidence use across generated
tokens implicit. We introduce VAttnPO (Visual
Attention-Guided Policy Optimization), which uses answer-conditioned visual
attention to refine
token-level credit while preserving the verifier-derived trajectory-mean advantage. A frozen old policy constructs a visual reference by teacher forcing the known correct answer and produces a visual distribution for each rollout
token. VAttnPO measures their agreement in visual-
attention space using Jensen–Shannon divergence, rather than reading guidance from teacher output-
token distributions or introducing an
attention-matching objective. Trajectory centering, relative-position weighting, and credit balancing turn this agreement into a zero-mean, position-aware correction to the GRPO sequence advantage.
Token intervention, visual-index permutation, and trajectory analysis provide diagnostic evidence linking this signal to model behavior, visual correspondence, and generation position. On Qwen3-VL-8B-Instruct, VAttnPO raises six-task image AVE from 70.97 with GRPO to 73.50 and scores higher on every reported structured,
video, and MME metric. Gains over GRPO also hold on Qwen2.5-VL-7B-Instruct and InternVL3-8B-Instruct.
Read the full abstract ▾
All authors Lihao Zheng, Zhenwei Shao, Zhou Yu, Yan Yang, Haoyi Sun, Qian Wang, Ning Mao, Pan Zhou