Computer Vision
Texture Space Material Diffusion
Jacob Munkberg, Peter Kocsis and Jon Hasselgren
We present a method for generating high quality materials for 3D objects entirely in texture space. We finetune a
video diffusion transformer for text-guided material generation, multi-view material generation, and material upscaling. Our key insight is to use the known projection from image space to texture space, enabling the
diffusion process to generalize across arbitrary geometries and texture parameterizations. This approach also avoids the view consistency issues inherent in
video and multi-view
diffusion models. Because texture space is two dimensional, we can reuse the strong priors of pretrained
video diffusion models. We apply our method to high quality material reconstruction from posed photos captured under unknown lighting, as well as to text- and image guided material generation. Our method can scale to high resolutions (8K), 100+ input views, and neural material representations. In quantitative and qualitative evaluations we show state-of-the-art results for material generation and reconstruction.
Read the full abstract ▾
Computer Vision
VoxelSage: Tool-Augmented 3D CT Analysis and Simulator-Shielded Sequential Resection Planning for Liver Tumors
Binghong Qian, Xuanhe Liu, Yifan Xing et al. (+3 more)
Preoperative liver-tumor assessment requires segmentation, physical-space measurement, visual evidence, and resection planning from the same three-dimensional CT volume. Existing tools often handle these steps separately, while
language models cannot reliably compute physical measurements from CT. To provide an integrated workflow, we present VoxelSage, a multi-modal system for two- and three-dimensional visualization, liver-tumor analysis, and preoperative resection planning. Its dual-port architecture separates
language-model orchestration from image computation: Port A interprets requests and selects skills, while Port B applies them to CT volumes and segmentation masks and returns structured results. Keeping physical measurements in Port B prevents the LLM from computing them directly and reduces the risk of fabricated numerical results. Eight built-in skills support quantitative analysis, visual evidence generation, three-dimensional reconstruction, segmentation refinement, and sequential resection planning; user-defined skills can extend these functions. For sequence planning, a behavior-cloned neural ranker orders candidate resection targets, while a simulator-based shield checks them against predefined constraints. Across 256 unseen simulator scenes, this approach reduced mean simulated time from 34.274 to 33.388 min (0.886 min, 2.59%) and mean simulated blood loss from 300.847 to 183.852 mL (116.995 mL, 38.89%) relative to a deterministic baseline. These results demonstrate system integration and simulator-level performance, not clinical efficacy or safety. The public implementation is available at https://github.com/ZJUMAI/VoxelSage.
Read the full abstract ▾
All authors Binghong Qian, Xuanhe Liu, Yifan Xing, Wenjie Deng, Jian Wu, Haochao Ying
Computer Vision
Targeted Visual Counterfactual Explanations for Contrastive Vision-Language Model
Van Bach Nguyen, Jörg Schlötterer and Christin Seifer
Current explanation methods for contrastive
vision-
language models such as CLIP mainly identify important regions without showing how to change the input in order to get a target prediction. We introduce Mask-guided Adaptive Counterfactual Explanations (MACE), a targeted visual counterfactual method designed specifically for CLIP zero-shot classification. MACE constructs an editable region from either source attribution or source-target attribution differences and expands the mask only when needed to reach a specified target class. A latent
diffusion inpainting model then modifies the selected region, while a frozen CLIP model provides modification guidance and anchors the remaining image content to the original input. We evaluate MACE on ImageNet, Food-101, Oxford Pets, and CUB-200. The source-mask variant achieves the highest target top-1 success rate across all four
datasets, while the difference-mask variant produces the smallest pixel level and perceptual changes and the best realism scores. Both variants improve proximity and realism over a Stable
Diffusion-only baseline using the same generative backbone. These results show that adaptive mask-guided editing produces effective CLIP counterfactuals. They further reveal a tradeoff between counterfactual validity and source-image preservation.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Procedural Core: A Compact Recurrent Initialization for Vision Transformers
Zachary Shinnick, Christian Internò, Hemanth Saratchandran et al. (+2 more)
Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of abstract procedurally generated data can help acquire generic inductive structure at low cost. However, this adds a pretraining stage that must be repeated for every target model. We propose Procedural Core, an initialization strategy that captures this generic structure into a compact set of weights that can be reused across models. We train a minimal recurrent
transformer on procedural data, then expand its weights to initialize
transformers of arbitrary width and depth. The resulting initialization improves performance on image classification, self-supervised visual learning (DINO), and modeling natural
language (FineWeb-Edu) and code (CodeParrot). For image classification, expanding a 1M-parameter core to initialize an 85M-parameter ViT-Base improves ImageNet top-1 accuracy by 2.2 pp over standard random initialization. Our analysis identifies recurrence as essential for learning compact weights that transfer across models. In ViTs, we localize a key benefit in the suppression of high-norm
tokens that produces substantial improvements in zero-shot segmentation (ImageNet-S mAP 32.3 to 42.9), object localization (VOC07 CorLoc 9.9 to 18.4), and depth estimation (NYUv2 RMSE 1.104 to 0.998). This demonstrates that
transformers need not start from a blank slate, and can be initialized with generic capabilities at low cost with no domain- or task-specific data.
Read the full abstract ▾
All authors Zachary Shinnick, Christian Internò, Hemanth Saratchandran, Anton van den Hengel, Damien Teney
Computer Vision
TomoTransformer: Towards a Foundation Model for CT Reconstruction
AmirEhsan Khorashadizadeh and Benjamín Béjar
Supervised deep learning has advanced sparse-view tomographic reconstruction. However, conventional models, which typically map filtered back-projection (FBP) images or sinograms to clean reconstructions, are brittle under distribution shifts. Because they require retraining whenever projection counts and angles, detector resolutions, or data distributions change, their deployment in real-world applications remains limited. To address this, we introduce TomoTransformer, a
transformer-based architecture that treats each local filtered projection as an individual
token and predicts missing views via self-
attention. Crucially, TomoTransformer operates in a back-projection space that separates projections across spatial locations, making view interpolation geometrically well-posed and invariant to detector size. This design yields a single foundation model that can process any number of input projections, at arbitrary angular locations and detector dimensions, and query any number of target angles without retraining. Trained on a large-scale
dataset spanning diverse medical CT anatomies and natural images, TomoTransformer generalizes effectively across anatomies, materials, and resolutions. Extensive evaluations on several
benchmark sparse-view
datasets show that TomoTransformer significantly outperforms concurrent multi-purpose models like ViewTrans and matches or exceeds strong protocol-specific baselines, while remaining fully agnostic to the number of input and target projections. Furthermore, the model demonstrates robust zero-shot generalization on real experimental nanoscale
brain data collected from an X-ray synchrotron, showcasing its practical utility for real-world applications.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
When to Adapt: Multi-Signal Domain Shift Detection for Efficient Training-Free Adaptation in Open-Vocabulary Segmentation
Michele Antonazzi, Alejandra C. Hernandez, José Araujo et al. (+2 more)
Robust and reliable perception is essential for autonomous
robots operating in real-world environments, particularly in long-term missions where environmental conditions may change significantly over time. Although recent advances in Visual Foundation Models (VFMs) have improved open-vocabulary semantic segmentation, these models can still suffer from domain shift, which can significantly degrade performance if they are not adapted to the current environment. Training-free domain adaptation is a relevant paradigm for adaptation, consisting of adjusting the model online using lightweight adapters. Recent approaches apply this on a per-frame basis, which is impractical for deployments on resource-constrained robotic hardware. To tackle this, we propose a multi-signal domain shift detection method for training-free continual test-time adaptation (TF-CTTA) in open-vocabulary segmentation. Our method leverages temporal coherence across consecutive frames by monitoring and combining complementary aspects of domain shift (visual change, adapter mismatch, and semantic drift) to trigger adaptation only when needed. We validate our approach on a
benchmark including indoor and outdoor environments and using real robotic data. We demonstrate that our approach maintains segmentation accuracy while substantially reducing adaptations, making training-free adaptation practical and feasible for long-term, real-world robotic deployments.
Read the full abstract ▾
All authors Michele Antonazzi, Alejandra C. Hernandez, José Araujo, Olov Andersson, Patric Jensfelt
Computer Vision
FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learning
Xinyuan Zhao
Federated knowledge must remain usable by recipients with different modalities, private architectures, and tasks. We present FedSocket, which makes recipient execution a design requirement of the exchanged model. A shared Q combines recipient-computable inputs, task-owned outputs, and ownership-aware aggregation, connecting heterogeneous private models through a common prediction interface. Private models teach local Q copies; the returned Q supports local learning and Joint inference, with only Q parameters and counts exchanged. Across six
datasets, FedSocket improves missing-modality recipient accuracy over Local by 14.44 and 15.51 percentage points on MELD and UCF-51. Under matched inference capacity, Joint exceeds independent ensembles by 11.06 points in UCF-51 accuracy and 4.87 points in mean bidirectional Flickr30k R@1. Joint also improves over Q alone on all four heterogeneous endpoints, demonstrating the value of combining local and exchanged predictions. Teacher controls, sharing-path interventions, and component factorials identify the roles of supervision, sharing, and deployment. FedSocket makes exchanged knowledge directly usable from
federated training to recipient inference.
Read the full abstract ▾
Computer Vision
TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning
Jing Wang, Zhiping Wu, Dongdong Ren et al. (+3 more)
Vision-
Language Models (VLMs) excel at visual understanding and
reasoning but often incur substantial inference costs due to the large number of visual
tokens. Recent visual
token pruning methods increasingly follow a two-stage paradigm: they first remove visually redundant
tokens after the
vision encoder and then discard
tokens irrelevant to the textual query within the Large
Language Model (LLM). However, since the first stage typically relies solely on
vision-encoder saliency, it may prematurely eliminate query-relevant
tokens, depriving the subsequent text-guided stage of critical visual evidence. Our empirical analysis shows that incorporating query guidance into first-stage
pruning better preserves task-relevant evidence and consistently improves performance over
vision-only saliency-based
pruning. We further find that high-variance
attention heads are more sensitive to the textual query and yield more discriminative text-to-
vision attention signals for second-stage
pruning. Motivated by these findings, we propose TReVS, a training-free framework that combines textual relevance with
vision-encoder saliency for pre-LLM
pruning and leverages high-variance
attention heads to remove task-irrelevant
tokens at shallow-to-intermediate layers of the LLM. On LLaVA-1.5-7B, TReVS retains 92.8% of the unpruned baseline performance while
pruning 94.4% of visual
tokens, outperforming prior state-of-the-art methods.
Read the full abstract ▾
All authors Jing Wang, Zhiping Wu, Dongdong Ren, Youfang Han, Wei Zhao, Wenbin Li
Computer Vision
Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessment
Yu Zhao, Jiarui Wang, Huiyu Duan et al. (+5 more)
With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture fine-grained
alignment and generative artifacts. While large
multimodal models (LMMs) are increasingly adopted as evaluators, existing
benchmarks typically study semantic understanding, quality perception, and authenticity identification in isolation, while largely neglecting responsibility detection. This leaves a gap in unified and comprehensive validation. To bridge this gap, we introduce SQUARE-Bench, a comprehensive
benchmark that systematically evaluates LMM capabilities as evaluators of AI-generated images across four aspects: Semantics, Quality, Authenticity, and Responsibility. SQUARE-Bench introduces a granular taxonomy of 38 sub-dimensions to evaluate nearly 10K AI-generated images sampled from 22 diverse models, ranging from legacy to state-of-the-art generators, complemented by over 3K real-world images. The images are annotated with curated question-answering pairs. Extensive experiments on 23 LMMs reveal that top proprietary models, such as Gemini-3-Pro, already outperform the individual human expert baseline. However, the performance gap between models remains significant, exhibiting notable disparities in fine-grained inference and domain-specific robustness. Beyond benchmarking, we conduct a proof-of-concept study of LMM-guided iterative editing, in which dimension-specific LMMs provide diagnostic feedback to fixed image editors. The resulting guided system yields selective improvements in semantics, authenticity, and responsibility, while exhibiting a consistent visual-quality trade-off. SQUARE-Bench can serve as both a diagnostic tool for characterizing LMM evaluator capabilities and studying their use in T2I generation refinement. The
benchmark and
dataset will be released upon publication.
Read the full abstract ▾
All authors Yu Zhao, Jiarui Wang, Huiyu Duan, Ye Zhao, Jutao Tang, Juntong Wang, Guangtao Zhai, Xiongkuo Min
Computer Vision
Decompose Radicals, Then Reward: Fine-Grained Inspection for Accurate Chinese Text Rendering
Yazhen Xie, Xingsong Ye and Zhineng Chen
Rendering accurate Chinese text remains challenging for text-to-image models. Existing OCR-based reinforcement-learning rewards compare decoded transcripts with target strings. Such rewards overlook the compositional nature of Chinese writing: an ideograph consists of reusable components arranged through explicit spatial relations, yet OCR evaluates it as an atomic character. Consequently, visually different radical-level errors may receive equally coarse feedback, encouraging glyphs that merely resemble the target instead of faithfully reproducing its internal structure. We employ Ideographic Description Sequences (IDS), which comprise spatial operators and character components, and train an expert IDS recognizer to transcribe rendered Chinese text into this representation. Building on this recognizer, we introduce IDSpect, which deterministically decomposes the target text into IDS
tokens and aligns crop-level visual IDS predictions with the target sequence. Globally unique
token credit makes this comparison robust to the order of detected text regions. Combined with a whole-character semantic reward, IDSpect supplies fine-grained credit with component and spatial-relation without changing the image generator or adding inference-time cost. Experiments with GRPO post-training of Qwen-Image demonstrate that IDSpect achieves leading structural quality and semantic
alignment on LongText and GenTextEval.
Read the full abstract ▾
Computer Vision
APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants
Jianguo Huang, Jinming Liu, Qiyao Wang et al. (+9 more)
To serve as real-world personal assistants, streaming
video models need persistent
memory that retains past experiences for later use. Yet existing streaming
benchmarks and methods often focus on individual continuous
videos or short clips, overlooking that real-world interactions are often intermittent and require
memory to persist across interruptions. To fill this gap, we introduce APM-Bench, which reformulates real-world streaming interaction as multi-session life trajectories. It contains 549 sessions, 104 trajectories, and 2,719 candidates, spanning both objective and open-ended questions. Each session is a
video with fine-grained annotations, and sessions within a trajectory revolve around related activities. Models then use persistent
memory to answer questions about past sessions and provide proactive responses while maintaining real-time interaction. This raises challenges: persistent
memory must be storable, selectively retain information, be injected at the right time, and remain efficient. Moreover, finite storage may leave required evidence unavailable, so assistants should recognize missing evidence. Therefore, we systematically evaluate general
video models under different
memory protocols and diverse specialized streaming
memory systems, and test whether models acknowledge insufficient evidence. Our evaluation reveals a clear utility–latency–storage trade-off: existing methods still struggle to simultaneously achieve reliable long-term recall, low overhead, and effective proactive assistance across sessions. APM-Bench provides a comprehensive testbed for developing and comparing persistent
memory systems under realistic streaming conditions. We hope it encourages future work that jointly considers utility, latency, and storage toward more practical persistent
memory for real-world streaming assistants.
Read the full abstract ▾
All authors Jianguo Huang, Jinming Liu, Qiyao Wang, Liang Xu, Jianhang Li, Zhimian Wen, Mingda Li, Shule Lu, Zhicheng Wang, Yuhan Guo, Xin Jin, Wenjun Zeng
Computer Vision
Weeding Out Bad Seeds: Initial-Noise-Robust Unlearning for Text-to-Image Diffusion Models
Arian Komaei Koma, Seyed Amir Kasaei, Aida Aryafar et al. (+6 more)
Machine unlearning has emerged as a critical post-hoc safety measure to erase sensitive concepts from Text-to-Image (T2I) models without prohibitive retraining. However, we reveal that current state-of-the-art (SOTA) approaches are brittle due to a severe lack of robustness to noise initialization. We call this phenomenon “probabilistic forgetting”: suppressed concepts re-emerge under specific random initial noise conditions, despite appearing unlearned on other initializations. We trace this failure to the misalignment between standard Gaussian sampling during unlearning and the unlearning objective. Since the target concept manifests only in specific initial noise regions throughout the unlearning phase, uniform random sampling yields sparse, uninformative gradient updates that fail to drive robust erasure. To overcome this issue, we propose an adaptive, concept-conditioned sampling strategy that dynamically concentrates gradient updates on regions where the target concept manifests, down-weighting uninformative areas. We integrate our framework with six distinct SOTA unlearning methods across four
diffusion backbones and evaluate it across safety, object, and artistic-style unlearning, as well as under black-box and white-box
adversarial attacks. Our method reduces the conditional nudity re-emergence rate across random initializations by 67.2% on average over four baselines and lowers attack success rates across both
adversarial evaluations. Across concept domains, Adaptive Noise Sampling strengthens
adversarial robustness and non-target retention while preserving competitive generative quality and target-erasure performance.
Read the full abstract ▾
All authors Arian Komaei Koma, Seyed Amir Kasaei, Aida Aryafar, Matin Ghiasi, Ali Aghayari, Amirhossein Souri, Mohammad Mosayyebi, AmirMahdi Sadeghzadeh, Mohammad Hossein Rohban
Computer Vision · Cross-listed from cs.RO
RawVLA: Embodied Neural Image Signal Processor For Robotic Manipulation
Shuhong Liu, Heng Zhou, Lingfeng Qian et al. (+7 more)
Vision-
language-action (VLA) models typically operate on RGB images produced by a fixed camera image signal processor (ISP), leaving the imaging pipeline outside the learning and evaluation loop. We systematically examine the consequences of this overlooked design choice across five fundamental ISP dimensions: gain, sensor noise, chromatic response, tonal response, and bit depth. Our analysis reveals that RAW-to-RGB processing materially shapes both action prediction and manipulation success, with different ISP dimensions exerting substantially different effects. Guided by these findings, we introduce RawVLA, a streaming neural ISP that adaptively renders RAW observations for frozen VLA policies while concentrating its capacity on the imaging factors relevant to embodied behavior. We further present RawVLA-Bench, a RAW-domain manipulation
benchmark to expose image processing as an explicit evaluation variable across clean and adverse acquisition conditions. Experiments on RawVLA-Bench show that RawVLA preserves performance under standard conditions while substantially improving robustness under degraded imaging, establishing adaptive RAW processing as an effective interface between physical cameras and embodied policies.
Read the full abstract ▾
All authors Shuhong Liu, Heng Zhou, Lingfeng Qian, Yuhao Fang, Xianbao Hou, Qianyu Zhou, Lin Gu, Wei Sui, Jianfei Yang, Ziteng Cui
Computer Vision
Principled MAP estimation for inverse problems: bridging the gap between convergence and performance
Alexandre Lagier, Valentine Tosel, Anne Gagneux et al. (+2 more)
Pretrained denoisers provide a powerful way to incorporate image priors into restoration
algorithms. Plug-and-Play and RED approaches exploit fixed-noise-level denoisers within first-order optimization schemes, with convergence guarantees, but often struggle to achieve high-quality reconstruction on severely ill-posed inverse problems. In contrast, recent state-of-the-art approaches leverage denoisers derived from flow- or
diffusion-based generative models and evaluate them along a sequence of decreasing noise levels. While these methods achieve strong empirical performance, their convergence theory remains limited. In this paper, we bridge this gap by specifically designing an
algorithm that combines denoisers at decreasing noise levels with a schedule tailored to ensure convergence. From a
Bayesian perspective, we prove that our method converges to a $\textit{Maximum a Posteriori}$ (MAP) estimate, under suitable assumptions. Subsequently, we apply our method to various ill-posed inverse problems and show that it surpasses convergent methods while competing with state-of-the-art empirical ones.
Read the full abstract ▾
All authors Alexandre Lagier, Valentine Tosel, Anne Gagneux, Mathurin Massias, Ségolène Martin
Computer Vision · Cross-listed from cs.LG
Hyunjong Ok, Seunggu Kang and Jaeho Lee
Thorough evaluation of
vision-
language models (VLMs) has become prohibitively expensive, as
benchmarks span an ever-broader spectrum of capabilities and new models arrive at a relentless pace.
Benchmark compression methods that preserve model rankings at a fraction of the cost are well studied for
language models, but for VLMs the question remains under-explored. We present PRIMEBench (
Pruning Redundant Items for
Multimodal Evaluation), a
vision-aware hierarchical
benchmark compression framework that substantially reduces evaluation cost while preserving model rankings. This hierarchical framework operates in four stages: data cleaning to remove items answerable without the image and all-correct items, category representative selection to pick one
benchmark per capability category, item
pruning with
Vision-Aware Variance (VAW), and category-count
pruning. VAW combines inter-model variance with a
vision-dependence score computed from
multimodal embeddings alone, while encouraging coverage of diverse items within each
benchmark. On models held out from item selection, it has the highest mean fidelity at the released 5% retention. The hierarchical design lets practitioners stop at any stage to match their compute budget; the released suite removes over 97% of items while preserving model rankings. Beyond compression, our analyses show how VLM evaluation behaves as model panels grow and evolve, providing guidance for designing future
benchmarks that are more efficient, robust to model turnover, and explicit about the limits of evaluation-side
pruning.
Read the full abstract ▾
Computer Vision
LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension
Arka Mukherjee, Kaleen Shrestha, Larissa Zhu et al. (+1 more)
Modern
vision-
language models (VLMs) have shown promising results in long-
video understanding due to the rich semantic information they can capture. However, most methods focus on coarse captioning of extracted image frames that are computationally inefficient and require models with large context windows. While past work has explored efficient methods through
multimodal retrieval-augmented generation (RAG), they rely on lossy
embeddings that lose temporal context and fine-grained detail. Few works to date have investigated how VLM-based query-relevant information
retrieval can be optimized. We introduce LazySloth, an efficient tree-based search method that speeds up
video comprehension and
retrieval tasks 2.9-8.3x (compared to existing agentic methods) through bounded captioning of portions of the
video considered irrelevant by a VLM of the
video. Compared to contemporary specialized
video-understanding VLMs and RAG-based methods, LazySloth achieved similar or better final task accuracy across two recent open-source base VLMs–Gemma 4 31B and Qwen3.6 27B–across four
benchmarks. LazySloth reduced the gap between the base open-source model and a closed-source model, GPT-4o. Ablations showed that replacing VLM scene understanding with CLIP-based
retrieval cost 8.8-19.9% in accuracy, while lazy tree construction matches eager construction at a fraction of the captioning cost. With LazySloth, we demonstrate the possibility of faster long-
video comprehension without substantial loss in performance.
Read the full abstract ▾
All authors Arka Mukherjee, Kaleen Shrestha, Larissa Zhu, Maja Matarić
Computer Vision
Complementary Retrieval-Augmented Prompting for Consistent Long-Form Video Generation
Xianghan Wei, Xiaoda Yang, Zhi Wang et al. (+7 more)
While recent
video foundation models excel at generating high-quality short
videos, long-form
video generation remains a critical challenge, where a major bottleneck lies in conditioning independently generated shots to preserve consistent characters, scenes, and objects throughout a story. Existing training-free approaches typically condition target shots using retrieved historical visuals. However, these references often suffer from severe informational mismatch, either introducing irrelevant contextual redundancy or failing to provide the full combination of required elements for the target shot. To resolve this, we present Complementary
Retrieval-Augmented Prompting, an agentic framework that strategically aggregates a compact set of mutually supportive historical references to achieve complete and targeted conditioning for long-form
video generation without retraining or modifying the underlying generator. Specifically, our framework explicitly models the visual elements required by each target shot by parsing the narrative script into a text-grounded visual element registry that tracks characters, objects, scenes, and their shot-level states. A VLM-annotated keyframe library further maps these elements to past visual observations. Guided by the required elements, our
agent retrieves complementary references that maximize target-element coverage while minimizing historical noise. Finally, the retrieved references, structured element states, and grounding instructions are assembled into a unified
prompt for the frozen
video generator. This element-aware process provides comprehensive conditioning while remaining fully interpretable. Quantitative and qualitative evaluations on multi-shot story generation demonstrate that our method consistently outperforms recent-frame,
memory-based, and entity-level
retrieval baselines in cross-shot consistency and text-controllability.
Read the full abstract ▾
All authors Xianghan Wei, Xiaoda Yang, Zhi Wang, An Pan, Daoan Zhang, Huayi Zhang, Yan Zhang, Wei Xu, Zishun Liao, Jianwen Lou
Computer Vision
BeatDance: Generating Beat-Consistent 3D Dance with Hierarchical Spatial-Temporal Modeling
Xiaojian Shen, Dahu Shi, Jianrong Zhang et al. (+7 more)
Generating realistic 3D dance from music is a challenging task that requires accurate synchronization with musical rhythms while capturing the spatial complexity of human motion. Although existing methods can generate physically plausible dance motions, they often struggle to achieve precise
alignment with music, such as the beat. To address this limitation, we propose a novel
diffusion-based framework, BeatDance, with two components: 1) We present a Hierarchical Decoupled
Attention (HDA) module, which first disentangles the learning of human pose and temporal dynamics. A hierarchical structure is then employed to capture both short-term and long-term dependencies, thereby enhancing spatial-temporal modeling. 2) We adopt cycle-consistent learning by introducing an auxiliary dance-to-music module. During training, discrepancies between the reconstructed and original music induce a stronger loss signal, effectively encouraging the consistency property between the music and dance motion. Extensive experimental results demonstrate that our proposed approach outperforms recent competitive methods on two
benchmark datasets.
Read the full abstract ▾
All authors Xiaojian Shen, Dahu Shi, Jianrong Zhang, Hai Li, Hongwei Zhao, Dawei Zhang, Yunzhi Zhuge, Zhiliang Wu, Guanghui Yue, Wei Zhou
Computer Vision
Multi-task learning for the automatic grading of enlarged perivascular space burden using MRI
Jesse Phitidis, William N. Whiteley, Joanna M. Wardlaw et al. (+11 more)
Enlarged perivascular spaces (PVS) visible in
brain magnetic resonance imaging (MRI) are increasingly thought to be linked to poor
brain health. PVS are elongated structures of less than 3 mm in diameter and can be numerous. To reflect the incidence of PVS, radiologists visually score their burden following a clinical grading scale - a task that would benefit from automation to accelerate analyses and overcome the influence of inter-observer differences. We developed and evaluated methods for training machine learning models to score PVS incidence in the basal ganglia (BG) and centrum semiovale (CSO) leveraging the Potters/Wardlaw scale. The novelty in our work lies in the use of imperfect, semi-automatically generated "silver-standard" PVS segmentation masks during training, in addition to PVS radiological scores. We comparatively evaluated a conditional convolutional neural network (CNN) which accepts PVS masks as an extra input channel, a multi-task CNN which performs both PVS segmentation and scoring, and a logistic regression model which utilises features derived from PVS masks to predict PVS scores. Multi-task learning was the most effective method, achieving a mean average precision of 64.08% compared to 60.22% for the conditional CNN, 52.11% for a baseline CNN trained only to predict PVS scores, and 49.32% for the logistic regression model. The multi-task model showed an ability to localise individual PVS not shown by the other CNNs, and behaved in a probabilistically sensible way, predicting with lower confidence on inherently harder classes. Age, sex, hypertension status, white matter hyperintensity volume, and ischaemic stroke lesion status were shown to be associated with the multi-task model's PVS score predictions and the ground truth in a similar way.
Read the full abstract ▾
All authors Jesse Phitidis, William N. Whiteley, Joanna M. Wardlaw, Miguel O. Bernabeu, Yajun Cheng, Xiaodi Liu, Junfang Zhang, Una Clancy, Stephen Makin, Roberto Duarte Coello, Susana Muñoz Maniega, Mark E. Bastin, Simon R. Cox, Maria del C. Valdés Hernández
Computer Vision
Anatomy-Aware Prediction of Bronchoscopic Accessibility from 3D CT
Linkai Peng, Cuiling Sun, Bin Wang et al. (+10 more)
Pre-operative planning for bronchoscopy is critical for the diagnosis of lung lesions. Current accessibility assessment relies on subjective manual inspection of CT scans, which is time-consuming and prone to inter-observer variability. In this paper, we formalize bronchoscopy accessibility prediction as a novel supervised learning task and present the first end-to-end framework to address it. We propose an Anatomy-Aware Mixture-of-Experts (MoE) model that integrates specialized modules: a CT Expert for local morphological features, a Lobe Expert for anatomical priors, and a Path Geometry Expert that encodes the sequential constraints of the bronchial tree. To support this task, we curated the first clinical
dataset of 438 cases with pre-operative CT scans and documented procedural outcomes. Experimental results demonstrate that our method achieves an AUROC of 0.8052, significantly outperforming both state-of-the-art baselines and experienced human experts. This work establishes a new
benchmark for computer-aided interventional planning in pulmonary medicine. Our data and code will be publicly available at https://nubagcilab.github.io/BronchoAccess/.
Read the full abstract ▾
All authors Linkai Peng, Cuiling Sun, Bin Wang, Jamie Rowell, Catherine Gao, Oyku Ikizgul, Eminenur Sen Tasci, Andrea Bejar, Halil Ertugrul Aktas, Gorkem Durak, Momen Wahidi, Christopher Kapp, Ulas Bagci
Computer Vision
Do-JEPA: From Masking to Intervention in Latent World Models
Hossein Resani and Javen Qinfeng Shi
Latent world models are trained to predict what happens next, so nothing in their objective separates what an action caused from what merely co-occurred with it. Object-masking models such as C-JEPA intervene on what the predictor can see; we intervene on what physically happens. From one saved simulator state we run the dynamics under an action $a$ and under a reference action $a_{\varnothing}$, and train the model to predict the difference $\Delta z=z^{a}-z^{a_{\varnothing}}$ between the two latent futures. The resulting objective, Do-JEPA, has an effect loss, a support loss (where the action enters), a propagation loss (where its effect travels) and invariance losses (what must not change). In a synthetic system with object-aligned variables, support supervision finds the directly intervened object in 99.95% of test cases, where a sparse action mask sends the action to a nuisance slot in every case, and response-onset supervision recovers the ring-shaped propagation graph (edge AUROC 0.975 vs. 0.624). From pixels, the effect loss beats a control trained on exactly the same data: it lowers latent effect error by 28.4% on an end-to-end LeWM model and physical effect error by 13.5% when trained and tested on natural action sequences, and on three independently generated CausalWorld
benchmarks it lowers responsive effect error by about 20% under physics shifts and the latent context sensitivity of predicted effects by 66%. Trained from scratch it costs factual accuracy; fine-tuning an existing model with it removes this cost. Together, these results show that intervening on the world, rather than on what the model sees, helps latent world models predict what their actions cause.
Read the full abstract ▾
Computer Vision
MG-Thinker: Bi-Axial Self-Reflection for Multi-Image Reasoning Grounding
Heyu Huang, Chi Chen, Zonghao Guo et al. (+3 more)
Reinforcement learning (RL) has recently delivered substantial gains in
multimodal reasoning, opening a promising route for fine-grained visual perception. Yet for multi-image
reasoning grounding (MRG),
reasoning over real-world multi-image contexts toward pixel-precise localization, existing RL-based approaches overlook two characteristics intrinsic to this paradigm: a coarse-to-fine hierarchical
reasoning pattern, and heterogeneously distributed task–sample difficulties. In this work, we present MG-Thinker, a post-training RL framework that advances a new MRG paradigm featuring such hierarchical
reasoning, supported by a curated 25K MRG
dataset with task-adaptive Chain-of-Thought (CoT) annotations that elicit multi-perspective evidence before conclusion. To remedy the heterogeneous task–sample difficulties, we further propose Bi-Axial DAPO (BiA-DAPO), which decomposes rollout advantages along an intra-group signal axis and an inter-group competence axis through two complementary mechanisms, both grounded on our defined candidate pool for stable group-level statistics. Extensive experiments show that MG-Thinker achieves state-of-the-art performance on multi-image
reasoning grounding while consistently improving generalization across multi-image understanding and diverse
multimodal benchmarks.
Read the full abstract ▾
All authors Heyu Huang, Chi Chen, Zonghao Guo, Yuhua Li, Maosong Sun, Ruixuan Li
Computer Vision
Think Before You Score: Thinking Reward Model for Visual Generation
Xuehai Bai, Zhenchen Tang, Yang Shi et al. (+9 more)
Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and candidate outputs directly to scalar rewards, leaving implicit what should be evaluated for each individual case. We introduce Think Before You Score, a paradigm that explicitly determines what matters for each case before judging how well the candidate performs. Following this principle, we propose the Thinking Reward Model (TRM), which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards. We further observe that conventional pairwise preference optimization can induce score polarization, and introduce Pairwise Dual-Group Relative Policy Optimization (PD-GRPO), which leverages pairwise supervision to improve reward discrimination while preserving fine-grained pointwise scoring. Extensive experiments on image generation and editing reward-modeling
benchmarks demonstrate that TRM achieves state-of-the-art performance among open-source reward models while remaining highly competitive with proprietary alternatives. Moreover, using TRM as a reward for reinforcement learning consistently improves diverse visual generation models, demonstrating that its fine-grained, case-adaptive rewards translate into effective optimization signals for visual generation.
Read the full abstract ▾
All authors Xuehai Bai, Zhenchen Tang, Yang Shi, Dianyi Wang, Tengfei Liu, Wanshun Su, Xuanyu Zhu, Ruohui Wang, Haiwen Diao, Haotian Wang, Xiaoling Gu, Yuanxing Zhang
Computer Vision
Visual Anomaly Synthesis for Model Selection in Data Scarcity
Daniel Pröll, Thomas Kraxner, Tobias Schaefer et al. (+1 more)
Defect detection systems for industrial condition monitoring can only be relied upon if they are validated, yet defective samples are rare and, for a specific asset, often nonexistent. We present a framework that synthesizes severity-graded defects on real non-defective images without any defect references for the target asset, that can be used for model selection and validation. A defect taxonomy for common failure modes is distilled from literature into prescriptive
prompts at varying defect severities. Regions of interest are cropped from in defect-free images and edited with a pre-trained image generation model ("FLUX.2 [klein]"). Color-matching and blending are employed to improve structural coherence with the original image. Generations are filtered out by a scorer and by estimated detection difficulty. Model selection experiments on MVTecAD show image AUROC choice regret over model selection can be nearly halved compared to the best fixed model chosen with access to test data. Experiments show the need for severity-graded anomaly synthesis. A case study investigates the proposed method for in-situ monitoring of Pelton turbine runners in hydropower, where real defect images are rare and expensive to collect. A PatchCorebased anomaly detection model is fit on Pelton turbine images and selected and validated using synthetic images, showing strong detection performance (94 % correct detection at optimal threshold and AUROC 0.97). The model reliably detects moderate and advanced defects, while early-stage defects remain challenging, indicating the synthetic data meaningfully stresses detector sensitivity.
Read the full abstract ▾
All authors Daniel Pröll, Thomas Kraxner, Tobias Schaefer, Sebastian Hegenbart
Computer Vision · Cross-listed from cs.RO
Encore: Few-Shot Agentic Discovery of Manipulation Strategies
Yifan Kang, Zihan Wang, Zhiwen Fan et al. (+1 more)
Coding
agents can now write, run, and debug programs with little human help.
Robot tasks, however, are usually specified by a sentence that leaves out how to grasp, in what order to make contact, and what the result should look like, and an
agent given only the sentence must find these details by trial and error. We introduce ENCORE, which gives the
agent a few demonstrations as evidence to read rather than as training data. A deterministic builder distills each demonstration into a pack of multi-view keyframes, gripper events, frame strips, and the full trajectory. A coding
agent studies the pack, writes a policy program against a fixed perception and action API, refines it iteratively over a few development rollouts, and freezes it before a sealed evaluation that never reveals the success signal. On LIBERO-PRO, the
agent's first program already succeeds in half of the perturbed tasks with demonstrations and in one task without them, and the frozen programs outperform the strongest prior agentic system run with the same
language model (96.3% against 89.3%). On RoboDojo tasks whose instructions leave the goal unstated, no program succeeds without demonstrations. ENCORE also runs on a real bimanual
robot, learning cube handover and cup inversion from five demonstrations each.
Read the full abstract ▾
All authors Yifan Kang, Zihan Wang, Zhiwen Fan, Bangya Liu
Computer Vision
PCaPaint: Prostate Cancer Inpainting by Mitigating Shortcut Learning
Levente Lippenszky, Hongxu Yang, Marcell Dömötör et al. (+2 more)
The development of AI systems for tumor-specific applications is limited by the scarcity of labeled data. Synthetic tumor inpainting offers a promising approach but faces challenges for prostate cancer MRI which contains high-resolution multi-sequence data. Although methods leveraging latent
diffusion models (LDMs) enable large-volume synthesis, they are prone to shortcut learning, simply reproducing the condition image created by masking the lesion region. In this work, we introduce PCaPaint, a prostate cancer inpainting method based on LDMs that explicitly addresses this failure mode. To overcome shortcut learning that compromises synthetic tumor texture, we propose a simple yet efficient conditioning strategy in which the condition image is filled with Gaussian noise, and we provide theoretical justification. In addition, we propose a novel training objective for LDM that emphasizes the error within the lesion region. Furthermore, we introduce a multi-sequence latent design, in which T2w scans and DWI&ADC scans are compressed using two separate autoencoders to preserve their distinct frequency characteristics. Extensive experiments demonstrate that the generated synthetic data improves downstream performance in prostate lesion segmentation, patient-level classification and lesion-level detection. Furthermore, our method significantly outperforms a recent state-of-the-art LDM-based tumor inpainting method both in downstream performance and in synthetic image quality.
Read the full abstract ▾
All authors Levente Lippenszky, Hongxu Yang, Marcell Dömötör, Krisztian Koos, László Ruskó
Computer Vision
TAEC: Trajectory-Aware Evidence Coordination for Multi-Step Visual RAG
Yalun Wu, Bingzhou Wang, Boyang Wang et al. (+5 more)
Multi-step visual
retrieval-augmented generation (RAG) answers complex questions by repeatedly retrieving visual evidence, updating an intermediate state, and deciding whether to continue searching or answer. Yet retrieving relevant evidence does not ensure its effective use throughout the
reasoning trajectory. As multi-step
reasoning progresses, redundant sources occupy context capacity needed for missing evidence, observations tied to resolved requirements or unproductive searches linger in context, and visual sources are revisited with insufficient detail for fine-grained reading. We term this loss of usable evidence over a
reasoning trajectory trajectory-level evidence utilization degradation. To address it, we propose Trajectory-Aware Evidence Coordination (TAEC), a training-free framework that coordinates evidence use around unresolved answer requirements. TAEC tracks these requirements in a shared trajectory state to guide which evidence enters the context, how accumulated
memory is retained, and at what level of detail visual evidence is examined. Under a unified evaluation protocol on ViDoSeek, SlideVQA, and MMLongBench-Doc, TAEC achieves the best overall performance against leading training-free visual RAG baselines, with the highest average accuracy across multiple proprietary
vision-
language models. These results demonstrate that aligning evidence with evolving
reasoning needs improves evidence use throughout multi-step visual RAG.
Read the full abstract ▾
All authors Yalun Wu, Bingzhou Wang, Boyang Wang, Peiying Wang, Shaojie He, Yunhan Wang, Shaozu Yuan, Jiawei Wang
Computer Vision
When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs
Xiang Hu, Jiazuo Yu, Lu Zhang et al. (+2 more)
Streaming
video understanding requires
Video Large
Language Models (
Video-LLMs) to reason over continuous visual streams under causal constraints. As the visual history grows, a bounded visual?processing budget requires evidence selection that balances temporal recency with query relevance. Recent-only selection excludes potentially relevant historical evidence, whereas Semantic-only
retrieval can displace useful recent context when relevance scores are ambiguous. We introduce WRWS (When to Retrieve, When to Stay), a training-free framework for uncertainty-adaptive evidence allocation. A lightweight external
vision-
language encoder scores query relevance across the observed history, while an adaptive allocation module uses the normalized
entropy of the similarity distribution as a proxy for
retrieval uncertainty. WRWS favors semantic
retrieval when relevance cues are reliable and strengthens the recency prior under uncertainty. Following a retrieve-first, encode-later pipeline, WRWS selects evidence before target-model visual encoding, such that only the selected observations are processed by the costly target
Video-LLM. Experiments across four
Video-LLM families and multiple model scales demonstrate competitive accuracy on StreamingBench and OVO-Bench. In our efficiency evaluation, WRWS reduces average
vision-to-answer time to 47.93% of the state-of-the-art method. Code will be released.
Read the full abstract ▾
All authors Xiang Hu, Jiazuo Yu, Lu Zhang, Yunzhi Zhuge, Huchuan Lu
Computer Vision
HyperSAM: A Promptable Foundation Model for Hyperspectral Remote Sensing
Li Pang, Xinqiao Wu, Jing Yao et al. (+5 more)
Hyperspectral remote sensing provides dense spectral measurements that are indispensable for material-level Earth observation, yet the construction of a general-purpose hyperspectral foundation model remains difficult. Two bottlenecks are especially limiting. First, large hyperspectral corpora rarely provide high spatial resolution together with reliable dense annotations. Second, many hyperspectral models are still trained almost from scratch, so the
geometric and interactive priors learned by modern
vision foundation models are not fully reused. To alleviate these issues, we \highlight{present} HyperSAM, a promptable hyperspectral foundation model that couples a data-centric hyperspectral synthesis pipeline with a spectral adaptation architecture based on Segment Anything Model 3 (SAM3). On the data side, HyperSAM synthesizes full-spectrum hyperspectral cubes from high-resolution SpaceNet multispectral imagery through a physics-informed abundance-transfer generator, while SAM3-derived pseudo-masks provide object-centric supervision. On the model side, the latest implementation uses a frozen SAM3 RGB image branch, a trainable hyperspectral side encoder initialized from the RGB
vision transformer (ViT), ControlNet-style zero-initialized feature injection, and a lightweight mixture-of-experts mask refiner. To enhance training robustness against noisy pseudo-labels, Cross-modal Sample Selection (CromSS)-style confidence selection is incorporated for noisy-label weighting. Extensive experiments show that HyperSAM obtains strong generalization on diverse hyperspectral tasks (e.g., classification, anomaly detection, change detection, target detection, and airborne oil-spill mapping) and that high-quality synthetic hyperspectral data can be more effective than simply scaling noisy hyperspectral supervision.
Read the full abstract ▾
All authors Li Pang, Xinqiao Wu, Jing Yao, Pedram Ghamisi, Jun Zhou, Zhengchao Chen, Deyu Meng, Xiangyong Cao
Computer Vision
UGO: Unified Architecture for General Multi-Object Tracking by Segmentation
Jer Pelhan, Alan Lukezic and Matej Kristan
General multi-object tracking (GMOT) tracks all instances of a user-specified category from a single first-frame exemplar. Prior work relies on bounding boxes and surrogate training, and struggles with non-rigid objects, crowded scenes, and distractors. We introduce UGO, a unified GMOT tracker that pairs a pretrained exemplar-conditioned detection head with an instance-propagation head in a common architecture. A novel training-free, energy-minimization consolidation method converts overlapping proposals into exclusive pixel-wise masks and detections, resolving over-segmentation, duplicates, and conflicts. A hierarchical
memory spanning global and instance levels improves recall and per-instance segmentation accuracy using a new
memory management protocol. UGO sets a new state-of-the-art on GMOT
benchmarks and
video object counting, and is competitive with specialist MOT methods, establishing a strong paradigm for unified, open-category multi-object tracking.
Read the full abstract ▾