Computer Vision
Creating Impactful Autonomous Driving Datasets: A Strategic Guide from Research Gap to Benchmark
Richard Schwarzkopf, Jonas Merkert, Frank Bieder et al. (+22 more)
Well-designed autonomous driving
datasets have fundamentally shaped research progress, yet existing literature primarily describes what
datasets contain rather than how to strategically design impactful ones. This is especially limiting for small and medium-sized labs and startups that cannot afford to misallocate scarce resources. We argue that impactful
dataset creation begins with a diagnosis: whether a research question is blocked by a data problem or an evaluation problem, and proceeds by selecting the minimal data operator(s) that closes the resulting gap, recording new data only when no cheaper operator(s) suffices. We analyze the
evolution of major autonomous driving (AD)
datasets through this lens and distill a strategic framework spanning gap identification, operator choice, sensor suite design, and annotation strategy. We ground the framework in a running case study of our KITScenes
dataset family. The
datasets are available at: https://kitscenes.com/
Read the full abstract ▾
All authors Richard Schwarzkopf, Jonas Merkert, Frank Bieder, Annika Bätz, Alexander Blumberg, Carlos Fernandez, Felix Hauser, Fabian Immel, Christian Kinzig, Hendrik Königshof, Fabian Konstantinidis, Martin Lauer, Willi Poh, Nils Rack, Kevin Rösch, Yinzhe Shen, Marlon Steiner, Gleb Stepanov, Dominik Strutz, Ömer Şahin Taş, Julian Truetsch, Kaiwen Wang, Royden Wagner, Jan-Hendrik Pauls, Christoph Stiller
Computer Vision
Imprint: Online Memory Compression for Long-Horizon Egocentric QA
Kousik Das and Debaditya Roy
Long-horizon
egocentric question answering involves answering about events that have occurred hours or days in the past. This requires
memory representations that remain both
retrieval-effective and scalable over days or weeks of recording. Existing long-horizon
egocentric QA methods construct
memory as hierarchical textual summaries of observations. While effective for reducing
memory size, summarization optimizes for descriptive compression rather than
retrieval: repeated interactions are absorbed into coarse textual descriptions instead of being preserved as explicit, recurring
memory units, making long-horizon evidence aggregation difficult. We propose Imprint, an interaction-centric
memory framework that formulates long-horizon
egocentric memory as an online
memory compression problem rather than summarization. Incoming observations are first represented as structured Interaction Records and continuously organized into recurring interaction patterns. Using human
memory consolidation signals of recurrence, recency, and distinctiveness, Imprint selectively retains and compresses interactions into a compact
retrieval-oriented
memory. We evaluate Imprint on EgoLifeQA, a seven-day
egocentric benchmark containing questions that require
reasoning over interactions occurring hours to days before the query. With the same LLM, Imprint improves QA accuracy from 31.0% to 35.8%, increases evidence-grounded answers by $6\times$ compared with EgoRAG, reduces
memory footprint by $2.3\times$, and decreases
retrieval latency by $11.8\times$. These results demonstrate that
memory compression provides a scalable and
retrieval-effective foundation for long-horizon
egocentric question answering.
Read the full abstract ▾
Computer Vision
LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter
Tobias Christian Nauen, Anosh Billimoria, Federico Raue et al. (+3 more)
Comparing
transformer backbones for image
segmentation is confounded: each is paired with a different decoder, recipe, and pretraining, so reported differences rarely reflect the backbone itself. We introduce the Lightweight Universal Mask Adapter (LUMA), a lightweight, backbone-agnostic mask-
transformer head that treats any backbone as a black-box feature extractor, letting a set of queries read from its features through cheap cross-
attention. LUMA matches the accuracy of EoMT, the state-of-the-art efficient ViT-segmenter, at cost parity, while attaching unchanged to hierarchical, convolutional, and mixture-of-experts backbones alike. Holding this head fixed, we
benchmark 20 backbones, 11 pretraining schemes and a range of resolutions on ADE20K, COCO, and Cityscapes under one modern recipe. We find that “efficient”
token mixers fail to deliver efficiency even at the high resolutions that motivate them, with plain ViT holding the throughput Pareto front at every resolution. Additionally, the pretraining objective, not the architecture, the lever the field has tuned hardest, governs
segmentation quality. Code: https://github.com/tobna/LUMA
Read the full abstract ▾
All authors Tobias Christian Nauen, Anosh Billimoria, Federico Raue, Stanislav Frolov, Brian B. Moser, Andreas Dengel
Computer Vision
ABot-M0.5: Unified Mobility-and-Manipulation World Action Model
Ronghan Chen, Yandan Yang, Zuojin Tang et al. (+18 more)
Mobile manipulation is a key capability for general-purpose
robots, yet remains challenging for current
embodied learning methods. VLA policies are typically reactive and lack explicit world modeling, while existing World Action Models (WAMs) are still poorly aligned with the structure of mobile manipulation: they operate on coarse
video chunks, model entangled
navigation-manipulation actions, and train inverse dynamics under supervision that does not match autoregressive inference. As a result, they often miss fine-grained contact dynamics, suffer from action-distribution conflicts, and accumulate errors over long-horizon rollouts. We propose ABot-M0.5, a new WAM built on the insight that mobile manipulation requires
alignment at three levels: temporal granularity, action space, and train-test consistency. To align temporal granularity, we introduce intermediate latent actions that capture local visual state transitions and serve as an bridging action space between
video latents and embodiment-specific controls. To align action space, we design a dual-level Mixture-of-
Transformers architecture that disentangles both modality representations and heterogeneous action subspaces such as base movement and arm manipulation. To align inference conditions, we propose the dream-forcing training strategy that progressively trains inverse dynamics on model-predicted
videos, improving train-test
alignment and robustness during autoregressive prediction. Experiments on challenging mobile and fine-grained manipulation
benchmarks demonstrate that ABot-M0.5 achieves state-of-the-art performance in both long-horizon task success and finegrained control accuracy. These results highlight the critical importance of granularity-aligned, action-disentangled, and inference-consistent world-action modeling.
Read the full abstract ▾
All authors Ronghan Chen, Yandan Yang, Zuojin Tang, Dongjie Huo, Tong Lin, Haoning Wu, Haoyun Liu, Yuzhi Chen, Lulu Zheng, Botai Yuan, Tianlun Li, Mingxin Wang, Dekang Qi, Bin Hu, Wei Mei, Yuze Xuan, Haolong Yang, Yanqing Zhu, Mu Xu, Zhiheng Ma, Xinyuan Chang
Computer Vision
DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding
Zhengbo Zhang, Mark He Huang, Zhigang Tu et al. (+1 more)
Zero-shot
video temporal grounding (VTG) localizes events in untrimmed
videos from natural
language queries without task-specific training. Existing methods rely on frame-query feature matching, which suffices for simple events but struggles with complex multi-stage queries that require understanding temporal ordering and causal structure – a disparity we call the
reasoning gap. We propose DART (Difficulty-Adaptive Routing for Temporal Grounding), which bridges this gap by coupling difficulty-aware routing with structured
reasoning in large
vision-
language models. A query-conditioned Determinantal Point Process (DPP) serves a dual role: selecting diverse, query-relevant keyframes as temporal evidence, and providing spectral
entropy as a difficulty indicator. Simple queries are routed to a Fast path for direct prediction, while complex queries follow a Slow path with Temporal Markup Prompting, which decomposes localization into global event analysis, per-frame temporal role annotation, and boundary extraction. On Charades-STA and ActivityNet Captions, DART achieves state-of-the-art zero-shot performance across both identically distributed and multiple out-of-distribution settings, improving mIoU by up to 3.5 points over the strongest baseline while using over 7 times fewer frames. The project homepage is available at https://dart-vtg.github.io/.
Read the full abstract ▾
All authors Zhengbo Zhang, Mark He Huang, Zhigang Tu, Ming-Hsuan Yang
Computer Vision · Cross-listed from cs.RO
Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shifts
Taewook Kang, Taeheon Kim, Donghyun Shin et al. (+1 more)
Vision-
Language-Action (VLA) models often fail to perform the same learned tasks under environmental shifts, such as changes in camera pose and shifts to a different but similar
robot (e.g., from Panda to UR5e). Adapting these models to the shifted environment (i.e., target domain) often requires training on multiple demonstrations for each task, which are costly to collect. To reduce the burden of data curation and training, we propose an analogy-based method that adapts VLA models under environmental shifts through weight vector arithmetic with domain-specific information addition, named Domain ARiThmetic (DART). Unlike prior approaches, DART requires collecting only a single demonstration, enabling efficient adaptation. To accurately isolate domain-specific information for addition, DART performs subspace
alignment between singular components in weight vectors to filter out noisy components. In both simulated and real-world experiments, DART outperforms existing VLA adaptation methods in one-shot scenarios across diverse visual and embodiment shifts. Code is available at https://github.com/snumprlab/dart.
Read the full abstract ▾
All authors Taewook Kang, Taeheon Kim, Donghyun Shin, Jonghyun Choi
Computer Vision
Linguistic Relative Policy Optimization for Video Anomaly Reasoning
Jiaxu Leng, Jiankang Zheng, Mengjingcheng Mo et al. (+4 more)
Video anomaly detection (VAD) with
multimodal large
language models has shown strong potential, yet most existing methods still depend on large-scale annotations or expert-designed priors, limiting their ability to acquire anomaly knowledge with as little human intervention as possible. To address this, we propose Linguistic Relative Policy Optimization (LRPO), which distills group-relative semantic advantages from multiple
reasoning trajectories into a linguistically expressed anomaly experience prior, and adapts the model by injecting this prior into the context to steer its output distribution without any parameter updates. LRPO builds two complementary experience representations: general experience captures transferable anomaly preferences across scenarios, while scenario experience models context-dependent anomaly rules for targeted refinement. To further improve the learned experience, we introduce an anomaly
alignment reward that guides trajectory optimization to match human risk preferences and reinforce temporally grounded
reasoning. Extensive experiments on XD-Violence, UCF-Crime, and UBnormal demonstrate that LRPO significantly outperforms existing state-of-the-art methods under tuning-free settings.
Read the full abstract ▾
All authors Jiaxu Leng, Jiankang Zheng, Mengjingcheng Mo, Zhanjie Wu, Haosheng Chen, Ji Gan, Xinbo Gao
Computer Vision
Not All Prediction Targets Keep Training-Free Diffusion Guidance on the Manifold
Yunsung Lee and Hyeongmin Lee
Training-free guidance (TFG) steers a pretrained
diffusion model toward a desired attribute at inference. To be effective, this guidance must be applied from the earliest, high-noise steps of sampling. Because its objective (a classifier or energy) is defined on clean images, $\epsilon$- and $v$-prediction models must first estimate the clean image $\hat{x}$ from the noisy state at each step, and the accuracy of that estimate determines how easily guidance drifts off the data
manifold. $x$-prediction, a recent alternative, outputs the clean image directly, removing this source of error even at high noise. This is our motivation. We provide a theoretical analysis of how each prediction target shapes this accuracy, and introduce guided-class FID (Child FID), a metric that exposes the
manifold damage standard evaluation misses. Experiments on a new fine-grained bird
benchmark and on style transfer confirm that $x$-prediction keeps guided samples on the
manifold most reliably, making it the strongest foundation for training-free guidance. Code is available at https://github.com/ManLuML/on-
manifold-tfg
Read the full abstract ▾
Computer Vision
Uncertainty-aware tree height change regression
Max Gaber, Dimitri Gominski, Jaime C. Revenga et al. (+3 more)
Monitoring canopy height change is essential for understanding carbon sinks and forest dynamics. Remote sensing enables consistent, large-scale observations of such changes, increasingly integrated with deep learning architectures such as Geospatial Foundation Models (GFMs). However, existing methods and
datasets frame the problem as binary change detection, which overlooks both the continuous nature of change, especially for vegetation, and the inherent uncertainty in labels. We present the Canopy Height Change (CHC)
dataset, providing 3 $\mathrm{m}$ resolution continuous canopy height differences and associated spatially resolved uncertainties across 10598 $\mathrm{km}^2$ of northern and western Spain. The
dataset is paired with a co-located time series of PlanetScope satellite imagery. Based on the
dataset, we introduce the task of uncertainty-aware change regression, associated metrics and strategies for fine-tuning GFMs. Furthermore, we evaluate state-of-the-art GFMs and highlight promising directions and remaining challenges for advancing continuous canopy height change estimation.
Read the full abstract ▾
All authors Max Gaber, Dimitri Gominski, Jaime C. Revenga, Stefan Oehmcke, Rasmus Fensholt, Martin Brandt
Computer Vision
Learning to Watch: Active Video Anomaly Understanding via Interleaved Policy Optimization
Mengjingcheng Mo, Jiaxu Leng and Xinbo Gao
Video anomaly understanding (VAU) relies on sparse, context-dependent cues. However, existing passive paradigms suffer from observational aliasing, where static sampling fails to disambiguate semantically distinct events. To overcome this, we propose $Anom\text{-}\pi$, a closed-loop framework that reconceptualizes
video understanding as an active sequential decision-making process within a dynamic environment. Inspired by human
video-reviewing behavior, this framework unifies internal cognitive
reasoning and strategic evidence acquisition into an interleaved policy, utilizing temporal atomic operators such as local backtracking, temporal expansion, and fine-grained sampling to endow the model with perceptual proactivity. To learn such complex interaction strategies under
video-level weak supervision, we design Interactive Direct Preference Optimization (iDPO) to achieve trajectory-level policy
alignment, guided by an Active Evidence Inquiry (AEI) utility that balances task success, informative evidence acquisition, and interaction cost. This approach enables the
agent to learn to actively disambiguate hypotheses while suppressing redundant exploration. Extensive experiments demonstrate that our framework, with only 2B parameters, achieves highly competitive performance, significantly outperforming state-of-the-art large-scale VAU models in complex scenarios.
Read the full abstract ▾
Computer Vision
Identifying Latent Concepts and Structures for Generalized Category Discovery
Boyang Dai, Chaoqi Chen and Yizhou Yu
Generalized Category Discovery (GCD) aims to recognize known classes while autonomously discovering novel ones in open-world settings. However, current approaches primarily focus on designing clustering objectives, often overlooking a critical bottleneck: standard
vision backbones yield high-rank, entangled
token representations that are ill-suited for unsupervised discovery of latent concepts and structures. In this paper, we propose Compositional Primitive Fields (CPF-GCD), a novel representation learning framework that reshapes the feature space to make such latent structure identifiable by enforcing a low-rank compositional organization. Our core hypothesis is that all categories, whether known or novel, can be expressed as compositions and spatial arrangements of a finite set of learnable visual primitives that capture reusable concepts. CPF instantiates this
geometric constraint via a spatial field mechanism. Inserted between the backbone and the head, it rewrites noisy patch
tokens through low-rank primitive mixtures, effectively decomposing images into reusable atomic parts and their spatial layouts. By explicitly modeling the spatial distribution of primitives, CPF enables novel categories to emerge naturally as new activation patterns over a shared vocabulary. This shifts the focus of representation from merely partitioning global
embeddings to constructing a structured and separable primitive field. Extensive experiments demonstrate that CPF serves as a generic, plug-and-play module that consistently boosts performance across diverse GCD baselines, validating that identifying and leveraging low-rank compositional structure is a crucial inductive bias for open-world recognition.
Read the full abstract ▾
Computer Vision
Diffusion-Based Multi-Class Normality for OOD Detection: An Application to CDP Authentication
Bolutife Atoki, Iuliia Tkachenko, Bertrand Kerautret et al. (+1 more)
Reconstruction-based generative models offer a natural framework for unsupervised out-of-distribution (OOD) detection, but multi-class normality modelling requires a single detector to capture multiple in-distribution
manifolds and produce comparable anomaly scores across classes. We study this problem in copy detection pattern (CDP) authentication, where authentic and counterfeit samples are visually similar but differ in subtle printing-and-digitisation (P&D) signatures. We propose a
diffusion-based multi-class normality framework in which a single class-conditional ControlNet is trained exclusively on authentic CDPs from multiple P&D classes and detects counterfeits through reconstruction error under authentic-class conditioning. We further introduce dual template masking, which hides complementary regions of the input template and scores only withheld pixels, reducing reliance on visible binary structure. On the Indigo 1 x 1 Base
dataset, the proposed method outperforms traditional and adapted generative baselines under multi-class authentic-versus-counterfeit evaluation, without using counterfeit samples for training or threshold calibration.
Read the full abstract ▾
All authors Bolutife Atoki, Iuliia Tkachenko, Bertrand Kerautret, Carlos Crispim-Junior
Computer Vision
Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gap
Bingchen Huang, Zhiling Wang, Yifu Chen et al. (+1 more)
Recent work on Thinking with Images makes
vision a dynamic part of
reasoning, but does so through generation: the model invokes external tools, synthesizes code, or imagines new imagery, each at the cost of a tool protocol, brittle code, or an expensive training pipeline. A fourth route makes
vision dynamic without generating anything, by retrieving labeled exemplar images and
reasoning over them, yet it remains underexplored despite being train-free. We present ReVisIT, a train-free framework that realizes this
retrieval-based route by treating each retrieved image-label pair as a unit of visual thought. ReVisIT combines structured class definitions, per-query
multimodal retrieval of exemplars, and alternating user/assistant injection of those exemplars before joint multi-attribute decoding, and degrades gracefully to whichever components a task admits. On VL-ICL Bench Fast Open MiniImageNet, Qwen3-VL-30B-A3B with ReVisIT reaches 98.5% at 4-shot, statistically indistinguishable from the 72B LLaVA-OneVision SOTA (98.7%) on this near-saturated task at about 1/2.4 the parameters, while the same backbone without the scaffold sits at chance. The turns layer alone adds 26.1 points to GPT-4.1 on free-form concept induction (Bongard-OpenWorld), and the full stack yields a 4-6 point macro gain across three backbones on MAAC-Bench, a new license-clean 27-class, 5-attribute
benchmark, significant by paired bootstrap on the curator-derived attributes. Component analysis shows that
retrieval-plus-turns is the universal lever while structured definitions are need-adaptive, and that 83% of the
retrieval gain comes from
retrieval quality rather than from the presence of exemplars. MAAC-Bench is released with a rubric-grounded LLM verification protocol that replaces author spot-check on subjective attributes.
Read the full abstract ▾
All authors Bingchen Huang, Zhiling Wang, Yifu Chen, Yuanchao Du
Computer Vision
Semantic-Guided Reading Order Reconstruction in Historical Armenian Newspapers with LLMs
Chahan Vidal-Gorène, Nadi Tomeh and Victoria Khurshudyan
This paper addresses reading order reconstruction in historical Armenian newspapers, which combine complex layouts with limited
language resources. We introduce a new annotated
dataset of 66 pages and compare
geometric heuristics, YOLO-based layout parsing, an end-to-end document model ECLAIR, and a hybrid method combining semantic zone detection with a generative LLM. Our hybrid method achieves the lowest error rates of all evaluated approaches, reducing ordering errors by up to 76% over the strongest
geometric baseline, and remains robust in multi-page settings and under noisy OCR. Rather than targeting production the method is designed as a data bootstrapping strategy enabling rapid annotation in highly under-resourced scenarios. Alongside the
dataset, we release a specialized Tesseract OCR model for historical Armenian print.
Read the full abstract ▾
Computer Vision
GADA: Geometry-Aware Deformable Aggregation for Image-Based Gaussian Splatting
Siwoo Lim, Sunjae Yoon, Gwanhyeong Koo et al. (+1 more)
Gaussian Splatting has achieved significant improvements by incorporating warping-based techniques. However, such methods suffer from pixel-level inaccuracies due to uncertain geometry. This uncertainty leads to spatial misalignments in the warped images, which disrupt residual learning used in warping-based methods and fundamentally limit the gains of correction, particularly on thin structures and high-frequency details. Driven by our insight that useful visual cues are not lost but locally preserved under slight displacement, we propose Geometry-Aware Deformable Aggregation (GADA). This method introduces an iterative refinement module with deformable offsets to actively correct spatial misalignments and recover these displaced cues. Furthermore, to address the limitations of standard pipelines where visibility checks (i.e., thresholding) often discard valid pixels and multi-view warped image fusion relies on naive mean aggregation, our module is coupled with an implicit confidence weighting mechanism that selectively suppresses unreliable evidence. Consequently, our approach outperforms prior warping-based Gaussian Splatting, preserving high-frequency quality while achieving 2.13 times faster FPS.
Read the full abstract ▾
All authors Siwoo Lim, Sunjae Yoon, Gwanhyeong Koo, Chang D. Yoo
Computer Vision
Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers
Cong Liu, Xiaofang Li and Simon X. Yang
Vision Transformers (ViTs) commonly rely on injected positional mechanisms to address self-
attention's permutation invariance. Motivated by the spatial regularities of natural images, we ask whether spatial organization can be induced from data rather than explicitly injected. Under controlled, matched from-scratch training, we propose Active Spatial Guidance (Guidance), a training-only objective that disables positional injection and applies an auxiliary 2D coordinate-regression loss to the final-layer patch
tokens. The guidance head is used only during training and removed for inference; the deployed model consists of a positional-injection-free ViT encoder and the task-specific prediction module. Using DINOv3 ViT backbones, Guidance consistently improves performance on ImageNet-100 classification, ADE20K semantic
segmentation, and Hypersim
monocular depth estimation, outperforming strong injected baselines such as learned absolute positional
embeddings and rotary positional
embeddings under identical training protocols. On ImageNet-100, broader comparisons against representative injected positional designs further support Guidance's effectiveness. Guidance also improves robustness under resolution transfer, and multi-resolution training further strengthens accuracy across input sizes. Overall, our results suggest that spatial inductive bias in ViTs need not be architecturally injected, but can be shaped through training-time supervision. The code used for training and evaluation is publicly available in https://github.com/cloudlc/asg.
Read the full abstract ▾
Computer Vision
EPO: Boosting 3D Foundation Models with Edge-based Pose Optimization
Mattia D'Urso, Christian Sormann, Mattia Rossi et al. (+1 more)
We introduce Edge-based Pose Optimization (EPO), a trackless
geometric optimization framework specifically designed to boost the Structure-from-Motion reconstructions generated by 3D Foundation Models. These models achieve rapid inference by bypassing the time-consuming feature extraction and matching stages of traditional pipelines, where explicit correspondences between each 3D point and multiple images, referred to as tracks, are established. However, their
geometric accuracy currently falls short of traditional pipelines. While this can be addressed in a post-processing step via Bundle Adjustment-like refinement, doing so requires extracting feature tracks, thus defeating the original speed advantage. Instead, our fully differentiable framework uses edge map
alignment as a proxy for
geometric optimization, avoiding feature extraction and track construction entirely. Through extensive evaluation across multiple
datasets and tasks, we demonstrate that EPO matches or outperforms Bundle Adjustment-like methods while requiring significantly lower runtime and
memory. Notably, its reduced
memory footprint makes EPO suitable for consumer-grade hardware, where competing refinement methods cannot run.
Read the full abstract ▾
All authors Mattia D'Urso, Christian Sormann, Mattia Rossi, Friedrich Fraundorfer
Computer Vision
Caption Bottleneck Models
Seref Baris Cagliyan, Umut Ozdemir, Merve Tapli et al. (+1 more)
Concept Bottleneck Models (CBMs) provide interpretability by routing predictions through a layer of human-understandable concepts. However, defining an optimal concept set for a specific
dataset remains an open challenge. Existing approaches rely on expensive expert annotations or LLM-generated lists based solely on class names. Even "open-vocabulary" variants typically depend on static concept sets, which restrict discovery and introduce label bias. Furthermore, traditional CBMs often suffer from information leakage, where unmodeled visual features bypass the bottleneck and compromise the integrity of the explanations. To overcome these limitations, we propose Caption Bottleneck Models (CaBM), a framework that circumvents the need for predefined concept sets by replacing rigid concept layers with free-form natural
language. By representing images via LMM-generated captions and training a classifier strictly on this text, CaBM ensures a leakage-free architecture by construction. Additionally, by analyzing the text classifier post-training, CaBM autonomously discovers high-quality,
dataset-specific concepts. Our results across fine- and coarse-grained
benchmarks demonstrate that CaBM achieves competitive accuracy while preserving interpretability without the constraints of external dictionaries or manual labeling.
Read the full abstract ▾
All authors Seref Baris Cagliyan, Umut Ozdemir, Merve Tapli, Emre Akbas
Computer Vision
BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructure
Zijian Dong, Yi Lin, Fang Ji et al. (+3 more)
Diffusion MRI probes
brain microstructure with particular sensitivity to early cerebrovascular and neurodegenerative changes. Neurite Orientation Dispersion and Density Imaging (NODDI) decomposes the
diffusion signal into three biophysically interpretable maps: neurite density index (NDI), orientation dispersion index (ODI), and free water fraction (FWF), capturing neurite packing, fiber coherence, and extracellular fluid. These 3D maps offer a rich substrate for transferable microstructural representations, yet integrating them is challenging: standard representation learning struggles to disentangle the unique information in each map from their shared and synergistic interactions. We present BrainFIBRE, the first foundation model for
brain microstructure, pretrained on NODDI-derived maps from 55,592 UK Biobank participants. We propose Self-supervised Partial Information Decomposition (SPID), which extends PID-guided
multimodal learning to the self-supervised regime for the first time. A novel Counterfactual Candidate Construction (CCC) paradigm perturbs inter-modality
alignment through modality dropping and swapping, providing the contrastive signal for a Mixture-of-Experts architecture to disentangle unique, synergistic, and redundant information without any downstream label. On both Caucasian and Asian cohorts, BrainFIBRE achieves state-of-the-art performance across diverse tasks predicting age, sex, cerebrovascular and neurodegenerative markers, and cognition, while yielding neurobiologically interpretable representations that reveal task- and cohort-specific interaction patterns. BrainFIBRE establishes a versatile foundation for neuroimaging analysis at the microstructural level.
Read the full abstract ▾
All authors Zijian Dong, Yi Lin, Fang Ji, Jianxiong Zhou, Kwun Kei Ng, Juan Helen Zhou
Computer Vision
EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes
Jihyeok Jung, Jeewu Lee, Sanghyeop Kim et al. (+2 more)
Existing
egocentric benchmarks have primarily constructed the
egocentric setting from first-person-view data, which makes it difficult to evaluate
egocentric perspective itself in isolation. However, understanding first-person-view input and taking an
egocentric perspective are separable abilities, especially when first-person body cues are absent or when other
agents are present. To isolate
egocentric perspective understanding, we introduce EgoGapBench, a diagnostic
benchmark for measuring action selection in multi-
agent egocentric scenes. We define the ability measured by this
benchmark as
Egocentric Action Selection (EAS): selecting an appropriate action from the
agent's perspective in the presence of other
agents. On EgoGapBench, humans answer reliably, whereas both open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible
agents. Fine-tuning on existing
egocentric data fails to close this gap and can even be detrimental. In contrast, fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance. These results show that EAS is difficult to acquire from first-person-view data alone, and that MLLMs should be evaluated and trained not only for scene understanding but also for
egocentric action selection.
Read the full abstract ▾
All authors Jihyeok Jung, Jeewu Lee, Sanghyeop Kim, Chanhee Han, Seong Joon Oh
Computer Vision
ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation
Yu-Hsiang Chen, Wei-Jer Chang, Yi-Ting Chen et al. (+1 more)
Controllable traffic simulation is critical for testing autonomous driving systems, yet existing approaches often require retraining large generative models with extensive annotated data. We introduce a lightweight control adaptation framework that enables multi-modal controllability (sketch, latent behavior codes, and text) for pretrained state-of-the-art
diffusion and autoregressive traffic models. By modulating intermediate features through identity-initialized FiLM layers, our method efficiently adds new control modalities while preserving the base model's generative prior. Evaluated on Waymo Open Sim
Agents Challenge, our approach demonstrates strong controllability with less than 1% of the paired control data. Through context-aware condition transfer, our framework enables counterfactual scenario generation and long-tail synthesis while maintaining stable closed-loop driving realism and safety. Our framework unlocks new possibilities for controllable traffic simulation, enabling targeted scenario generation through lightweight adaptation of pretrained generative models. Project page: https://ecosim-web.github.io/
Read the full abstract ▾
All authors Yu-Hsiang Chen, Wei-Jer Chang, Yi-Ting Chen, Masayoshi Tomizuka
Computer Vision
GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Engine
Yanan Wang, Wen Li, Yibin Ying et al. (+1 more)
Reasoning segmentation requires localizing targets based on complex, implicit queries. Current end-to-end models typically entangle perception and deduction into an opaque black box, severely limiting interpretability and scalability. To address this, we propose GEAR-Seg (Grounded Explainable
Agent for
Reasoning Segmentation), an explicitly decoupled
agent that shifts the paradigm by translating visual pixels into dense, attribute-rich text. By decoupling class-agnostic
segmentation, semantic description, and Large
Language Model (LLM) deduction, GEAR-Seg transforms implicit
reasoning into an explicit, trackable logic chain. As a zero-shot inference framework, it achieves highly competitive performance across diverse
reasoning and fine-grained referring
segmentation benchmarks. Furthermore, GEAR-Seg inherently functions as a highly scalable data engine. Utilizing this engine, we construct GEAR-131K, a massive
benchmark (over 38k images, 656k QA-mask pairs) introducing a multifaceted taxonomy tailored for complex real-world manipulation-oriented
reasoning. Finally,
distillation experiments demonstrate that lightweight models supervised exclusively by our automated pipeline closely match the upper-bound performance of costly human-annotated baselines.
Read the full abstract ▾
All authors Yanan Wang, Wen Li, Yibin Ying, Zhenghao Fei
Computer Vision · Cross-listed from cs.LG
Flow-Map GRPO: Reinforcement Learning for Few-Step Flow-Map Generators via Anchored Stochastic Composition
Zhiqi Li, Wen Zhang and Bo Zhu
Few-step flow-map generators, such as consistency models and MeanFlow, accelerate sampling by learning long-range transport maps between noise and data. However, their deterministic transitions do not directly provide the
stochastic trajectories and tractable likelihood ratios required by reinforcement learning (RL) post-training. Existing SDE-based stochasticization techniques target velocity-based samplers and do not directly extend to long-range flow-map transitions. We propose Flow-Map GRPO, an online RL post-training framework for deterministic few-step flow-map generators. Its key component, Anchored
Stochastic Flow Map Composition (ASFMC), combines deterministic transport with anchor-based conditional resampling. We establish the conditions under which this construction preserves the marginal probability path and develop tractable local- and endpoint-anchor policies for two-time and single-time flow maps. These policies enable a unified GRPO training procedure. Experiments on FLUX-based MeanFlow and sCM generators demonstrate substantial improvements in OCR, PickScore, and GenEval at different numbers of inference steps, including joint OCR–PickScore gains with mixed rewards. Controlled ablations show that the
stochastic transition design is essential for translating training rewards into generation quality. Flow-Map GRPO enables effective RL
alignment of pretrained deterministic flow-map generators while retaining their original parameterization, without retraining them as native
stochastic models.
Read the full abstract ▾
Computer Vision
NoPA: Non-Parametric Online 3D Scene Graph Generation
Qi Xun Yeo, Seungjun Lee, Yan Li et al. (+1 more)
Classic 3D scene graph generation approaches fail to work in real-time due to the heavy computational cost of environment mapping and the need to generate intermediate point-cloud representations. To alleviate this issue, a recent work eschews point clouds in favor of a lightweight Gaussian distribution for each object. This approximation drastically speeds up inference and enables real-time 3D scene graph generation. However, the representation has two key weaknesses. 1) Each object is approximated by a single 3D Gaussian, which causes a severe loss of 3D
geometric detail. 2) The discrepancy between this approximation and the true object geometry exacerbates the inaccurate merging of object candidates during online inference. To address these issues, we propose NoPA, which represents each object as a separate non-parametric distribution. This formulation retains 3D
geometric information while preserving real-time inference of the parametric Gaussian formulation. To build upon our novel object representation, we propose a tailored merging strategy to recover coherent object instances. Specifically, we leverage maximum mean discrepancy on kernel density estimates to enable robust merging of object candidates during online exploration while minimizing added computational complexity. The key is to maintain a fixed particle set per object. Furthermore, to rectify the relation loss caused by misclassified objects, NoPA propagates relationships between objects with high affinity. Experiments show that NoPA substantially outperforms current methods without sacrificing real-time inference speed.
Read the full abstract ▾
All authors Qi Xun Yeo, Seungjun Lee, Yan Li, Gim Hee Lee
Computer Vision
SPECSIA: Stylization Dataset for Novel-View Enhancement in Drawing-based 3D Animation
Kyuwon Kim, Sunjae Yoon and Chang D. Yoo
Generating animation from a single 2D drawing is challenging because the output must preserve character appearance while remaining plausible and temporally coherent under motion. Existing drawing-based 3D animation pipelines often use sample-wise 2D refinement to align animated renderings with the input image, but such optimization tends to overfit to the observed view and fails to correct projection-induced artifacts in novel views. To address this limitation, we introduce SPECSIA-15K, a paired stylization
dataset containing 14,980 artifact-corrupted projection/refinement-target pairs from 1,498 3DBiCar characters. We further present DraViE (Drawing-based View Enhancement), a lightweight plug-and-play module trained with data-level priors to remove novel-view artifacts while preserving style and motion plausibility. Experiments show consistent gains in novel-view fidelity and temporal coherence with lower per-character adaptation cost than sample-wise fine-tuning.
Read the full abstract ▾
Computer Vision
Restore3D: Breathing Life into Broken Objects with Shape and Texture Restoration
Xiaolong Shen, Zongxin Yang and Yi Yang
Restoring incomplete or damaged 3D objects is crucial for cultural heritage preservation, occluded object reconstruction, and artistic design. Existing methods primarily focus on
geometric completion, often neglecting texture restoration and struggling with relatively complex and diverse objects. We introduce Restore3D, a novel framework that simultaneously restores both the shape and texture of broken objects using multi-view images. To address limited training data, we develop an automated data generation pipeline that synthesizes paired incomplete-complete samples from large-scale 3D
datasets. Central to Restore3D is a multi-view model, enhanced by a carefully designed Mask Self-Perceiver module with a Depth-Aware Mask Rectifier. The rectified masks learned by the self-perceiver guide an image integration and enhancement phase, helping retain observed shape and texture patterns while refining the generated regions and mitigating the low-resolution limitations of the base model, yielding high-resolution, semantically coherent, and view-consistent multi-view images. A coarse-to-fine reconstruction strategy is then employed to recover detailed textured 3D meshes from refined multi-view images. Experiments on synthetic and real broken-object
benchmarks show that Restore3D improves multi-view restoration quality and textured-mesh reconstruction over representative inpainting, completion, and reconstruction baselines in the evaluated settings. Project Page: restore3dx.github.io
Read the full abstract ▾
Computer Vision
Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning
Trung Thanh Nguyen, Hai Nguyen-Truong, Tu Vo et al. (+2 more)
Automatic understanding of dynamic 4D point clouds, the 3D-point sequences captured over time by depth sensors and
LiDAR, is central to robotics and
embodied perception. Yet annotating them densely is expensive, making self-supervised pretraining the natural route to transferable representations. Existing pretext tasks, however, are almost entirely intra-modal, and the few methods that transfer knowledge from 2D foundation models rely on a single global
embedding per clip, discarding the rich per-patch semantics that these models compute. To address this gap, we propose Cross4D-JEPA, a teacher-student method that distills a frozen 2D foundation model, an image model DINOv2, or a
video model V-JEPA 2, into a 4D point encoder. The proposed method combines (1) a dense cross-modal correspondence that maps every 3D point to the teacher patch feature it projects to, and (2) a per-point objective that trains the student to match these features in latent space with no masking, negatives, or decoder. We evaluate Cross4D-JEPA on four
benchmarks, MSR-Action3D, DeformingThings4D, NTU-RGB+D 60, and HOI4D, against intra-modal and global cross-modal baselines. Experimental results show that, under a matched protocol, the proposed method consistently outperforms intra-modal and global cross-modal baselines across the four
benchmarks and is competitive with heavier published 4D methods; further analysis attributes this gain primarily to the granularity of the correspondence rather than the teacher modality. Beyond recognition accuracy, the dense representation learned by Cross4D-JEPA transfers across domains, improves label efficiency, and improves full-label fine-tuning under the same training budget, while a 13x smaller encoder matches a heavyweight pooling backbone.
Read the full abstract ▾
All authors Trung Thanh Nguyen, Hai Nguyen-Truong, Tu Vo, Hoang M. Truong, Tuan-Anh Vu
Computer Vision
AnF-DiffPET: Anatomy- and Frequency-Guided Diffusion for PET/CT Denoising
Xuepeng Liu, Ruili Li, Zetong Liu et al. (+5 more)
Positron emission
tomography (PET) provides essential functional information for disease assessment, however reducing injected activity or acquisition time produces low-dose (LD) PET with stronger count dependent noise and less reliable uptake quantification.
Diffusion models offer a promising solution for PET denoising by progressively recovering high-dose (HD) PET images from LD inputs. However, LD-to-HD PET denoising is still challenging due to insufficient anatomical guidance, unstable multi-scale feature propagation, and uncertain frequency domain uptake recovery. We propose AnF-DiffPET, an anatomy- and frequency-guided
diffusion framework for computed
tomography (CT) conditioned LD PET denoising. The framework integrates Anatomical-Frequency Guidance (AFG), Multi-Scale Cross-
Transformer Reconstruction (MSCTR), and Frequency-Contrastive Hard Mining (FCHM) to enhance anatomy aware feature modulation and frequency domain consistency during denoising. Experimental results across four PET/CT
datasets show that the proposed method improves image fidelity, anatomical consistency, and quantitative fidelity over representative CNN-based, GAN-based,
transformer-based, and
diffusion-based methods. The code and trained models will be publicly released upon acceptance.
Read the full abstract ▾
All authors Xuepeng Liu, Ruili Li, Zetong Liu, Renyiming Li, Yan Li, Yin Dai, Chao Li, Yueyang Teng
Computer Vision · Cross-listed from physics.med-ph
Closed-loop coupling of personalised and foundation models for real-time treatment guidance with MRI
James Grover, Emily A. Hewson, Andrew Phair et al. (+5 more)
Image-guided therapies, including radiotherapy, biopsy and deep
brain stimulation, rely on real-time targeting of anatomical structures. However, in the presence of motion, imaging latencies create a temporal misalignment between observed and true anatomy, compromising treatment accuracy. Artificial intelligence-based frameworks have increasingly been presented to close this latency gap, but leading personalised models can fail due to a lack of stable anatomical grounding. Foundation models can provide grounded behaviour, but they do not adapt to real-time, individual patient dynamics. Here we introduce a closed-loop coupling framework that synergises patient-specific temporal prediction with continuous
segmentation-based anatomical interpretation from a foundation model. A personalised model predicts future anatomy to compensate for system latency, while a streaming foundation model provides anatomical supervision used to continuously update the temporal predictor in real time during treatment. We validate the framework using a digital phantom and intrafraction
magnetic resonance imaging (MRI) from patients undergoing MRI-guided radiotherapy. For a prediction horizon of 400 ms, the proposed method improves anatomical prediction and reduces dosimetric error compared with existing approaches, within clinically relevant latency constraints. These results establish closed-loop coupling as a general strategy for real-time image-guided intervention.
Read the full abstract ▾
All authors James Grover, Emily A. Hewson, Andrew Phair, Michael Ferraro, Hilary L. Byrne, Paul Keall, Michael G. Jameson, David E.J. Waddington
Computer Vision
Prior-Anchored Debiasing for Long-Tailed Multi-Organ Pathology Report Generation
Feng Yang, Jie Liu, Yubo Pang et al. (+5 more)
Automated pathology report generation from Whole Slide Images (WSIs) has attracted increasing
attention in digital pathology. However, existing methods are predominantly developed under single-organ settings, overlooking the multi-organ scenarios encountered in clinical practice, where organ types typically follow a long-tailed distribution. To address this gap, we identify two critical biases: (1) visual representation bias, where the encoder favors head-class patterns over tail-class discriminative features, and (2) textual decoding bias, where the decoder overfits to head-class narrative patterns, yielding diagnostically unreliable outputs for tail-class organs. To mitigate these two biases, we propose a novel Prior-anchored multi-Organ pathology report Generation framework (PriOrGen). Specifically, a Visual-Prototype Anchored Bottleneck module leverages the information bottleneck principle with learnable anchor representations to selectively retain diagnostically relevant visual information while filtering out head-biased redundancy. Secondly, a Meta-Report Anchored Bank module constructs an organ-specific meta-report anchored bank and retrieves organ-faithful textual priors to steer the decoder away from head-class narrative patterns. Extensive experiments on a multi-organ pathology
dataset demonstrate that our method effectively mitigates long-tail biases and achieves superior report generation performance across both head and tail organ categories compared to state-of-the-art methods.
Read the full abstract ▾
All authors Feng Yang, Jie Liu, Yubo Pang, Peilin Chen, Xinheng Lyu, Shiqi Wang, Howard Leung, Ping Chen