Computer Vision
Learning a Particle Dynamics Model with Real-world Videos
Chanho Kim, Suhas V. Sumukh and Li Fuxin
Data-driven learning approaches for physics simulation, sometimes referred to as world models, have emerged as promising alternatives to traditional physics simulators due to their differentiable nature. Prior work has demonstrated impressive results in predicting the motions of rigid and non-rigid objects in complex scenes involving multiple interacting bodies. However, these models are typically trained in simulated environments because obtaining perfect state information such as complete scene point clouds and point correspondences over time is challenging in real-world settings. This reliance on synthetic data can limit their applicability when the sim-to-real gap is large. In this work, we aim to overcome these limitations by introducing a novel framework for training neural object dynamics models directly from unlabeled real-world
videos. Specifically, we propose to learn a particle-based dynamics model compatible with a Gaussian splatting framework, which operates on dense particles derived from Gaussians (i.e., particles with scales and rotations) and predicts their position and rotation changes over time. The model is trained via rendering supervision, enabling learning from real-world
videos without requiring particle-level labeled states. Our model operates directly on dense Gaussians without relying on heuristic subsampling anchor points. To enable this study, we also present a real-world
dataset consisting of about 500
videos capturing diverse object interactions.
Read the full abstract ▾
Computer Vision
MuellerPT: Decomposition Driven Pretraining for Dense Learning in Mueller Polarimetry
Adam Tlemsani, Yingdian Li, Maxime Giot et al. (+4 more)
Mueller
matrix imaging provides rich, physically meaningful contrast for biomedical tissue analysis, but supervised learning is hindered by scarce dense annotations and strong domain shifts across specimens and acquisition settings. We introduce MuellerPT, a physics guided pre-training approach that learns transferable dense representations by predicting Lu-Chipman decomposition maps from per-pixel 4x4 Mueller
matrices. To scale pre-training, we collected a new large Multispectral Animal Polarimetric Organ
dataset (MAP-Org). The pre-trained encoder is adapted with a
segmentation head for grey vs. white matter
segmentation in lamb
brain. A classification head is used for colorectal cancer vs. non-cancer classification. Both
segmentation and classification are evaluated across few-shot learning scenarios. In
segmentation, MuellerPT improves label efficiency and cross specimen transfer compared to models without pre-training, achieving an absolute DICE gain of over 20% compared to the baseline trained from scratch when using 5% of the training data. In classification, MuellerPT also enhances label efficiency, improving overall accuracy by 8% compared to the baseline when using 1% of the training data. We demonstrate MuellerPT's robustness to domain shift with a qualitative evaluation of its predicted Lu-Chipman maps on an ex vivo human oesophagus sample. These results suggest that predicting Lu-Chipman decomposition is an effective and practical pretext task for robust biomedical inference from Mueller polarimetry and can pave the way for future work on label efficient Mueller imaging.
Read the full abstract ▾
All authors Adam Tlemsani, Yingdian Li, Maxime Giot, Naim Slim, Christopher J. Peters, Abhijeet Ghosh, Daniel S. Elson
Computer Vision
Decomposing Queries into Tool Calls for Long-Video Keyframe Retrieval
Michal Shlapentokh-Rothman, Prachi Garg, Yu-Xiong Wang et al. (+1 more)
Keyframe selection is a direct way to provide verifiable visual evidence for long-
video question answering (QA). Queries differ in what they require, and finding the right frames depends on knowing what to look for. Existing keyframe selectors either score every frame against a single query, or decompose the query into a fixed schema evaluated by a single visual tool. We propose ToolMerge, a keyframe
retrieval method based on decomposition and merging: an Large
Language Model (LLM) based planner decomposes the query into tool calls and specifies how their per-tool rankings are merged using boolean operators. To evaluate
retrieval directly, we construct Molmo-2 Moments (M2M), a
benchmark in which every question is anchored to a specific time interval by construction. Across QA, question
retrieval, and caption
retrieval, ToolMerge is competitive with prior keyframe selectors, most notably on caption
retrieval, outperforming other methods by 5%. Code and data can be found at https://github.com/michalsr/ToolMerge .
Read the full abstract ▾
All authors Michal Shlapentokh-Rothman, Prachi Garg, Yu-Xiong Wang, Derek Hoiem
Computer Vision
Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot
Jorge Chang Ortega, Bastien Le Lan, Thomas Serre et al. (+1 more)
A central question in computational
vision is whether human-like visual representations are better explained by discriminative or generative learning. Existing comparisons, however, often confound the learning objective with architecture, scale, and training data, leaving open whether the objective itself drives
alignment. We address this confound using Joint Energy-Based Models (JEMs), which interpolate continuously between discriminative and generative training within a fixed architecture. By varying a single mixing coefficient, we isolate the effect of the learning objective and evaluate the resulting models across six human-
alignment benchmarks spanning perceptual similarity, gloss perception, human response uncertainty, robustness, shape-texture cue conflict, and diagnostic feature attribution. Across this diverse suite, human
alignment is consistently maximized at intermediate points of the generative-discriminative continuum, rather than at either endpoint. Hybrid JEMs combine the categorical structure induced by discriminative learning with the sensitivity to input structure induced by generative learning, yielding more human-like behavior across multiple levels of
vision. These results suggest that the generative-discriminative dichotomy is the wrong axis for understanding human-aligned
vision:
alignment emerges not from choosing one objective over the other, but from balancing both.
Read the full abstract ▾
All authors Jorge Chang Ortega, Bastien Le Lan, Thomas Serre, Victor Boutin
Computer Vision · Cross-listed from cs.LG
Debiased Negative Mining Improves Out-of-distribution Detection with Pre-trained Vision-Language Models
Bo Peng, Jie Lu, Guangquan Zhang et al. (+1 more)
Aiming at identifying unexpected inputs from unknown classes, out-of-distribution (OOD) detection has emerged as a pivotal approach to enhancing the reliability of machine learning models. This paper focuses on the burgeoning paradigm of post-hoc OOD detection with pre-trained
vision-
language models (VLMs), where a popular pipeline is to detect OOD inputs by examining their affinities between ID labels and negative labels, i.e., those semantically different from ID labels. Due to the unavailability of target OOD labels, existing works predominantly rely on heuristic rules to mine negative labels from unlabeled wild corpus data. Despite the empirical success, we argue that the power of VLM-based OOD detection has yet to be fully unleashed since the notorious false negative problem is far from addressed in the literature. With this motivation, we are interested in addressing the challenge of mining true negative labels for OOD scoring. To this end, we develop a theoretical framework for correcting the sampling bias of negatives labels by indirectly approximating the distribution of negative labels. Perhaps surprisingly, we show that the debiased negative mining can be naturally converted into Monte-Carlo sampling based on ID labels and the unlabeled wild corpus data. Extensive experiments empirically manifest that our method establishes a new state-of-the-art in a variety of OOD detection setups. Code is publicly available at \href{https://github.com/60pen9/Debiased-Negative-Mining-Improves-OOD-Detection-with-Pre-trained-VLMs}{\textcolor{red}{here}}.
Read the full abstract ▾
All authors Bo Peng, Jie Lu, Guangquan Zhang, Zhen Fang
Computer Vision
Exploring deep learning for Event-Based Saliency Prediction with a Transformer-based model
Romaric Mazna, Jean Martinet and Sai Deepesh Pokala
Saliency prediction has been extensively studied in RGB images and
videos as a computational model of human visual
attention. In contrast, predicting saliency from event-based data remains largely unexplored, despite the biological inspiration and favorable sensing properties of event cameras. Two obstacles have held this direction back: the absence of large-scale event saliency
datasets, and the lack of a strong baseline. In this paper, we introduce SEST (Swin Event-based Saliency
Transformer), a
transformer-based model for saliency prediction from event data, bridging the data scarcity barrier through event-native pretraining and synthetic supervision. SEST leverages a self-supervised pretrained event-based Swin
Transformer backbone combined with a lightweight CNN decoder to produce dynamic saliency maps. To address the scarcity of annotated event-based saliency data, we introduce two new
benchmark datasets, N-DHF1K and N-UCF Sports, generated from large-scale RGB saliency
benchmarks. Experimental results show that SEST clearly outperforms existing event-based saliency methods and narrows the performance gap with state-of-the-art RGB models. Zero-shot evaluation on a real event camera
dataset further demonstrates that our model trained on synthetic data remains transferable on real event streams. To the best of our knowledge, this work is the first to apply deep learning to event-based saliency prediction, opening a new research direction at the intersection of event-based
vision and neuromorphic visual
attention.
Read the full abstract ▾
Computer Vision
Machine learning applied to emerald gemstone grading: framework proposal and creation of a public dataset
FB Pena, D Crabi, Sandro C Izidoro et al. (+2 more)
The grading of gemstones is currently a manual procedure performed by gemologists. A popular approach uses reference stones, where those are visually inspected by specialists that decide which one of the available reference stone is the most similar to the inspected stone. This procedure is very subjective as different specialists may end up with different grading choices. This work proposes a complete framework that entails the image acquisition and goes up to the final stone categorization. The proposal is able to automate the entire process apart from including the stone in the created chamber for the image acquisition. It discards the subjective decisions made by specialists. This is the first work to propose a machine learning approach coupled with image processing techniques for emerald grading. The proposed framework achieves 98% of accuracy (correctly categorized stones), outperforming a deep learning approach. Furthermore, we also create and publish the used
dataset that contains 192 images of emerald stones along with their extracted and pre-processed features.
Read the full abstract ▾
All authors FB Pena, D Crabi, Sandro C Izidoro, Érick O Rodrigues, G Bernardes
Computer Vision
A Novel Approach for the Counting of Wood Logs Using cGANs and Image Processing Techniques
João VC Mazzochin, Giovani Bernardes Vitor, Gustavo Tiecker et al. (+4 more)
This study tackles the challenge of precise wood log counting, where applications of the proposed methodology can span from automated approaches for materials management, surveillance, and safety science to wood traffic monitoring, wood volume estimation, and others. We introduce an approach leveraging Conditional Generative
Adversarial Networks (cGANs) for eucalyptus log
segmentation in images, incorporating specialized image processing techniques to handle noise and intersections, coupled with the Connected Components
Algorithm for efficient counting. To support this research, we created and made publicly available a comprehensive database of 466 images containing approximately 13,048 eucalyptus logs, which served for both training and validation purposes. Our method demonstrated robust performance, achieving an average Accuracy_pixel of 96.4% and Accuracy_logs of 92.3%, with additional measures such as F1 scores ranging from 0.879 to 0.933 and IoU values between 0.784 and 0.875, further validating its effectiveness. The implementation proves to be efficient with an average processing time of 0.713s per image on an NVIDIA T4 GPU, making it suitable for realtime applications. The practical implications of this method are significant for operational forestry, enabling more accurate inventory management, reducing human errors in manual counting, and optimizing resource allocation. Furthermore, the
segmentation capabilities of the model provide a foundation for advanced applications such as eucalyptus stack volume estimation, contributing to a more comprehensive and refined analysis of forestry operations. The methodology's success in handling complex scenarios, including intersecting logs and varying environmental conditions, positions it as a valuable tool for practical applications across related industrial sectors.
Read the full abstract ▾
All authors João VC Mazzochin, Giovani Bernardes Vitor, Gustavo Tiecker, Elioenai MF Diniz, Gilson A Oliveira, Marcelo Trentin, Érick O Rodrigues
Computer Vision
PhotoFlow: Agentic 3D Virtual Photography Missions
Jiarui Guo, Haojia Wei, Yiming Zhang et al. (+5 more)
Virtual photography asks an
agent to enter a prepared 3D scene with no preselected camera pose or reference image, infer a suitable shot from scene information and a
language intent, choose executable camera parameters, and render the final photograph. Recent progress in
vision-
language models makes this kind of spatial
agent increasingly plausible, but the task stresses two capabilities that remain hard to evaluate together: complex 3D spatial understanding and abstract aesthetic judgment. We introduce PhotoFlow, a Director-Reviewer-Reflector
agent for closed-loop camera search. The Director builds a soft photographic blueprint and proposes diverse candidate cameras; the Reviewer combines rule checks, visual critique, and pairwise incumbent selection; and the Reflector converts failures into region
memory, dead-zone suppression, and high-explore relocation. We also introduce VPhotoBench, a
benchmark of 47 open-license Blender scenes and 141
language-conditioned photography missions spanning subject placement, relational composition, and atmosphere/style. On held-out experiments, PhotoFlow achieves the strongest external quality-
alignment composite and success rate among one-shot prediction, single-chain reflection, anchor-bank selection, and random search under a six-round rendering budget. To our knowledge, this is the first work to make
language-conditioned virtual photography in arbitrary Blender scenes an executable
agent task, and our results show that an LLM-centered spatial
agent can already produce strong photographs in a setting designed to challenge both 3D
reasoning and aesthetic choice.
Read the full abstract ▾
All authors Jiarui Guo, Haojia Wei, Yiming Zhang, Yifei Liu, Yuning Gong, Hongjie Zhang, Xue Yang, Zhihang Zhong
Computer Vision
Revitalizing Dense Material Segmentation: Stabilized Vision Transformers and the Generalization Paradox
Allan Kazakov, Duygu Cakir, Hilal Kurt İrfanoğlu et al. (+1 more)
Material
segmentation, the pixel-wise classification of physical surface properties, remains a challenging problem in computer
vision, requiring physicochemical understanding distinct from object-centric parsing. Despite the introduction of the rigorous Apple Dense Material
Segmentation (DMS)
dataset, the
benchmark has suffered from attrition and stagnation, increasingly overshadowed by geometry-biased foundation models. In this paper, we revive the Apple-DMS
benchmark to establish a modern
Vision Transformer baseline. We conduct an exhaustive evaluation of SegFormer and Mask2Former architectures, revealing that standard training paradigms fail on amorphous texture fields due to high-variance gradients. To address this, we introduce a stabilized training recipe featuring High-Fidelity Logit Projection, Query
Entropy Regularization, and a domain-specific, physics-compliant augmentation pipeline. Our optimized SegFormer-B5 achieves a new State-of-the-Art (SOTA) of 0.4572 mIoU on the original
dataset split, significantly surpassing the prior convolutional baseline. Furthermore, we identify a critical "Generalization Paradox": while re-partitioning the
dataset into a data-rich 80/10/10 split inflates the metric to 0.5276 mIoU, expert qualitative analysis reveals this induces distributional homogenization, severely degrading real-world, out-of-distribution performance. By releasing our recovered
dataset index and robust training framework, we demonstrate that material perception is far from solved and urge the community to leverage the rigorous original split to drive genuine progress in physically grounded artificial intelligence.
Read the full abstract ▾
All authors Allan Kazakov, Duygu Cakir, Hilal Kurt İrfanoğlu, Yavuz İrfanoğlu
Computer Vision
CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models
León Begiristain, Olaf Dünkel and Adam Kortylewski
Video prediction is increasingly viewed as a path toward generalizable world models, yet it remains unclear whether these systems learn underlying causal structure or merely exploit superficial visual correlations for future prediction. We introduce CRONOS, an intervention-based
benchmark to evaluate counterfactual physical consistency: whether a model's predictions of physical events respond appropriately to controlled changes in the visual input, such as variations of camera viewpoint, scene, object category, and object appearance. Built in a photorealistic Unreal Engine environment, CRONOS enables controlled, high-fidelity generation of
videos across diverse scenes and dynamics, including sampling multiple physically plausible outcomes of the same physical event. In contrast to previous
benchmarks, that evaluate
video prediction on scenes CRONOS systematically intervenes on four factors -camera viewpoint, scene, object category, and object appearance- while keeping the underlying physical event type, such as a fall, collision, or occlusion, fixed. Our evaluation of recent open-source
video generators reveals substantial failures in counterfactual physical consistency: prediction quality for the same physical event type is affected by appearance, environment, and particularly viewpoint changes. CRONOS provides a controlled and reproducible testbed for diagnosing how generated
video quality changes across interventions, establishing a concrete target for developing models that perform consistently across changes of multiple conditions. The
dataset and evaluation code are available at our project page.
Read the full abstract ▾
Computer Vision
RiGS: Rigid-aware 4D Gaussian Splatting from a Single Monocular Video
Chenyu Wu, Wanhua Li, Zhu-Tian Chen et al. (+1 more)
Reconstructing dynamic 3D scenes from
monocular videos is a fundamental yet highly challenging task, as real-world motions often involve both long-term smooth transformations and short-term complex deformations. Existing methods either struggle to maintain temporal consistency or fail to capture high-frequency dynamics due to limited motion modeling capacity. In this work, we present Rigid-aware 4D Gaussian Splatting (RiGS), which simultaneously captures motions across multiple temporal scales. Specifically, RiGS introduces three types of Gaussian primitives: static, rigid, and transient, which represent static backgrounds, long-term low-frequency motions, and short-term high-frequency dynamics, respectively. An object-wise dynamic mask is proposed to aggregate long-range spatiotemporal motion information and guide the decomposition of static and dynamic regions. To jointly model motion across scales, rigid Gaussians are allowed to transition into transient Gaussians based on their temporal duration, and both are optimized under scene flow guidance, providing dense 3D motion supervision. Extensive experiments demonstrate that RiGS achieves state-of-the-art performance on novel view synthesis
benchmarks. Code is available at \hyperlink{https://github.com/ladvu/RiGS}{https://github.com/ladvu/RiGS}.
Read the full abstract ▾
All authors Chenyu Wu, Wanhua Li, Zhu-Tian Chen, Hanspeter Pfister
Computer Vision
Recursive Block-Diagonal Coupling for Resource-Efficient Training of Vision Models
Maxim Henry, Adrien Deliège, Sébastien Piérard et al. (+1 more)
Training high-capacity
vision models from scratch requires substantial computational resources. To improve training efficiency of a wide target model, existing growth methods often assume the availability of narrower models, obscuring the true computational cost of the entire pipeline. We propose an efficient training protocol, RBDC, that builds wide models by coupling in a parameter-free block-diagonal way narrower, independently trained models in a recursive way. This allows a flexible allocation of the training budget available across all the models involved. Evaluated with
vision transformers (DeiT) and convolutional networks (ResNet) on ImageNet, our RBDC training protocol shows a much better efficiency than models trained from scratch with the standard protocol, yielding 30% FLOPs reduction at similar test accuracies. It also achieves higher performances at same training FLOPs than training protocols from the model growth literature. Finally, we show that our models can serve as better backbones than their original counterparts for downstream object detection and instance
segmentation tasks.
Read the full abstract ▾
All authors Maxim Henry, Adrien Deliège, Sébastien Piérard, Marc Van Droogenbroeck
Computer Vision
CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception
Liupeng Li, Haoqian Kang, Zhenyu Lu et al. (+4 more)
High-resolution (HR) image perception presents a key bottleneck for
multimodal large
language models (MLLMs). While visual search offers a promising solution, existing methods struggle with the trade-off between coverage and efficiency. Visual expert-assisted search is efficient but prone to blind spots when proposals fail, whereas scan-based search guarantees coverage at the cost of computational redundancy and semantic fragmentation. To address this dilemma, we introduce CVSearch, a training-free adaptive framework that dynamically schedules search strategies via an Assess-then-Search workflow. Specifically, CVSearch first invokes expert-assisted search when global information is insufficient, and only triggers a novel semantic-aware scanning mechanism upon failure. Distinct from rigid grid partitioning, this efficient scanning paradigm incorporates Semantic Guided Adaptive Patching to decompose images into semantically consistent regions, effectively mitigating object fragmentation. Furthermore, we devise a Dynamic Bottom-Up Search strategy driven by a Visual Complexity prior to enable efficient and precise iterative exploration of local details. Extensive experiments on HR
benchmarks demonstrate that CVSearch achieves state-of-the-art accuracy while substantially improving search efficiency. Code is released at https://github.com/liliupeng28/ICML26-CVSearch.
Read the full abstract ▾
All authors Liupeng Li, Haoqian Kang, Zhenyu Lu, Jinpeng Wang, Bin Chen, Ke Chen, Yaowei Wang
Computer Vision
ExpOS: Explainable Open-Surgery Skills Assessment Using 3D Hand Reconstruction
Roi Papo, Idan Smoller and Shlomi Laufer
Timely and transparent feedback is essential for effective
surgical training, yet current assessment remains dependent on expert observation, limiting scalability and opportunities for autonomous practice. We present ExpOS, an explainable framework for data-driven assessment of open-surgery skills designed to enable automatic, feedback-oriented evaluation. Rather than relying on expert-defined metrics, ExpOS learns discriminative temporal patterns directly from motion data and identifies the segments and behaviors most predictive of skill level. We trained and evaluated the method on 221
videos of medical students performing three open-surgery tasks. Hand poses and tool detections were extracted from each frame to derive kinematic descriptors and global motion statistics. Spatiotemporal hand-tool dynamics were modeled using a temporal convolutional backbone with
attention-based pooling to generate frame-level importance maps. These representations were fused with global motion statistics to predict skill level and to provide interpretable feedback. ExpOS provides multi-level explainability by identifying when informative events occur through
attention weights and which motion characteristics most influence predictions through global feature analysis. Across tasks, the framework achieved strong correlation with expert ratings, with best performance on fascial closure (r = 0.778, R2 = 0.74). These results demonstrate that combining weakly-supervised temporal importance learning with interpretable motion statistics enables scalable and actionable
surgical skill assessment.
Read the full abstract ▾
Computer Vision
DualMem: Bypassing the Objectness Bottleneck for Calibrated Unknown-Stream Filtering in Open-World Object Detection
Yingjun Xiao, Xi Chen, Gang Fang et al. (+1 more)
Open-world object detection (OWOD) requires detectors to localize known classes while identifying unknown objects for future incremental learning. We find that the unknown prediction streams of strong OWOD detectors are heavily polluted: on M-OWODB, across PROB, OW-DETR, and HypOW, future-task positive unknowns make up less than 10% of unknown predictions, whereas background false positives account for 46-71%. We show that this is not a missing-information problem, but an information bottleneck at the objectness head. On PROB Task 1, a linear probe on the 256-D decoder query achieves an AUROC of 0.908 for positive-versus-negative unknown discrimination, but the final one-dimensional objectness scalar drops to 0.642. A frozen SigLIP feature, without access to the detector, independently recovers much of this proposal-level separability at the filtering stage (AUROC = 0.871). Motivated by this finding, we propose DualMem, a calibrated post-hoc filter that assumes a small image-disjoint annotated calibration split of held-out future-task objects and performs a non-parametric likelihood ratio test in frozen SigLIP feature space. DualMem uses a k-nearest-neighbor positive
memory to protect future-task objects and a negative
memory to suppress background-like proposals. Its decision threshold is chosen by Neyman-Pearson calibration, giving users an explicit trade-off between false-unknown suppression and novel recall. Across PROB, OW-DETR, and HypOW on M-OWODB Task 1, DualMem reduces background-type false unknown proposals per image by 44.9%-66.3%, with a mean reduction of 56.6%. On PROB Task 1, it more than doubles the reduction achieved by a natural K-means prototype baseline, while leaving known-class mAP unchanged because known detections bypass the filter.
Read the full abstract ▾
All authors Yingjun Xiao, Xi Chen, Gang Fang, Siyuan Chen
Computer Vision
DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs
Jiazhen Pan, Weixiang Shen, Jun Li et al. (+7 more)
Medical diagnosis is not a single prediction from a fully specified vignette. It is a sequential workup: clinicians decide what evidence to obtain, revise a differential diagnosis, and stop when the diagnosis is sufficiently supported. Most medical AI
benchmarks instead reveal the relevant context upfront and score only the final answer, making unsupported correct guesses, premature closure, inefficient workups, and poor uncertainty updating invisible. We introduce DDX-TRACE, a physician-adjudicated
benchmark for
multimodal neuroradiology that evaluates diagnostic trajectories under hidden evidence over 211 challenging cases. Each case begins with limited clinical history; models request imaging studies in free form, receive matched image bundles when available, update a probabilistic differential diagnosis after each turn, and stop with a localized final diagnosis. Evaluating state-of-the-art VLMs, we find that final diagnosis scores can substantially misrepresent workup quality: models may guess plausible diagnoses without essential evidence, request useful studies but misinterpret raw images, or acquire evidence inefficiently while updating uncertainty poorly. Controlled evidence variants isolate bottlenecks in planning, visual evidence extraction, and downstream differential
reasoning. DDX-TRACE shifts medical AI evaluation from final answers to evidence-supported diagnostic trajectories.
Read the full abstract ▾
All authors Jiazhen Pan, Weixiang Shen, Jun Li, Julian Canisius, Felix Bitzer, Paula Roßmüller, Jiancheng Yang, Virginie Kreutzinger, Daniel Rueckert, Benedikt Wiestler
Computer Vision
EM-Vid: Training-Free Entity-Centric Memory for Efficient and Consistent Multi-Shot Video Generation
Jente Vandersanden, Matheus Gadelha, Chun-Hao P. Huang et al. (+2 more)
Multi-shot
video generation requires maintaining a consistent appearance of recurring entities across shots while remaining faithful to shot-specific text
prompts. Recent autoregressive methods reuse previously generated frames as
memory. However, full-frame storage entangles persistent entity information with transient scene context, leading to irrelevant information leakage and high computational cost. We propose an entity-centric
memory in the form of an entity-indexed bank of latent patches. We introduce sparse
token conditioning compatible with pretrained models, restricting self-
attention to entity-relevant
tokens and reducing computational cost. To support this, we introduce a structured multi-shot script format. We additionally propose a budgeted
memory update strategy to maintain a compact, evolving
memory. Finally, we equip the entity representation with a noise-injection mechanism that enables fine-grained appearance control, preventing leakage of irrelevant information. Our method improves
prompt adherence and efficiency while preserving subject consistency.
Read the full abstract ▾
All authors Jente Vandersanden, Matheus Gadelha, Chun-Hao P. Huang, Hyeonho Jeong, Yulia Gryaditskaya
Computer Vision
GlowGS: Generative Semantic Feature Learning for 3D Gaussian Splatting in Nighttime Glow Scenes
Beibei Lin, Xiao Cao, Jingyuan Guo et al. (+1 more)
Existing 3DGS methods effectively render high-quality novel views in clear-day scenes. However, they struggle with night scenes, particularly in glow regions, due to the lack of structural features such as textures and edges, which are key cues for splatting-based reconstruction. To address this problem, we leverage a
diffusion model and a
Vision Foundation Model (VFM) to compensate for missing structural cues. Our method consists of two key novel ideas: semantic feature generation and novel-view semantic learning. First, semantic feature generation produces high-quality semantic features as implicit structural cues for novel views. Specifically, a
diffusion model synthesizes novel views with unknown camera poses from training views, while a VFM evaluates their quality. Once high-quality novel views are identified, the VFM extracts robust features to construct the semantic feature bank. Second, novel-view semantic learning enables 3DGS to optimize rendered novel views without requiring ground truth. It achieves this by extracting semantic features from a rendered novel view, searching the feature bank for the most similar features, and minimizing their distance. This process enforces implicit structural constraints, ensuring semantically coherent, artifact-free rendered views. Extensive experiments demonstrate the effectiveness of our GlowGS in generating semantically accurate 3D views, showing significant improvements over existing methods.
Read the full abstract ▾
All authors Beibei Lin, Xiao Cao, Jingyuan Guo, Robby T. Tan
Computer Vision · Cross-listed from cs.LG
Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning
Trinh Pham, Viet Huynh, Hongzhi Yin et al. (+2 more)
The rapid advancement of machine learning has led to an unprecedented expansion of model ecosystems, making it increasingly difficult to assess the reliability of newly released models on unseen and unlabeled data. Existing evaluation pipelines typically rely on costly annotation, repeated fine-tuning, or assumptions that do not generalize well to new models. We introduce MetaEvaluator, a cost-effective, model-agnostic framework for fast, label-free evaluation of unseen models across diverse architectures and modalities. MetaEvaluator meta-learns over a pool of reference models to acquire an effective initialization for accurate assessment of unseen models, thereby amortizing evaluation cost and eliminating the need for per-model retraining. To the best of our knowledge, this is the first model-agnostic framework that evaluates new models on unlabeled
datasets. Extensive experiments demonstrate that MetaEvaluator delivers stable and accurate performance estimates at substantially lower cost than conventional approaches, enabling scalable benchmarking on unlabeled
datasets for emerging models. The code is available at: https://github.com/phkhanhtrinh23/MetaEvaluator.
Read the full abstract ▾
All authors Trinh Pham, Viet Huynh, Hongzhi Yin, Quoc Viet Hung Nguyen, Thanh Tam Nguyen
Computer Vision
Calibration-Informative Region Selection for Online LiDAR–Camera Calibration in Agricultural Environments
Rajitha de Silva and Grzegorz Cielniak
Reliable multi-modal calibration requires identifying which observations truly constrain the extrinsic parameters and which ones mainly add noise or ambiguity. In this paper, we propose a support-map-driven approach to multi-modal calibration that decouples four functional blocks: initial calibration, cross-modal residual extraction, support-map estimation, and support-aware refinement. We instantiate this formulation for online
LiDAR–camera calibration using MDPCalib, a target-less
LiDAR–camera calibration method based on motion and deep point correspondences, and CMRNext, a dense
LiDAR–camera matching model that predicts optical-flow-like image-plane residuals. The key contribution is a dense calibration support map that aggregates cross-modal agreement over aligned observations and highlights where calibration evidence is consistently reliable. Across the Bacchus Long-Term (BLT)
dataset and KITTI, we show that calibration evidence is spatially and semantically non-uniform, indicating that some semantic regions provide stronger cues for calibration than others. On KITTI, support-guided refinement improves the calibration performance with better translation accuracy while rotational gains remain limited.
Read the full abstract ▾
Computer Vision
PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA
Chunze Yang, Qidong Liu, Wenjie Zhao et al. (+10 more)
Whole-slide image visual question answering (WSI-VQA) frames pathology as an extreme-context search problem: to answer a free-form clinical query, a system must first navigate a gigapixel slide under a strict inspection budget to locate sparse, high-resolution evidence. Existing approaches largely fall into two paradigms: i) supervised pathology
multimodal large
language models (MLLMs) and
agents can absorb localization and
reasoning into learned modules, but they often couple
navigation to task-specific supervision and retraining, limiting their practicality; ii) training-free pathology
agents avoid this cost by keeping core models frozen, but often follow a question-first design, constructing the initial candidate set mainly from query-conditioned relevance. This can miss decisive morphology that is not named in the question, and force heavier inference-time scaffolding. To address this challenge, we introduce PathNavigate, a training-free pathology
agent built around a scan-search-readout routine. Before question matching, PathNavigate scans the current slide at low magnification with a shared online
memory module over frozen pathology features, producing a slide-specific surprise field that marks an abnormal-region pool. It then applies question-conditioned PLIP relevance only within this pool to select high-magnification search targets. Finally, it extracts local high-magnification evidence and answers with a frozen perceptor-adjudicator stack, using the same online
memory as slide-level context. Experiments on WSI-VQA and SlideBench-BCNB show that the proposed scan-search-readout design improves answer accuracy and yields more interpretable evidence-selection trajectories with higher efficiency.The code is available online.
Read the full abstract ▾
All authors Chunze Yang, Qidong Liu, Wenjie Zhao, Yue Tang, Jiusong Ge, Di Zhang, Jiashuai Liu, Lei Wu, Junbo Lu, Ni Zhang, Xian Wu, Zeyu Gao, Chen Li
Computer Vision
Generator-Refiner-Examiner: A Tri-Module Data Augmentation Framework for 3D Human Avatar Learning from Monocular Videos
Gangjian Zhang, Jian Shu, Sicheng Yu et al. (+3 more)
This paper addresses the challenge of reconstructing photorealistic and animatable 3D human avatars from
monocular videos. While existing methods rely on combining per-subject optimization with generic human priors, they often fail to capture fine-grained details when training frames are limited. To mitigate this data scarcity, we propose TrioMan, a systematic tri-module framework for augmented 3D avatar learning. Our approach comprises three synergistic components. The Generator creates diverse unseen samples by imposing Gaussian perturbations on pose and camera. The Refiner improves the quality of generated data through one-step
diffusion guided by texture and geometry cues. The Examiner selects subject-consistent samples using a dual-branch
attention-based similarity evaluation. Experiments on the X-Humans and NeuMan
benchmarks show that TrioMan outperforms state-of-the-art methods.
Read the full abstract ▾
All authors Gangjian Zhang, Jian Shu, Sicheng Yu, Wenhao Shen, Yu Feng, Hao Wang
Computer Vision
PixIE: Prompted Pixel-Space Low-Light Image Enhancement
Ruirui Lin, Guoxi Huang, David Bull et al. (+1 more)
Low-light images suffer from severe noise, contrast loss, and semantic ambiguity, making enhancement a joint problem of denoising and detail recovery. We propose PixIE, a feed-forward pixel-space LLIE framework semantically prompted by a foundation model (FM). PixIE first performs cross-scale denoising to suppress noise while preserving structure, then refines details using Prompted Pixel Blocks (PPBs), which inject intermediate FM features through a novel spatially continuous modulation (SCMo). To make pixel-space
attention efficient across scales, we introduce Spatial-Channel Compaction (SCC), which jointly reduces the spatial
token grid and channel dimension. We further propose Multi-Receptive-Field Pixel
Embedding (MRPE) to provide neighborhood-aware pixel representations before semantic prompting, improving robustness to signal-dependent noise beyond point-wise
embeddings. Experiments on standard LLIE
benchmarks demonstrate state-of-the-art performance, achieving the best PSNR, SSIM, and LPIPS on the challenging LOLv2-Real
benchmark. Qualitative comparisons further show sharper details with more natural and consistent textures, improving both reconstruction fidelity and perceptual quality.
Read the full abstract ▾
All authors Ruirui Lin, Guoxi Huang, David Bull, Nantheera Anantrasirichai
Computer Vision
ComPose: When to Trust Hands for Object Pose Tracking
Jisu Shin, Junoh Lee, JunGyu Lee et al. (+5 more)
Reconstructing the motion of objects from
videos is a key component for
embodied AI and
robot manipulation. While diverse approaches to object pose tracking have been studied, they rely heavily on strong external priors, such as depth data or 3D templates, and remain highly vulnerable to severe occlusions by hand grasps despite the use of explicit masks. In this work, we present ComPose, a 6DoF object tracking framework designed for hand-aware object pose estimation from RGB
video. Rather than treating the hand purely as an occluder, our method harmonizes hand motions as a complementary cue for object tracking. In detail, we recover a variety of object motions over time by combining object and hand cues from foundation models within a unified tracking pipeline. Here, ComPose adaptively selects informative hand joints, combines object- and hand-derived cues for motion estimation, and refines the resulting object motion using visible
geometric evidence and a learned correction. We further enforce the temporal consistency over both rotation and translation, yielding stable 3D object trajectories over time without any external smoothing. Extensive experiments show that our method is accurate, efficient, and robust under severe hand occlusion and
geometric ambiguity. In addition, the resulting trajectories can also effectively transfer to downstream
robot manipulation by enabling
robots to reconstruct human actions from online
videos.
Read the full abstract ▾
All authors Jisu Shin, Junoh Lee, JunGyu Lee, Inhwan Bae, Dohyeon Lee, Hokyun Im, Youngwoon Lee, Hae-Gon Jeon
Computer Vision · Cross-listed from cs.LG
Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models
Jade Zou, Tao Huang, Weijie Kong et al. (+7 more)
Reinforcement learning (RL) has become an effective way to improve
prompt alignment and perceptual quality in
diffusion and flow-matching generators. A critical step for applying online RL to flow matching is turning the deterministic sampling trajectory into a
stochastic policy, typically by replacing the reverse-time Ordinary Differential Equation (ODE) with a
Stochastic Differential Equation (SDE). The
stochastic sampler, controlling the exploration behavior and denoising dynamics, is thus part of the policy, and its design can significantly affect the reward optimization performance. We break down the sampler design into two interdependent components: choosing the right amount of
stochastic exploration, and discretizing the resulting SDE faithfully at the small step counts used in RL. To address the first component, we analyze the inherent tension between exploration and stability in denoising and derive an SDE schedule that balances the two. Turning to the discretization challenge, we use a toy example to show that existing samplers can deviate from the flow-matching process, either by introducing excessive discretization noise or by relying on heuristic rules that do not guarantee convergence to the data distribution. To address these issues, we propose Precise, a new
stochastic sampler that balances effective exploration with stability. Crucially, Precise keeps the denoising trajectory SDE-consistent through a novel approximation that freezes the clean-latent posterior mean, resolving the excess noise issue in standard samplers. Extensive experiments demonstrate that this formulation leads to significantly faster and more stable reward optimization via reinforcement learning, achieving state-of-the-art
alignment scores (e.g., PickScore, HPSv2.1) while requiring 13.1-53.2% less wall-clock training time to match the best in-domain performance of prior samplers.
Read the full abstract ▾
All authors Jade Zou, Tao Huang, Weijie Kong, Junzhe Li, Yue Wu, Qi Tian, Jiangfeng Xiong, Jianwei Zhang, Liefeng Bo, Zhao Zhong
Computer Vision
VINS-120K: Ultra High-Resolution Image Editing with A Large-Scale Dataset
Zhizhou Chen, Shanyan Guan, Zhanxin Gao et al. (+6 more)
Directly editing ultra-high-resolution (UHR) images is valuable but underexplored, primarily due to the lack of high-quality data and the challenge in modeling high-frequency texture details. We introduce VINS-120K, the first large-scale
dataset for instruction-based UHR image editing, comprising 120K carefully curated triplets of instruction, input image, and edited image. Each image exceeds 4K resolution ($\geq$4096 $\times$ 4096) and is filtered through a rigorous multi-stage pipeline to ensure visual quality, instruction
alignment, and aesthetic fidelity. Built on VINS-120K, we further develop a high-frequency-aware post-adaptation strategy to extend pretrained non-high-resolution models to the UHR regime. We also present VINS-4KEval, a
benchmark covering diverse editing types, to facilitate consistent evaluation in UHR settings. Experiments confirm that our work improves fine-grained detail synthesis and texture realism in UHR image editing.
Read the full abstract ▾
All authors Zhizhou Chen, Shanyan Guan, Zhanxin Gao, En Ci, Yanhao Ge, Wei Li, Zhenyu Zhang, Jian Yang, Ying Tai
Computer Vision · Cross-listed from cs.GR
DrawVideo: Grounded and Faithful Multi-Shot Video Generation from Storyboard Keyframe Sketches
Chuanzhi Xu, Huiqi Liang, Bang Shi et al. (+7 more)
Long
video generation requires high-fidelity visual synthesis, coherent narrative organization, shot-level structure, and explicit user control. Existing text-to-
video methods typically generate
videos from a single long-form
prompt, making it difficult for creators to directly control character pose, camera composition, spatial layout, and local motion. We propose DrawVideo, a sketch-guided and storyboard-driven framework that grounds multi-shot
video generation in creator-provided spatial structure, appearance, and motion. Each shot is specified by a black-and-white sketch, an appearance
prompt, and a motion
prompt. DrawVideo first generates a structure-aligned reference keyframe, expands the motion description into derivative keyframes representing ordered action states, and then synthesizes local
video clips between adjacent keyframes. We further introduce SketchLongVideo, to the best of our knowledge the first evaluation
dataset of ordered multi-shot storyboards with aligned sketch, appearance, and motion conditions. Extensive experiments demonstrate strong structural controllability, appearance consistency, intra-shot visual stability, and motion
alignment, providing an effective solution for director-oriented
video creation.
Read the full abstract ▾
All authors Chuanzhi Xu, Huiqi Liang, Bang Shi, Huiming Zhang, Guangcheng Lin, Yifan Xiao, Haodong Chen, Qiang Qu, Zhicheng Lu, Weidong Cai
Computer Vision
MDS-DETR: DETR with Masked Duplicate Suppressor
Chanho Lee, Seunghee Koh, Yunho Jeon et al. (+1 more)
The DEtection
TRansformer (DETR) is a powerful end-to-end object detector, yet its one-to-one matching strategy suffers from slow convergence and low recall. A common approach to address this issue is to use one-to-many label assignment to provide more positive samples. However, existing methods that use one-to-many matching as an auxiliary objective lead to increased training costs, with their auxiliary decoders discarded during inference. To address this limitation, we propose MDS-DETR, which leverages both one-to-one and one-to-many supervision within a single decoder. Specifically, we introduce a Masked Duplicate Suppressor (MDS) that injects asymmetry into self-
attention via confidence-based causal masking. MDS filters out the duplicates generated by the one-to-many supervised layer, enables explainable, duplicate-free predictions in a fully end-to-end framework. MDS-DETR outperforms existing one-to-many DETR variants such as MS-DETR, MR.DETR and Relation-DETR, without relying on any additional queries or auxiliary decoders. Under a 12-epoch training schedule on MS COCO with a ResNet-50 backbone, MDS-DETR achieves a +2.8 mAP improvement over Deformable-DETR with only a 5% increase in training time, and outperforms the state-of-the-art MR.DETR by +0.3 mAP while being even 20% faster in training. Our code and models are available at \href{https://github.com/dcholee/mds-detr}{https://github.com/DChoLee/MDS-DETR}.
Read the full abstract ▾
All authors Chanho Lee, Seunghee Koh, Yunho Jeon, Junmo Kim
Computer Vision
B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation
Mario Markov, Stefan Maria Ailuro, Mohammad Mahdi et al. (+2 more)
Segmentation is a fundamental task in computer
vision, underpinning pixel-level scene understanding and serving as a cornerstone for applications ranging from autonomous perception to medical image analysis. For complex referring
segmentation, recent methods pair large
vision-
language models with
segmentation decoders: the former analyzes the image and
prompt, while the latter predicts the target mask. Although reinforcement learning improves
reasoning-intensive
vision-
language systems, trainable tools such as
segmentation decoders are typically optimized separately with differentiable objectives, and the principled integration of such objectives into reinforcement learning remains underexplored. Thus, we introduce group relative tool optimization (GRTO), a mathematically grounded framework for jointly optimizing a policy with differentiable tool use. GRTO reuses group relative policy optimization (GRPO) rollouts to optimize the auxiliary tool objective, letting decoder gradients complement policy rewards. Further, we derive Bootstrapped-GRTO (B-GRTO), a pre-training method that cheaply bootstraps the tool, leading to faster convergence and superior performance. Across three challenging referring
segmentation settings, B-GRTO results in substantial improvements over plain GRPO, matching or surpassing domain-specific state-of-the-art methods. This demonstrates the value of unifying reinforcement learning with differentiable auxiliary objectives for
reasoning-intensive
segmentation.
Read the full abstract ▾
All authors Mario Markov, Stefan Maria Ailuro, Mohammad Mahdi, Luc Van Gool, Danda Pani Paudel