Computer Vision
Pose Anything Anywhere:Model-free Object Poses from Arbitrary References
Hongli Xu, Jiaqi Hu, Junwen Huang et al. (+5 more)
Estimating the 6D pose of unseen objects is a fundamental yet challenging problem for open-world robotics and
embodied perception. Model-based methods are accurate but depend on CAD assets or heavy onboarding, while most model-free approaches are still limited to pairwise single-anchor matching and thus fail under occlusion and large viewpoint changes with low query-reference overlap. Therefore, we present PANY, a unified model-free framework that seamlessly supports both RGB and RGB-D inputs, operates on one or sparse pose-free reference views, and generalizes effectively to novel objects. Built on a multi-view
transformer geometry backbone, PANY moves beyond pairwise matching by learning view-consistent geometry and cross-view
alignment cues that remain stable under wide baselines and limited overlap. When additional unposed assist views are available, PANY aggregates them via pose-graph canonical registration to increase
geometric coverage and reinforce the final pose. Extensive experiments show that PANY achieves state-of-the-art performance across multiple
benchmarks, substantially outperforming existing model-free methods, improving pose accuracy by +12% on YCB-V and over +20% on LM-O. Furthermore, PANY consistently performs well under both single-reference and sparse-reference settings, demonstrating strong robustness in real-world environments.
Read the full abstract ▾
All authors Hongli Xu, Jiaqi Hu, Junwen Huang, Boyang Zhong, Peter KT Yu, Nassir Navab, Benjamin Busam, Slobodan Ilic
Computer Vision
Hedgementation = Hedgerow Segmentation: A Remote Sensing Benchmark
Nathan Senyard, Salem Hamdani, Astrid Zhang et al. (+4 more)
We propose Hedgementation: a new
benchmark to evaluate machine learning models for hedgerow mapping from remote sensing data at country scale and 10m$^2$ spatial resolution. We combine and harmonize multiple remote sensing data products and ground truth labels sourced from a hedgerow inventory in France. We measure the ability of three baseline models to generalize across spatial distance, and across climatic zones, a more explicitly challenging task. Our
benchmark tests both supervised and self-supervised learning approaches for remote sensing, applied to tracking fine-scale features of high agricultural importance. The code to reproduce the
benchmark and baselines results is available at https://github.com/hedgementation/hedgementation.
Read the full abstract ▾
All authors Nathan Senyard, Salem Hamdani, Astrid Zhang, Derek Wang, Evan Shelhamer, Mathias Lécuyer, Joséphine Gantois
Computer Vision
Data Selection Through Iterative Self-Filtering for Vision-Language Settings
Andrei Liviu Nicolicioiu, Sarvjeet Singh Ghotra, Morgane M. Moss et al. (+1 more)
The availability of large amounts of clean data is paramount to training neural networks. However, at large scales, manual oversight is impractical, resulting in sizeable
datasets that can be very noisy. Attempts to mitigate this obstacle to producing performant
vision-
language models have so far involved heuristics, curated reference
datasets, and using pre-trained models. Here we propose a novel, bootstrapped method in which a CLIP model is trained on an evolving, self-selected
dataset. This evolving
dataset constitutes a balance of filtered, highly probable clean samples as well as diverse samples from the entire distribution. Our proposed Self-Filtering method iterates between training the model and selecting a subsequently improved data mixture. Training on
vision-
language datasets filtered by the proposed approach improves downstream performance without the need for additional data or pre-trained models.
Read the full abstract ▾
All authors Andrei Liviu Nicolicioiu, Sarvjeet Singh Ghotra, Morgane M. Moss, Aaron Courville
Computer Vision
Vera: A Layered Diffusion Model for Content-Preserving Video Editing
Hongkai Zheng, Ta-Ying Cheng, Benjamin Klein et al. (+2 more)
Video diffusion models have enabled remarkable progress in
video generation and editing. However, content preservation remains a core challenge: existing methods regenerate every pixel and often alter elements that should remain unchanged, such as characters or background scenes. We introduce Vera, a layered
diffusion framework for content-preserving
video editing. Instead of regenerating the entire
video, Vera generates an edit layer along with an alpha matte for compositing with the source
video, separating creative editing from content preservation by design. To encourage coherent composition with the source
video, we extend the text-to-
video DiT into a Mixture-of-
Transformers (MoT) architecture, with separate DiTs for each layer that interact through joint self-
attention. To support the training of Vera, we further construct a high-quality layered
dataset with accurate alpha mattes, diverse scenes and dynamics, and visual effects. Across our quantitative
benchmark and human preference study, Vera outperforms leading open-source
video editing models in content preservation while remaining competitive in edit quality, using 486K frames of layered training data.
Read the full abstract ▾
All authors Hongkai Zheng, Ta-Ying Cheng, Benjamin Klein, Yisong Yue, Zhuoning Yuan
Computer Vision · Cross-listed from cs.LG
Discovering Latent Groups for Robust Classification
Ankur Garg, Ulrich Aïvodji, Samira Ebrahimi Kahou et al. (+1 more)
Machine learning models exploit spurious correlations, achieving high average accuracy but failing disproportionately on underrepresented subgroups. Existing methods address this by adjusting network parameters, guided either by subgroup annotations or inferred pseudo-group labels. Yet at inference, these methods produce only a class prediction, with no insight into a sample's latent subgroup. We propose neural classification trees (NCT), a framework that achieves robustness by encoding subgroup structure in its tree-shaped architecture. By routing each sample to an "easy" or "hard" node of this tree - based on prediction correctness - and reusing these routes as pseudo-labels for the next iteration, NCT disentangles conflicting subgroups, without requiring subgroup supervision. We evaluate NCT on five
benchmarks spanning binary and multi-class spurious correlations. Our experiments show that the learned tree
topology provides strong interpretability by consistently isolating minority subgroups, which provides a transparent mapping between the model architecture and the data's latent group structure, while yielding competitive robustness with state-of-the-art methods.
Read the full abstract ▾
All authors Ankur Garg, Ulrich Aïvodji, Samira Ebrahimi Kahou, Vincent Michalski
Computer Vision · Cross-listed from cs.RO
Autonomous Subsea Cable Search and Tracking with Graph-Optimised Priors and Visual Tracking
Ibrahim Fadhil Djauhari, Adrian Bodenmann, Samuel Simmons et al. (+6 more)
Global communications rely on subsea cable infrastructure that remains vulnerable to damage from natural hazards and human activity. Autonomous
underwater vehicles (AUVs) offer an efficient means to inspect long sections of exposed cable, but uncertainty in cable route maps, small cable diameters and partial burial makes continuous tracking a challenge. This paper presents a novel cable search and tracking method that leverages uncertain prior cable route maps. Graph-based optimisation continuously update the cable route to remain consistent with visual observations. Route uncertainty is constrained as a function of distance from observations using physics-based catenary models that account for cable parameters (i.e., lay depth, diameter, and density), bounding the search space to physically feasible regions and improving search efficiency. Cable detection is performed using a semi-supervised classifier running in real-time on-board a camera-equipped AUV. These detections both update the graph-based optimisation and enable visual cable tracking. When tracking is lost due to misclassification, burial or imperfect control, the bounded search space enables efficient recovery. The approach was demonstrated in field trials using the University of Southampton's Smarty200 AUV. The system successfully located the cable despite deliberate errors in it initial cable route map, updating this to be consistent with observations and using visual tracking to inspect up to 59% of a 120m test cable, with successful recovered after tracking loss.
Read the full abstract ▾
All authors Ibrahim Fadhil Djauhari, Adrian Bodenmann, Samuel Simmons, Cailei Liang, David White, Susan Gourvenec, Tom Bennetts, Darryl Newborough, Blair Thornton
Computer Vision
Polycepta: Object-Centric Appearance Estimation for Multi-Object Tracking
Mohamed Nagy, Naoufel Werghi, Jorge Dias et al. (+1 more)
The tracking-by-detection paradigm in multi-object tracking (MOT) typically relies on static appearance descriptors to complement motion estimation. However, these descriptors are frame-independent, limiting their robustness as visual cues. Since such descriptors are often obtained from computationally intensive pretrained backbones, real-time MOT systems frequently abandon appearance cues altogether and rely solely on motion prediction and
geometric association. In this work, we introduce Polycepta, an object-centric appearance state estimation framework that reformulates appearance modeling as a recursive estimation problem rather than a frame-wise matching task. Polycepta constructs and continuously updates an independent appearance state for each tracked object, enabling future appearance representations to be estimated from accumulated observations. Polycepta is encouraged to learn the appearance-state construction of object-specific representations rather than memorize them through a proposed learning strategy, enabling appearance estimation for unseen classes. A key property of Polycepta is that the quality of appearance estimation improves as object states evolve during inference. While conventional appearance descriptors remain static or degrade over time, Polycepta progressively refines appearance estimates as additional observations are accumulated. Extensive experiments on KITTI, the Waymo Open
Dataset, and MOT17 demonstrate consistent reductions in identity switches and improvements in tracking performance when integrated into the tracking-by-detection pipelines. Polycepta operates at 90.57 Hz and delivers state-of-the-art performance on the KITTI
benchmark, achieving a MOTA of 92.27%.
Read the full abstract ▾
All authors Mohamed Nagy, Naoufel Werghi, Jorge Dias, Majid Khonji
Computer Vision · Cross-listed from cs.RO
Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery
Seyed Hamid Reza Roodabeh, Zongyu Li and Homa Alemzadeh
Robot-assisted minimally invasive surgery improves
surgical precision but introduces complexity, making technical error detection essential for ensuring patient safety. Current executional error detection methods using
video data often overlook fine-grained contextual descriptions of activities and error types within the hierarchical structure of
surgical procedures. They also under-utilize complementary
multimodal information. We propose a unified framework for executional error detection that leverages
multimodal input, including
video, kinematics, and descriptive textual
prompts. Through activity prompting, we integrate descriptive
language in gesture-level activities, instrument-object interactions, and error definitions. We also introduce activity-aware visual
embeddings derived from
vision encoders pretrained on
surgical activity labels to compare the effectiveness of contrastive
language-image
embeddings with traditional image-based
embeddings for error detection. By seamlessly integrating kinematic data with
video and textual modalities, our framework significantly improves error detection performance. Achieving up to 5% and 16.6% F1 score improvements over state-of-the-art baselines on the JIGSAWS and SAR-RARP50
datasets, respectively, we demonstrate the value of combining curated textual
prompts with
multimodal data for accurate error detection.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.DC
Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse
Bole Ma, Jan Eitzinger, Harald Koestler et al. (+1 more)
Multimodal agents repeatedly re-examine the same
video frames, UI screenshots, and rendered artifacts as their context window slides and
reasoning iterates, yet every look-back re-encodes from scratch, because prefix caches serve reuse only at a fixed leading position. We show this recompute is avoidable, and identify exactly what naive KV reuse loses: the cross-chunk conditioning a chunk absorbs from its neighbours. This loss is asymmetric. The direct readout of a cached chunk is recovered exactly and for free by the standard state-merge. What remains is a diffuse, low-rank residue concentrated in deep layers, invisible to single-hop
retrieval but precisely what multi-hop
reasoning binds on. Blind reuse therefore leaves single-hop recall intact while halving multi-hop accuracy; this is the failure mode prior position-independent caches, designed for single-context or single-image reuse, do not address. We repair it with a small, training-free low-rank conditioning patch stored alongside each position-free chunk. Reuse reduces to one operator across MLA, GQA, and MHA: exact RoPE re-rotation to any target position, plus the patch that restores cross-chunk binding. This makes three window operations cheap: reorder (one patch serves every ordering of a cached set), sliding-window survival (surviving chunks relocate via rotation only, zero re-encode), and recall (an evicted chunk is rehydrated by its patch, never re-encoded). A rank-m patch recovers full task accuracy on cross-chunk-binding
benchmarks, MM-NIAH across two
attention families and two-page doc-QA, at a fraction of the KV footprint, and reconstructs re-prefill KV to within bf16 rounding in a production SGLang kernel across six backbones. The conditioning signal is strongest in redundant
vision and
video streams, making our solution most impactful where
multimodal agents spend their recompute budget.
Read the full abstract ▾
All authors Bole Ma, Jan Eitzinger, Harald Koestler, Gerhard Wellein
Computer Vision · Cross-listed from cs.RO
HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memory
Xiaolin Zhou, Liu Liu, Tingyang Xiao et al. (+9 more)
LLM
agents follow a practical execution loop in digital environments: they reason over structured states, invoke tools, inspect feedback, and revise actions. Extending this loop to physical
robots is difficult because physical execution is continuous, embodiment-dependent, uncertain, and constrained by safety. Existing
embodied-AI systems have advanced manipulation, spatial understanding,
navigation, and
humanoid control, but these capabilities often remain specialized modules or loosely coupled decision loops. In this work, we introduce HoloAgent-0, a unified
embodied agent framework for real-world
robot deployment.
Embodied AgentOS converts
language instructions into executable skill graphs, schedules
robot resources, monitors execution, and triggers clarification or re-planning from runtime feedback. HoloAgent-0 organizes heterogeneous
robot models and controllers through three coupled layers:
Embodied AgentOS for closed-loop execution, 3D spatial
memory for physical world grounding, and
embodied skills for
robot action. We deploy HoloAgent-0 on real hardware and evaluate its spatial
memory, long-horizon
navigation, and closed-loop execution across motion generation, object search, cross-
robot coordination, and mobile manipulation.
Read the full abstract ▾
All authors Xiaolin Zhou, Liu Liu, Tingyang Xiao, Wei Feng, Fa Fu, Xinrui Meng, Xinjie Wang, Jialiang Han, Boyang Yu, Yun Du, Wei Sui, Zhizhong Su
Computer Vision
Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views
Jiho Choi, Seonho Lee, Seojeong Park et al. (+1 more)
Multi-view 3D Visual Question Answering (MV3D-VQA) requires integrating partial observations into a coherent 3D scene representation and selecting informative viewpoints for multi-step spatial
reasoning. However, current
multimodal LLMs are typically trained with sparse, answer-level supervision, which often yields inconsistent cross-view
reasoning and brittle view selection. We present DR-MV3D (Dense Reward for MV3D-VQA), a map-grounded learning framework that provides dense, verifiable rewards to supervise the
reasoning process. Our approach decomposes MV3D-VQA into (i) allocentric global map construction, (ii) question-conditioned view-trajectory planning, and (iii)
egocentric grounding for answer prediction. To make intermediate steps learnable without manual annotations, we introduce two rewards: a global consistency reward that aligns the predicted map with geometry-consistent pseudo targets from frozen 3D
vision foundation models (e.g., VGGT + SAM3), and a local trajectory reward that supervises ordered viewpoint selection. We optimize the full pipeline with trajectory-level policy optimization (GRPO). Experiments on MindCube, VSI-Bench, and BLINK (MV) show that DR-MV3D consistently improves over strong multi-image baselines, supporting the effectiveness of process-level dense supervision for multi-view 3D
reasoning.
Read the full abstract ▾
All authors Jiho Choi, Seonho Lee, Seojeong Park, Hyunjung Shim
Computer Vision · Cross-listed from cs.AI
VeriEvol: Scaling Multimodal Mathematical Reasoning via Verifiable Evol-Instruct
Haoling Li, Kai Zheng, Jie Wu et al. (+4 more)
Scaling reinforcement learning for visual mathematical
reasoning requires more than generating harder questions: as data volume grows, the reward labels themselves must remain reliable. Yet existing data pipelines scale supervision while trusting the labeller, and policy-side methods assume the underlying answers are already correct. We instead treat scaling as a verifiable data-construction problem and decouple two axes before any policy update:
prompt difficulty, expanded by route-specific
evolution operators, and answer reliability, enforced by offline hypothesis-test falsification. We instantiate this as VeriEvol, an iterative framework with two extensible components: a type-aware
evolution module that rewrites low-difficulty image-question seeds into harder, image-grounded
prompts; and HTV-
Agent, a verifier that accepts an answer only after multi-source counter-evidence has failed to refute it. The resulting verified data scales in volume, extends by adding
evolution routes or verifier channels, and plugs directly into existing GRPO-style RL recipes. On a five-
benchmark visual-math suite, scaling evolved SFT data from 10K to 250K samples raises the mean accuracy from 35.42 to 54.73; then, with backbone, SFT initialization, and GRPO recipe held fixed, VeriEvol adds a cumulative +3.88 over an un-evolved RL baseline, of which +1.82 comes from evolved
prompts and +2.06 from the HTV-
Agent verifier. We release the
prompts, data, models, code, and the full verifier trace of every sample, so that downstream work can scale and audit the pipeline rather than only inspect its outputs.
Read the full abstract ▾
All authors Haoling Li, Kai Zheng, Jie Wu, Can Xu, Qingfeng Sun, Han Hu, Yujiu Yang
Computer Vision
AwakeForest: An Interactive Geospatial Platform for Large-Scale Forest Imagery
Suraj Prasai, Kangning Cui, Rongkun Zhu et al. (+5 more)
Forest imagery analysis often involves multiple tightly coupled
vision tasks, which must be performed under substantial variation in geographic regions, sensors, and acquisition conditions. However, practitioners often lack a unified tool that is geospatial-native, cloud-optimized, and ML-integrated for end-to-end workflows spanning annotation, prediction, visualization, and downstream analysis at scale. We present AwakeForest, an interactive end-to-end platform designed for large-scale forest imagery that integrates model-assisted inference, automatic annotation, and human-in-the-loop refinement within a single workflow. Our platform supports plug-and-play integration of pretrained models and enables scalable interaction with forest imagery ranging from standard aerial scenes to large orthomosaics that can span several gigabytes to hundreds of gigabytes. AwakeForest produces analysis-ready outputs that can be directly used for downstream analysis and to support iterative model and annotation updates on new scenes. We demonstrate the system on the PALMS
dataset and illustrate how AwakeForest supports an end-to-end workflow for practical forest management and analysis.
Read the full abstract ▾
All authors Suraj Prasai, Kangning Cui, Rongkun Zhu, Sarra Alqahtani, Ying Zhang, Victor Paul Pauca, Miles R. Silman, Fan Yang
Computer Vision
LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement
Tongkun Guan, Haocheng Wang, Wei Shen et al. (+1 more)
Visual document
retrieval requires rapidly locating relevant pages from large multi-modal corpora in response to user queries. While recent methods powered by Multi-modal Large
Language Models (MLLMs) show competitive accuracy, they suffer from prohibitive computational costs by applying intensive MLLM encoding to every single page. Meanwhile, we observe that user queries are typically keyword-anchored, containing semantically rich words that are expected to appear directly in the visible text of relevant pages, offering an efficient cue for quickly narrowing down candidate pages. Building on this insight, we propose LightSTAR, an efficient framework that decomposes visual document
retrieval into: 1) LLM-free Visual Selection, which utilizes content-grounded query encoding to focus on informative words and employs LLM-free visual
embeddings to produce a high-recall candidate set; and 2)
Vision-adaptive Semantic Refinement, which further performs fine-grained semantic matching exclusively on these top candidates via adaptive region-wise feature fusion to effectively combine textual and layout cues, optimized through a hardness-aware contrastive objective. Experimental results demonstrate that LightSTAR achieves state-of-the-art
retrieval accuracy while reducing end-to-end latency by several-fold, offering a highly practical solution to the accuracy-efficiency trade-off in visual document
retrieval. Code is available at https://github.com/bokufa/LightSTAR.
Read the full abstract ▾
All authors Tongkun Guan, Haocheng Wang, Wei Shen, Xiaokang Yang
Computer Vision
Scaling State-Space Models from Lines to Paragraphs: An Ablation of Mamba-based OCR
Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas et al. (+2 more)
End-to-end OCR increasingly relies on autoregressive sequence models, where the quadratic cost of
Transformer attention limits efficient transcription of long, paragraph-level text. State-Space Models (SSMs) such as Mamba offer linear-time decoding and have recently been shown to match
Transformer accuracy on printed historical lines, but their behavior as sequences grow from short lines to full paragraphs, and their generalization to handwriting, remain poorly understood. We study how a Mamba-based OCR recognizer scales from lines to paragraphs. We first conduct a systematic exploration of its four core hyperparameters (decoder depth, state dimension, expansion factor, and connector depth) on synthetic paragraphs from 100 to 1,000 characters, identifying the recurrent state dimension and the expansion factor as the dominant levers for long-sequence accuracy. We then compare the recognizer against a
Transformer baseline trained under an identical protocol. On clean synthetic paragraphs, both models stay below 1% CER at every length while the SSM runs 1.4 to 4.5 times faster, the speedup growing with sequence length. On real handwriting, however, the SSM lags clearly behind: it reaches 8.2% CER on IAM lines and 10.0% on IAM paragraphs, against 4.2% and 3.5% for the
Transformer baseline. Through controlled experiments we show that a substantial part of this gap stems from data scarcity rather than from an intrinsic architectural limit: the autoregressive SSM decoder is markedly data-hungry on long sequences. Our study clarifies when SSMs are a practical choice for large-scale document transcription and when they are not.
Read the full abstract ▾
All authors Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas, Clément Chatelain, Thierry Paquet
Computer Vision
Arbor: Explicit Geometric Conditioning for Controllable 3D Asset Generation
Jan-Niklas Dihlmann, Andreas Engelhardt, Simon Donne et al. (+2 more)
Text and image conditioned 3D models now generate convincing assets, but they still offer little direct control over the space an object should occupy or avoid. In authoring, this spatial intent is often known before generation starts. A chair should fit a seating envelope, a prop should leave clearance for motion, or a part should expose a contact surface.
Prompts and image views are poor carriers for such constraints, requiring the need for an explicit control interface. We present Arbor, a trainable attachment for text conditioned latent 3D generation. Arbor introduces constraint meshes as a native 3D control interface. The interface uses hull regions where geometry should exist, avoidance regions that should remain empty, and touch regions the object should contact. Unlike completion or whole object scaffold control, these meshes are not target evidence. They are local typed requirements and can include regions where no surface should appear. Arbor keeps this signal as geometry by converting constraint meshes into
tokens and learning a routed attachment inside a frozen denoiser. Each latent region can therefore receive the part of the constraint that matters for its spatial location. We evaluate Arbor on automatic and artist curated control
benchmarks with hull, avoidance, and touch constraints, and compare the metric trends to a user preference study. Even without dedicated compliance losses, Arbor improves constraint obedience while preserving object quality and variation under fixed constraints.
Read the full abstract ▾
All authors Jan-Niklas Dihlmann, Andreas Engelhardt, Simon Donne, Hendrik P.A. Lensch, Mark Boss
Computer Vision
UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation
Yohann Perron, Guillaume Astruc, Nicolas Gonthier et al. (+2 more)
Vision Transformers (ViT) dominate computer
vision. However, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales, and resolutions vary widely. We introduce UniverSat, a ViT-style backbone built around a Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal resolutions, and from both optical and non-optical sensors, into a shared
embedding space with a shared set of weights. This enables training a single model on heterogeneous
multimodal corpora via self-supervision, yielding robust, sensor-agnostic spatial features. We validate this approach with strong results across classification and
segmentation on standard EO
benchmarks from GeoBench, PANGEABench, and SpectralEarth. Our code and models are available at https://github.com/gastruc/UniverSat.
Read the full abstract ▾
All authors Yohann Perron, Guillaume Astruc, Nicolas Gonthier, Clement Mallet, Loic Landrieu
Computer Vision
Brain-Adapter: A Dual-Stream Vision-Language MIL Framework for Comprehensive 3D CT Diagnosis of Acute Intracranial Pathologies
Zhenyu Yi, Zhiyun Song, Yusong Sun et al. (+6 more)
Automated diagnosis of 3D
brain CT scans is essential for critical care, yet it remains challenging due to the heavy reliance on manual annotations and the limited semantic understanding of conventional models. While 2D foundation
vision-
language models (VLMs) have shown remarkable generalization, effectively transferring their representational power to 3D volumes remains an open problem. In this paper, we propose
Brain-Adapter, a novel dual-stream multiple instance learning (MIL) framework that leverages pre-trained 2D biomedical VLMs and raw diagnostic reports for robust scan-level multi-label classification. Specifically, we introduce a Text-Conditioned
Attention (TCA) mechanism, utilizing raw diagnostic sentences as semantic queries to dynamically align visual cues with specific disease concepts. Concurrently, a parallel visual MIL stream captures global scan characteristics, supervised by structured labels extracted via a Large
Language Model (LLM). To ensure representation coherence, a consistency constraint enforces synergy between the two streams. During inference, an Uncertainty-Aware Refinement (UAR) module dynamically calibrates and fuses these dual-stream predictions to resolve ambiguous cases. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art 3D models and standard MIL approaches. By eliminating the reliance on dense annotations,
Brain-Adapter provides a highly scalable and clinically viable solution for 3D acute intracranial pathology analysis.
Read the full abstract ▾
All authors Zhenyu Yi, Zhiyun Song, Yusong Sun, Zelin Liu, Manman Fei, Zhenhao Li, Jiaxuan Zhao, Xu Han, Lichi Zhang
Computer Vision · Cross-listed from cs.GR
MeshFlow: Mesh Generation with Equivariant Flow Matching
Qi Sun, Kiyohiro Nakayama, Jing Nathan Yan et al. (+6 more)
Meshes are among the most common 3D scene representations, but directly generating meshes is challenging because the representation contains important
symmetries, including permutation invariance of faces and vertices. MeshFlow learns to generate triangle meshes directly as triangle soups, avoiding the need to serialize meshes into long autoregressive sequences. We adopt equivariant optimal-transport flow matching models that respect the key
symmetries of triangle soups: arbitrary permutations of faces and permutations of the vertices within each face. Toward this goal, we propose a simple yet effective modification to the
Diffusion Transformer architecture, resulting in a scalable network capable of modeling a velocity field while maintaining the desired equivariance. We further introduce an optimal-transport-based training objective that improves convergence by eliminating supervision signals that violate these
symmetries. MeshFlow achieves mesh quality comparable to state-of-the-art autoregressive mesh generators while providing about an 18$\times$ speedup during inference. Project page is at https://qiisun.github.io/MeshFlow/.
Read the full abstract ▾
All authors Qi Sun, Kiyohiro Nakayama, Jing Nathan Yan, Qixing Huang, Alexander Rush, Leonidas Guibas, Gordon Wetzstein, Jing Liao, Guandao Yang
Computer Vision
From Reconstruction to Decision: A Post-Encoder Plug-in Adapter for Curvilinear Segmentation
Qin Lei, Jiang Zhong, Xin Xiao et al. (+2 more)
Curvilinear object
segmentation, including vessels and cracks, is challenging due to extreme spatial sparsity and topological fragility, where small local errors can cause severe structural disconnections. Meanwhile, modern
segmentation pipelines increasingly rely on strong but hard-to-modify foundation encoders whose heavy downsampling limits fine structural recovery. Motivated by this, we focus on the post-encoder stage and study two recurring and actionable failure modes: a reconstruction bottleneck in high-resolution feature restoration and a decision bottleneck in binarization. We present PEPA, a lightweight Post-Encoder Plug-in Adapter for 2D curvilinear
segmentation pipelines with accessible decoder/head features and target, query, or class descriptors. PEPA couples (i) Target-Conditioned Snake Upsampling (TCSU), which uses target-conditioned continuous snake-like sampling to better recover thin and tortuous structures during upsampling, and (ii) Target-Adaptive Differentiable Thresholding (TADT), which predicts target-specific thresholds and optimizes a soft-threshold surrogate with explicit safeguards against trivial bias shifting. Under this post-encoder interface, PEPA can be attached to both
prompt-based decoders and conventional dense predictors. Experiments on five medical and industrial
benchmarks show that adding PEPA to frozen-encoder baselines yields consistent improvements, with gains in topological connectivity (clDice) typically exceeding those in region overlap (IoU), indicating improved structural continuity. With only $\sim$0.26M additional parameters, PEPA offers a practical post-encoder enhancement for structure-centric
segmentation.
Read the full abstract ▾
All authors Qin Lei, Jiang Zhong, Xin Xiao, Yuming Yang, Hao Wu
Computer Vision
C^2GR: Coupled Comprehensive Generative Replay for a Continually Learnable Universal Segmentation Model
Wei Li, Jingyang Zhang, Guoan Wang et al. (+4 more)
Universal
segmentation models exhibit significant potential for diverse tasks involving different imaging modalities and
segmentation objectives. Task-Incremental Learning provides a
privacy-preserving approach to continually evolve a universal model on tasks from sequentially-arriving medical departments. However, training the model solely on the incoming task induces forgetting on past tasks, since consecutive tasks exhibit concurrent shifts in image appearance and
segmentation objective. To address this problem, we propose a novel Coupled Comprehensive Generative Replay (C^2GR) framework that simultaneously synthesizes image-mask pairs of previous tasks to mitigate forgetting under concurrent appearance and objective shifts. This requires preserving image-mask correspondence for structure-realistic generation and bridging asynchronous optimization of the generator and segmentor for
segmentation-oriented generation. Specifically, we propose a
Bayesian Joint
Diffusion (BJD) method that formulates the correspondence as conditional distributions optimized via conditional denoising. Furthermore, we develop a Relation-aware Unified
Prompt Synchronization (RUPS) scheme to simultaneously modulate the generator and segmentor via a shared task-relation-aware
prompt for synchronizing their optimization. Experiments on 20 tasks spanning diverse modalities and objectives demonstrate that C^2GR exhibits only a 2.44% drop in overall performance compared to joint training with all task data, effectively alleviating forgetting from the concurrent shifts. Our code will be made publicly available at https://github.com/mar-cry/C2GR.
Read the full abstract ▾
All authors Wei Li, Jingyang Zhang, Guoan Wang, Junzhi Ning, Yang Chen, Guang Yang, Lixu Gu
Computer Vision
MeGAS: Thermomechanical Dynamic Gaussian Splatting for Thermophysical Scene Editing
Zesong Yang, Yuanhang Lei, Liyuan Cui et al. (+6 more)
Recent advances integrate physically grounded Newtonian dynamics with neural rendering frameworks, narrowing the gap between photorealistic scene reconstruction and physics-based animation. However, existing approaches focus on mechanically driven dynamics while neglecting temperature, a fundamental yet invisible physical factor underlying phenomena such as melting, solidification, and other thermomechanical processes. In this paper, we propose MeGAS, a novel framework that incorporates thermomechanical phase-change dynamics into 3D Gaussian Splatting (3DGS). Specifically, we propose a new thermomechanical dynamic Gaussian Splatting representation that augments 3DGS with temperature attributes and employs a heat advection-
diffusion solver with MPM dynamics incorporating phase transitions, enabling physically plausible and visually realistic synthesis of thermophysical phenomena. Furthermore, a new
topology-adaptive Gaussian rendering strategy is proposed to mitigate cracking and floaters under extreme deformation. Extensive experiments demonstrate that MeGAS produces physically consistent thermomechanical behavior while maintaining high-fidelity photorealistic rendering, advancing toward physics-integrated world models.
Read the full abstract ▾
All authors Zesong Yang, Yuanhang Lei, Liyuan Cui, Yihang Chen, Jiaer Huang, Boming Zhao, Peter Yichen Chen, Hujun Bao, Zhaopeng Cui
Computer Vision
Rethinking Object-Centric Representations for Video Dynamics Modeling
Amaury Wei, Ismail Nejjar and Olga Fink
Learning to decompose
videos into persistent objects is a fundamental challenge in unsupervised object-centric representation learning. Despite recent progress, existing methods struggle to simultaneously achieve accurate object
segmentation, consistent identities over time, and reliable foreground-background separation. To address these challenges, we introduce UniSlot (Unified Slots), an unsupervised framework for learning robust and disentangled object-centric representations from
videos. UniSlot explicitly separates object appearance from its 3D-aware
geometric pose in the scene, linking object identity to appearance while leveraging depth to better distinguish objects from their surroundings. UniSlot achieves state-of-the-art performance in unsupervised object-centric
video decomposition and tracking across synthetic and real-world
benchmarks, yielding substantially tighter object masks and reducing background leakage while preserving object identities. Beyond decomposition and tracking, these improved representations translate directly to downstream tasks such as unsupervised object dynamics prediction, enabling more accurate
forecasting of future object trajectories.
Read the full abstract ▾
Computer Vision
Hiroaki Aizawa
Medical image
segmentation is a fundamental task for medical image processing and computer-assisted intervention, yet data imbalance and small lesion detection pose significant challenges. Dice Loss, which measures the overlap between predicted and ground truth regions, is widely used to mitigate these issues. To further emphasize its properties, we propose
Polynomial Dice Loss, a
polynomial extension of Dice Loss. Specifically, by leveraging the
geometric characteristics of Dice Loss and formulating the loss function as a
polynomial representation via Taylor expansion, we enable the adjustment of the contribution of higher-order components to the loss function. In our experiments, we evaluate the proposed method against loss functions derived from conventional Dice and Tversky coefficients. Experimental results and further analysis show that the
polynomial formulation provides a simple way to control the loss shape and achieves competitive performance across multiple
segmentation settings.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CR
TooBad: Backdoor Diffusion Models with Ultra-Low Poison Rate and Imperceptible Trigger
Vu Tuan Truong and Long Bao Le
Diffusion models (DMs), despite their impressive capabilities across a wide range of generative tasks, have been shown to be vulnerable to
backdoor attacks. However, existing
backdoor methods face critical trade-offs among key factors: attack performance, stealthiness, time complexity, and required poison rates. For example, achieving high attack performance typically demands a high poison rate and prolonged training, which undermines stealthiness, making the attack more detectable by
backdoor defenses. This paper proposes TooBad (trigger optimization for
backdoor diffusion models), a
backdoor framework which introduces a novel DM-tailored trigger optimization technique to dramatically enhance the performance of
backdoor attacks on DMs. Experiments on representative
benchmarks such as CIFAR-10 show that TooBad can achieve high ASRs ($> 85$%) at only 0.5% poison rate, significantly lower than the 10% typically required by prior work on the same
datasets. At 5% poison rate, TooBad reaches nearly 100% ASR within just 3-5
backdoor injection epochs, whereas existing methods need at least 30-50 epochs at double the poison rate for comparable results. Despite its potency, TooBad easily evades SOTA defenses and maintains high utility. These results reveal a critical threat on DMs and highlight the need for more robust defenses against such stealthy yet efficient attacks.
Read the full abstract ▾
Computer Vision
Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors
Tim G. Zhou, Anthony Fuller, Geoff Pleiss et al. (+1 more)
Machine learning models for remote sensing are trained and deployed on a static set of modalities. However, as we equip newer satellites with novel sensors and retire old ones, practitioners may wish to deploy an existing model on a substitution, superset, or subset of modalities with minimal retraining given data availability or practical computational constraints. We study the setting of updating existing models to changing modalities and identify three main scenarios: Modality Transfer (substitution), Addition (superset), and Peeking (subset). We propose DeluluNet, an architecture with modular components for all three changing modality scenarios. DeluluNet is trained end-to-end, learning a multi-modal model from a unimodal teacher and unlabeled
multimodal data via modality
hallucination–predicting missing modality representations from those that are present. As a result, DeluluNet can keep predicting even when input modalities change, providing a practical alternative to re-labeling and re-training in a changing world.
Read the full abstract ▾
All authors Tim G. Zhou, Anthony Fuller, Geoff Pleiss, Evan Shelhamer
Computer Vision
Faithful Grounded Visual Reasoning via Learned Proxy-Tokens
Tom Hodemon, Mohamed Chaouch, Aboubacar Tuo et al. (+1 more)
Multimodal Large
Language Models (MLLMs) have achieved remarkable success in Visual Question Answering (VQA), yet their "black-box" nature hinders deployment in critical domains. Grounded Visual
Reasoning (GVR) approaches attempt to improve interpretability by explicitly couple textual rationales with visual grounding information, which are typically textual coordinates. This mechanism lacks a learnable semantic link to the visual features, often resulting in a semantic-spatial gap where the model hallucinates coordinates that do not correspond to image evidences. In this work, we introduce Composer, a MLLM that leverages a novel visual grounding mechanism based on learned proxy-
tokens to promote faithful interpretability. These discrete symbolic pointers explicitly index the image latent space, allowing the model to manipulate visual regions as addressable, semantically manipulable sets. To rigorously validate our novel grounding mechanism, we constructed ComposerGCoT, a
dataset synthesized to enable holistic assessment of
reasoning consistency and grounding accuracy. Experimental results indicate that Composer achieves performance parity with its coordinate-based counterpart in final answer accuracy, while improving visual grounding accuracy by +9.0 points. By demonstrating that discrete proxy-
tokens capture spatial semantics more effectively than typical textual coordinates, we establish that visual grounding mechanisms with learnable semantic links represent a promising path toward trustworthy and reliable MLLMs.
Read the full abstract ▾
All authors Tom Hodemon, Mohamed Chaouch, Aboubacar Tuo, Angelique Loesch
Computer Vision
RT-DocLayout: Real-Time End-to-End Document Layout Analysis with Reading Order in the Wild
Cheng Cui, Tingquan Gao, Xueqing Wang et al. (+11 more)
Accurate document layout analysis remains a critical bottleneck for document parsing systems, due to the intricate coupling among heterogeneous document layout elements,
geometric distortions (\eg, paper warping and bending, perspective variations), and reading order within diverse layout structures. Existing approaches typically rely on fragmented multi-stage pipelines or computationally heavy generative
Transformer architectures, leading to error propagation and limited efficiency. In this paper, we present RT-DocLayout, a highly efficient end-to-end framework for document layout analysis, designed as a front-end for document parsing tasks. The proposed model unifies classification, detection, pixel-level
segmentation, and reading order prediction for layout elements within a single 33M-parameter architecture. Built upon the RT-DETR, our key contribution is a unified multi-task formulation within a single query-based decoder that simultaneously classifies, regresses bounding box, generates masks, and constructs relationship to reason reading order. By jointly learning
geometric and structural representations, RT-DocLayout introduces multi-task optimization that substantially improves robustness under real-world document distortions. Extensive experiments on public
benchmarks demonstrate state-of-the-art performance in document layout analysis while maintaining real-time inference speed(132.1 FPS). When coupled with downstream OCR engines, RT-DocLayout significantly improves full-document reconstruction quality, providing a scalable and practical foundation for real-world document intelligence systems.
Read the full abstract ▾
All authors Cheng Cui, Tingquan Gao, Xueqing Wang, Changda Zhou, Hongen Liu, Ting Sun, Yubo Zhang, Zelun Zhang, Jiaxuan Liu, Manhui Lin, Yue Zhang, Suyin Liang, Yiqing Xiang, Yi Liu
Computer Vision
VideoAgent: All-in-One Framework for Video Understanding and Editing
Hengji Zhou, Lingxuan Huang, Jian Wang et al. (+4 more)
Video editing has become essential in digital media creation, yet existing automated systems are restricted to short segment processing and domain-specific tasks. They face two critical limitations: i) inability to handle diverse
video comprehension and editing operations, and ii) lack of long-
video understanding for coherent narrative creation. We propose VideoAgent, an all-in-one agentic framework addressing these challenges through two key innovations. First, we develop automated
video shot creation with shot planning
agents for coherent narratives and cross-modal
retrieval for aligned visual content. Second, we design a multi-
agent orchestration framework integrating over thirty specialized editing
agents. Intent parsing filters relevant tools while textual-gradient graph optimization assembles complex editing pipelines. Extensive experiments on our newly-proposed VideoEdit
benchmark and public
datasets demonstrate VideoAgent's superiority over existing
multimodal LLMs and agentic systems. VideoAgent achieves 87-95% orchestration success rates while reducing API costs by 60%. Human evaluation across six
video categories shows VideoAgent produces professional-quality content approaching human-level performance, with ratings only 4% below human-created
videos. We release our code at https://github.com/HKUDS/VideoAgent.
Read the full abstract ▾
All authors Hengji Zhou, Lingxuan Huang, Jian Wang, Bing Zhou, Si Wu, Lianghao Xia, Chao Huang
Computer Vision
Ocean4D: Generative Underwater 4D Reconstruction via Medium-Aware Video Diffusion
Yuqiang Huang, Yuxi Wang, Junyu Dong et al. (+1 more)
Underwater 4D reconstruction remains challenging due to the coupling between degraded light transport in participating media and dynamic water variations. Most existing Methods are developed under in-air assumptions and do not explicitly account for
underwater absorption and backscatter. Additionally, near-static assumptions make these approaches sensitive to drifting particles and dynamic distractors , leading to unstable geometry and inconsistent cross-view results. To address these issues, we propose a generative framework for
underwater 4D reconstruction, named Ocean4D, which is built on two complementary components. Specifically, 4D-GCC constructs 4D geometrically consistent conditioning with improved cross-frame coverage, while the Medium-Aware Block performs implicit medium-aware denoising in the latent
diffusion process to stabilize
underwater appearance under absorption and scattering. Given a
monocular video and target cameras, our method generates
videos along the target trajectories while preserving global structure and cross-view consistency. Extensive experiments on both dynamic and static
underwater benchmarks demonstrate state-of-the-art performance on
underwater reconstruction.
Read the full abstract ▾
All authors Yuqiang Huang, Yuxi Wang, Junyu Dong, Zhaoxiang Zhang