Computer Vision · Cross-listed from cs.RO
EVA-Client: A Unified Data Collection, Inference, and Deployment Framework for Embodied Policies on Real Robots
Heqing Yang, Yang Yi, Liyao Wang et al. (+8 more)
We present EVA-Client, an open-source framework for deployment, data collection, and evaluation of trained manipulation policies on real
robots. Sitting between a policy server and the physical hardware, EVA-Client unifies the real-
robot stages of the policy iteration loop within a single codebase. It makes three contributions. First, a component-decoupled architecture in which
robot backends, inference strategies, and transport middlewares form an orthogonal grid: adding a
robot or a strategy touches only its own layer. Second, inspectable execution through Debug, Collect, and Eval workflows, with modes ranging from open-loop simulation to continuous real-time control. Third, every evaluation run doubles as a data collection, recording full rollouts in training-ready format alongside exhaustive logs and a side-by-side comparison viewer, so each evaluation feeds the next round of training rather than ending as an unrecorded impression. EVA-Client further consolidates major real-time inference strategies, synchronous and asynchronous execution, ACT-style temporal ensembling, Real-Time Chunking, and a naive-async ablation baseline, behind a single configuration surface.
Read the full abstract ▾
All authors Heqing Yang, Yang Yi, Liyao Wang, Linqing Zhong, Donglin Yang, Ruipu Wu, Zitong Bai, Fengjiao Chen, Manyuan Zhang, Linjiang Huang, Si Liu
Computer Vision
BiSLW: Bi-Spectral Latent Watermarking for Generative Diffusion Models
Aryan Pandit
Diffusion-based generative models have transformed visual content synthesis, yet they remain vulnerable to unauthorized usage and lack reliable attribution methods. Existing watermarking techniques often treat latent tensors as static spatial feature maps or depend on pixel-domain modification, and most do not explicitly leverage the internal frequency structure of the latent space for dual-band redundant
embedding, leaving them susceptible to the
stochastic nature of
diffusion and regeneration attacks. We introduce BiSLW, a trainable bi-spectral latent watermarking framework that jointly embeds aligned identity signals across complementary spectral bands of the decoded
diffusion latent using learned encoders and decoders, going beyond fixed-pattern frequency approaches. We leverage the inherent frequency structure of
diffusion latents to design a dual-band watermarking framework. Low-frequency components encode global semantics, while high-frequency components capture fine texture. We exploit this structure to embed watermarks across complementary spectral bands. The watermark is independently injected into both bands via learned encoders and recombined before decoding, ensuring it becomes intrinsic to the generative trajectory. Dual spectral decoders recover the watermark from each band, while a cross-band consistency constraint enforces
alignment between semantic and textural
embeddings. Experiments show that BiSLW achieves a strong balance between perceptual fidelity and robustness, improving PSNR by over 3 dB compared to prior latent
diffusion watermarking methods while preserving near-perfect bit accuracy under aggressive regeneration and common distortions, all with negligible computational overhead.
Read the full abstract ▾
Computer Vision
CLABTOOLKIT: An Open-Source Toolkit for Routine Processing, Manipulation, and Visualization of Neuroimaging Data
Yasser Alemán-Gómez, Nino Hervé and Patric Hagmann
Neuroimaging research requires manipulating heterogeneous data structures, including raw MRI volumes, volumetric parcellations, cortical surface meshes, tractograms, and connectivity
matrices, across tools with incompatible interfaces and file formats, forcing researchers to repeatedly re-implement routine but technically demanding operations. We present CLABTOOLKIT, an open-source Python package that consolidates these operations into a single, coherent framework by representing volumetric, surface, and streamline data as interoperable Python objects. Five core data structures (Parcellation, Surface, AnnotParcellation, Tractogram, and Connectome) encapsulate common neuroanatomical entities and provide consistent methods for loading, processing, and exporting data across standard neuroimaging formats (e.g., NIfTI, GIFTI, FreeSurfer annotations, TCK/TRK), including connectome generation from a parcellation and scalar-map projection onto tractogram streamlines. Complementary modules support BIDS
dataset management, FreeSurfer integration,
diffusion MRI processing, morphometric analysis, graph-theoretical network analysis, and GPU-accelerated multi-panel visualization via PyVista. The toolkit comprises 19 modules organised into six layers, exposing 13 object-oriented classes with 234 methods and 207 standalone functions, and a JSON-based configuration system enables workflow customization without code changes. Unlike existing neuroimaging libraries, which typically address these tasks separately, CLABTOOLKIT combines color and lookup-table management, parcellation manipulation, multi-surface visualization, and tractography utilities within a single framework. CLABTOOLKIT is compatible with Python 3.9-3.12 and released under the Apache 2.0 license. Source code, documentation, and example workflows are available at https://github.com/connectomicslab/clabtoolkit.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Federated Learning for Object Detection: Enabling Collaborative Drone Learning Without Centralizing Data
Daniel M. Jimenez-Gutierrez, Enrique Zuazua, Georgios Kellaris et al. (+3 more)
Object detection is a fundamental capability for AI-driven perception in safety-critical drone and edge-
vision systems, including disaster response, operational security environments, infrastructure monitoring and defense applications. Robust model performance in such environments depends on large, continuously updated
datasets. However, training high-performing detectors typically requires centralizing aerial imagery, which raises
privacy, regulatory, storage, and bandwidth challenges. This is especially problematic in distributed drone deployments, where visual data is generated onboard and is often impractical or undesirable to transfer to a centralized infrastructure. In this work, we apply
Federated Learning (FL) for object detection, enabling drones to improve a shared model while keeping image data local and private. We implement a
federated object detection pipeline using the Sherpa.ai FL platform on the KIIT-MiTA
dataset, and compare it with Single-drone and Centralized baselines using mean Average Precision (mAP) at IoU thresholds of 0.50 and 0.50-0.95. In our experiments, the proposed FL approach remains close to Centralized training while dramatically improving over Single-drone training, with the best lightweight model (YOLO26 nano), suitable for deployment even on very limited edge infrastructure, achieving relative gains of 52.89% and 67.80% in mAP@0.50 and mAP@0.50:0.95, respectively. These results show that FL enables scalable, high-performing, and
privacy-preserving object detection across distributed drone fleets without data centralization.
Read the full abstract ▾
All authors Daniel M. Jimenez-Gutierrez, Enrique Zuazua, Georgios Kellaris, Joaquin del Rio, Oleksii Sliusarenko, Xabi Uribe-Etxebarria
Computer Vision
SE-UNet: Singular Equivariant Imaging for Real-World Constrained Generation
Kanishk Awadhiya
While
diffusion models have revolutionized image synthesis, their application to real-world inverse problems is often hampered by the need for massive
datasets and the difficulty of imposing strict physical constraints. In this work, we introduce SE-UNet (Singular Equivariant UNet), a framework designed to solve ill-posed imaging tasks without extensive pre-training. By treating generation as an optimization problem constrained by
geometric equivariance ($D_4$ group) and singular value gating, SE-UNet effectively standardizes the solution space. We demonstrate that these strong inductive biases allow for state-of-the-art zero-shot inpainting results (80% missing pixels) on CIFAR-10. Our method surpasses Deep Image Prior (DIP) baselines by over 4 dB in PSNR and exhibits a characteristic "singular snap" convergence – rapidly locking into the signal
manifold. SE-UNet thus offers a data-efficient pathway for constrained generation, aligning with the ReALM-GEN goal of bridging theoretical priors with practical deployment.
Read the full abstract ▾
Computer Vision
WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory
Hanlin Wang, Hao Ouyang, Qiuyu Wang et al. (+10 more)
We present WorldDirector, a highly controllable
video world model framework designed for persistent dynamic object
memory and unrestricted viewpoint exploration. Unlike existing world models that entangle physical dynamics with pixel rendering and rely on continuous visual observation to sustain motion, our framework explicitly decouples semantic motion orchestration from visual generation. By leveraging an LLM to coordinate 3D trajectories with camera movements and subsequently employing these orchestrated trajectories as control signals for
video generation, our approach ensures strict physical logic and appearance stability, successfully preserving the exact visual identities of dynamic entities even when they re-enter the scene after prolonged periods out of view. Experimental results demonstrate that our method supports the synthesis of complex and extended events with unprecedented controllability and persistent dynamic object
memory. Project Page: https://worlddirector.github.io/
Read the full abstract ▾
All authors Hanlin Wang, Hao Ouyang, Qiuyu Wang, Wen Wang, Qingyan Bai, Ka Leong Cheng, Yue Yu, Yixuan Li, Yihao Meng, Zichen Liu, Yanhong Zeng, Yujun Shen, Qifeng Chen
Computer Vision
Alignment Is All You Need For X-to-4D Generation
Qiaowei Miao, Kehan Li, Yawei Luo et al. (+1 more)
Generative
diffusion models excel at synthesizing high-quality images,
videos, and 3D content under
multimodal control. However, arbitrary user-defined modality-to-4D (X-to-4D) generation remains challenging due to the high cost of constructing diverse
datasets and the limited scalability of existing methods. This paper presents Align4D, a flexible framework that translates any-modal input into coherent
video-3D pairs, using
video to guide 4D motion and 3D data to shape 4D geometry. Align4D introduces three key techniques: (1) Object Distance
Alignment, which searches
Video-Aligned and Multiview-Aligned Object Distances (VAOD/MAOD), respectively, to reconcile 4D renderings with
video and the priors of multiview
diffusion models; (2) Motion-Geometry Joint
Alignment, which constrains known and unknown views through synchronized
video and 3D inputs, ensuring consistent 4D generation; and (3) Asynchronous Optimization, which decouples Gaussian attribute and deformation network training to enhance motion and geometry fidelity. We further propose the X4D
dataset, which integrates
prompt, image,
video, and 3D data for benchmarking. Experiments on X4D and Consistent4D demonstrate that Align4D achieves state-of-the-art quality and consistency in X-to-4D generation. Project page: https://miaoqiaowei.github.io/Align4D/.
Read the full abstract ▾
All authors Qiaowei Miao, Kehan Li, Yawei Luo, Yi Yang
Computer Vision
PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation
Haofei Xu, Rundi Wu, Philipp Henzler et al. (+7 more)
State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent
diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space
Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image
tokens from a pre-trained DINOv3. Unlike existing latent
diffusion approaches, we train our
diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based
diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper
geometric structure and is more robust in highly ambiguous regions, such as transparent objects.
Read the full abstract ▾
All authors Haofei Xu, Rundi Wu, Philipp Henzler, Nikolai Kalischek, Michael Oechsle, Fabian Manhardt, Marc Pollefeys, Andreas Geiger, Federico Tombari, Michael Niemeyer
Computer Vision
From SRA to Self-Flow: Data Augmentation or Self-Supervision?
Dengyang Jiang, Mengmeng Wang, Harry Yang et al. (+1 more)
Representation
alignment has become an effective way to accelerate
diffusion transformer training and improve generation quality. Recent self-
alignment methods, such as SRA and Self-Flow, further remove the dependency on external pretrained encoders by constructing
alignment within the
diffusion model itself. However, the mechanism behind the improvement from SRA to Self-Flow, dual-time scheduling, remains under-examined: Self-Flow attributes its gain to interactions between
tokens at different noise levels, where cleaner
tokens help infer noisier ones. In this work, we revisit this explanation and ask whether the gain instead comes from data augmentation along the noise dimension. To disentangle these factors, we introduce
Attention Separation, which preserves the same dual-timestep input as Self-Flow while blocking
attention between
tokens assigned to different noise levels. Surprisingly, removing such interaction does not degrade performance and can even improve it, suggesting that the improvement from SRA to Self-Flow mainly comes from data augmentation. Furthermore,We show that
Attention Separation itself provides an augmentation effect by splitting a single image into multiple effective training parts to expand the training data. Based on these observations, we combine self-representation
alignment with dual-timestep and
attention-separation augmentation, and demonstrate the effectiveness of this design on ImageNet.
Read the full abstract ▾
All authors Dengyang Jiang, Mengmeng Wang, Harry Yang, Jingdong Wang
Computer Vision · Cross-listed from cs.CL
Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas
Yuxuan Li, Lingxi Xie, Xinyue Huo et al. (+6 more)
Long-form TV dramas present a formidable challenge for comprehensive
video understanding, where deciphering complex storyline often relies on speaker recognition, the task of accurately attributing each spoken utterance to its respective character. In this paper, we advance this field through two primary contributions. (1) We introduce DramaSR-532K, a large-scale
benchmark comprising 532K annotated dialogue lines across more than 900 unique characters, necessitating the integration of auditory, linguistic, and visual cues for speaker recognition. (2) We propose DramaSR-LRM, a robust approach built upon a large
reasoning model (LRM). DramaSR-LRM is designed to autonomously aggregate contextual evidence via
multimodal tool-use, synthesizing diverse inputs to achieve high-fidelity attribution. Experimental results demonstrate that DramaSR-LRM significantly outperforms existing baselines, particularly on short utterances where acoustic biometrics are inherently unreliable. All the data and code will be made publicly available at the project page: https://www.github.com/198808xc/DramaSR-LRM.
Read the full abstract ▾
All authors Yuxuan Li, Lingxi Xie, Xinyue Huo, Jihao Qiu, Jiacheng Shao, Pengfei Chen, Jiannan Ge, Kaiwen Duan, Qi Tian
Computer Vision · Cross-listed from cs.RO
Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots
Ling Xu, Borui Li, Hao Wu et al. (+8 more)
Embodied AI models now span
vision-
language-action (VLA) models and world-action models (WAMs), but practical deployment remains fragmented across model-specific Python stacks, backend assumptions, and
robot-side glue code, especially on heterogeneous edge devices. Existing inference runtimes are designed mainly for request-response serving and therefore do not satisfy the runtime contract of
embodied deployment: multi-rate execution inside closed-loop control, latency-first batch-1 inference on heterogeneous hardware, and extensible
embodied interfaces beyond fixed
token I/O. We present
Embodied$.$cpp, a portable C++ inference runtime for
embodied models. Based on an architectural analysis of representative VLA models and WAMs,
Embodied$.$cpp captures a shared execution path and organizes it into five layers: input adapters, sequence builders, backbone execution, head plugins, and deployment adapters. The runtime provides modular multi-rate execution, latency-first fused inference, and extensible operator and I/O support, enabling deployment across heterogeneous devices,
robots, and simulators through one backend abstraction. We evaluate
Embodied$.$cpp on three VLA and two WAM models, using normalized comparisons across Python and C++ quantization configurations. Overall,
Embodied$.$cpp achieves 1.05x-2.70x inference speedups and 7%-77% lower VRAM relative to Python baselines, while maintaining near-baseline success for most configurations. These results show that
Embodied$.$cpp improves deployment efficiency while preserving high control quality across diverse
embodied model architectures. Project Link: https://github.com/SEU-PAISys/
Embodied.cpp
Read the full abstract ▾
All authors Ling Xu, Borui Li, Hao Wu, Chuyu Han, Xiangyu Li, Mohan Hua, Shiqi Jiang, Ting Cao, Chuanyou Li, Sheng Zhong, Shuai Wang
Computer Vision
Seek to Segment: Active Perception for Panoramic Referring Segmentation
Song Tang, Shuming Hu, Xincheng Shuai et al. (+2 more)
Existing referring
segmentation models passively process static images captured from fixed perspectives, limiting their applicability in
Embodied AI, where
agents must perform active perception in the continuous 360$^\circ$ environments. To bridge this gap, we introduce a novel task: Active Panoramic Referring
Segmentation (APRS). In this setting, an
agent is required to adjust its viewing direction ($\Delta\theta, \Delta\phi$) to explore the 360$^\circ$ environment, seeking the object specified by a user instruction for
segmentation. To tackle this challenging task, we propose PanoSeeker, a
memory-augmented
agent for efficient APRS. Rather than relying on heuristic scanning, PanoSeeker integrates a
Vision-
Language Model (VLM) with EgoSphere, an explicit spatial visual
memory. By progressively integrating sequential local observations into a unified 360$^\circ$ representation, EgoSphere enables the
agent to plan efficient and non-redundant search trajectories. Once the target is found, the
agent performs active viewpoint
alignment and outputs the
segmentation mask. Furthermore, we curate an expert-annotated search trajectory
dataset with
memory timelines for Supervised Fine-Tuning, followed by Reinforcement Learning post-training to explicitly optimize PanoSeeker's exploration efficiency. Extensive experiments on our newly established APRS
benchmark demonstrate that PanoSeeker achieves superior search efficiency and
segmentation accuracy, significantly outperforming adapted state-of-the-art baselines.
Read the full abstract ▾
All authors Song Tang, Shuming Hu, Xincheng Shuai, Henghui Ding, Yu-Gang Jiang
Computer Vision
Towards Robustness against Typographic Attack with Training-free Concept Localization
Bohan Liu, Wenqian Ye, Guangzhi Xiong et al. (+3 more)
Models trained via Contrastive
Language-Image Pretraining (CLIP) serve as the foundational
vision encoders for most modern Large
Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving. To achieve interpretable and effective robustness against TA, we propose a novel, training-free mechanistic interpretability method. Our method provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual
attention heads. Through probabilistic analysis and
circuit mining, we isolate specific
Vision Transformer (ViT) components that disproportionately encode lexical information, thereby identifying the mechanistic source of TA. We further show that simple interventions applied directly to the identified
circuits, without any additional training, can substantially improve robustness against Typographic Attacks in object classification. These interventions, such as selective adjustment of
attention weights, also outperform both supervised and training-free defense methods. Our experiments demonstrate that applying the proposed intervention to the
vision encoders of several state-of-the-art LVLMs yields substantial gains in Visual Question Answering accuracy under Typographic Attack interference on RIO-Bench. These results confirm both the efficacy and the generalizability of our mechanistic approach. Code is released at https://github.com/Liu-524/SamplingTAR.
Read the full abstract ▾
All authors Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He, Sanchit Sinha, Aidong Zhang
Computer Vision · Cross-listed from cs.CL
Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning
Liyan Tang, Fangcong Yin and Greg Durrett
Large
vision-
language models can reason over
multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT
reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images. To address this issue, we propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection. First, we randomly mask trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions rather than making early mistakes. Second, we introduce buffered roll-ins from an experience replay buffer to expose the model to diverse failure states that it must learn to correct. We evaluate our approach on visual grounding tasks involving tables and charts, as well as spatial
navigation benchmarks. While off-the-shelf and conventionally fine-tuned models degrade substantially under distribution shift, our method substantially improves average out-of-distribution accuracy over standard RL and reflection-oriented fine-tuning baselines by using self-reflection effectively.
Read the full abstract ▾
Computer Vision
GeoMix: Descriptor-Free Visual Localization via Global Context and Multi-Detector Training
Yejun Zhang, Xinjue Wang, Zihan Wang et al. (+2 more)
Descriptor-free visual localization eliminates high-dimensional descriptor storage, preserves scene
privacy, and simplifies map maintenance, yet its accuracy still lags far behind descriptor-based pipelines. We identify this gap to insufficient
geometric discriminability in geometry-only matching. Without visual appearance, current methods underutilize local geometry cues, lack the global context among keypoints, and overfit to a single keypoint detector. We further observe that descriptor-free matching naturally enables multi-detector training, as heterogeneous keypoints can be optimized in a shared geometry-only space without aligning descriptor spaces. Building on these insights, we propose GeoMix, a descriptor-free 2D-3D matching framework that strengthens
geometric discriminability at three levels. Locally, directional and distance-aware
embeddings enrich neighborhood aggregation with fine-grained spatial structure. Globally, learnable context nodes aggregate and redistribute scene-wide information via cross-
attention to resolve ambiguities beyond local receptive fields. At the training level, Mix-Training exploits this detector-agnostic geometry space to learn representations across multiple keypoint detectors. Extensive experiments on MegaDepth, Cambridge Landmarks, 7Scenes, and Aachen Day-Night show that GeoMix sets a new state of the art among descriptor-free methods, reducing 75th-percentile rotation error by 89% and translation error by up to 90% over the previous best, while generalizing zero-shot to unseen detectors and narrowing the gap to descriptor-based pipelines. Code is available at $\href{https://github.com/YejunZhang/Geomix}{\text{this links}}$.
Read the full abstract ▾
All authors Yejun Zhang, Xinjue Wang, Zihan Wang, Esa Rahtu, Juho Kannala
Computer Vision
Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning
Xuehui Wang, Xuankun Yang and Wei Shen
Visual
token pruning is a crucial strategy for accelerating VLMs by compressing redundant image patches, yet existing methods often fail to preserve critical cues under dense instructions and fine-grained queries. In this paper, we investigate this failure and identify two underlying bottlenecks: the widespread dispersion of textual noise that corrupts dense cross-modal scoring, and the feature fragmentation inherent to standard
token selection. To address these issues, we propose
Entropy-Aware Dense
Pruning (EADP), a framework that reformulates
pruning as a structured compression problem. EADP first leverages statistical
entropy to quantify and filter out textual noise, yielding a robust, fine-grained instruction relevance score. Subsequently, instead of naive Top-K selection, EADP casts
token selection as a submodular maximization problem with a spatial prior, explicitly ensuring a holistic and non-redundant visual representation. Extensive experiments demonstrate that EADP improves the accuracy-efficiency trade-off of VLMs, robustly preserving fine-grained visual cues under strict
token budgets while achieving SoTA performance on challenging
multimodal benchmarks.
Read the full abstract ▾
Computer Vision
EAGLE-360: Embodied Active Global-to-Local Exploration in 360$^\circ$
Jingtao Xu, Zizhuo Lin, Jianwen Sun et al. (+2 more)
While
Multimodal Large
Language Models (MLLMs) have demonstrated exceptional capabilities in standard visual understanding, adapting them for active visual search in 360$^\circ$ panoramic environments exposes fundamental limitations. Specifically, standard MLLMs struggle to effectively model inherent panoramic properties, such as severe polar distortion and continuous cylindrical
topologies, which significantly degrades target detection accuracy. Consequently, existing panoramic search methods attempt to compensate by relying heavily on fragmented local viewpoints. Burdened by rigid initialization and a lack of global panoramic priors, these approaches suffer from myopic, inefficient exploration and struggle with robust error recovery when targets are out of view. To overcome these challenges, we propose EAGLE-360, a novel
Embodied Active Global-to-Local Exploration framework. Rather than performing exhaustive local searches, EAGLE-360 leverages global priors to establish an initial holistic perspective, iteratively
reasoning and progressively narrowing the search space. Architecturally, we adapt RoPE Rolling, a coordinate-shifting positional encoding mechanism, to seamlessly model the continuous
topologies of panoramas. To facilitate this paradigm, we construct the large-scale EAGLE-360
dataset, comprising 14,000+ 4K panoramas and 70,000+ rounds of high-quality VQA dialogues. By employing a training pipeline that integrates Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO), we effectively elicit complex spatial
reasoning and tool-calling capabilities. Extensive experiments demonstrate that EAGLE-360 establishes a new state-of-the-art for 360$^\circ$ visual search, achieving nearly an 8-fold increase in accuracy over the base model while significantly enhancing exploration efficiency.
Read the full abstract ▾
All authors Jingtao Xu, Zizhuo Lin, Jianwen Sun, Yi Yang, Yawei Luo
Computer Vision
Interpretation-Oriented Cloud Removal via Observation-Anchored Residual Flow with Geo-Contextual Alignment
Ziyao Wang, Maonan Wang, Yucheng He et al. (+5 more)
Cloud removal (CR) is essential for optical remote sensing, serving as a prerequisite for reliable downstream interpretation, such as semantic
segmentation and change detection. However, existing CR approaches often prioritize visual realism while overlooking their impact on subsequent analytical tasks, leading to semantic drift and degraded downstream performance. To address this issue, we propose Geo-Anchored Cloud Removal (GACR), a unified framework that jointly ensures faithful reconstruction and robust interpretability. At its core, GACR incorporates Observation-Anchored Residual Flow (OAR-Flow), which reformulates CR as a physically grounded residual inversion process. By anchoring the generative trajectory to the cloudy observation rather than pure noise, OAR-Flow enables fast, stable, and faithful reconstruction. To further preserve semantic structures critical for downstream interpretation, GACR integrates Geo-Contextual Prior
Alignment (GCPA) to constrain the reconstruction within a semantic
manifold induced by a
Vision Foundation Model (VFM). Consequently, GACR strictly maintains the spatial-semantic integrity of complex landscapes. Extensive experiments across six CR
datasets and twelve downstream tasks demonstrate that GACR produces superior reconstruction quality while consistently improving downstream task accuracy. The code is available at https://github.com/wzy6055/GACR.
Read the full abstract ▾
All authors Ziyao Wang, Maonan Wang, Yucheng He, Xianping Ma, Ziyi Wang, Hongyang Zhang, Yirong Cheng, Man-on Pun
Computer Vision
OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers
Donghyun Lee, Jitesh Chavan, Duy Nguyen et al. (+5 more)
Diffusion transformers (DiTs) achieve state-of-the-art image and
video generation, but their multi-step sampling and growing parameter count make inference expensive. Post-training quantization (PTQ) is the natural remedy, yet DiT activations shift across timesteps,
prompts, and guidance branches, forcing prior methods to re-fit calibration data for every new checkpoint or modality. We present OrbitQuant, a data-agnostic weight-activation quantizer that bypasses range estimation by quantizing in a normalized, rotated basis. In this basis, a randomized permuted block-Hadamard (RPBH) rotation concentrates each coordinate around one fixed, known marginal regardless of the input, so a single Lloyd-Max codebook serves all timesteps,
prompts, and layers of a given input dimension. We extend the same quantizer to weight rows offline, absorbing the rotation into the weights so that it cancels inside each linear layer and only a forward rotation on the activations remains at runtime. The same recipe transfers from image to
video with no per-modality tuning. Across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, it sets the state of the art for PTQ at several low-bit settings. It also pushes PTQ of image
diffusion transformers to W2A4 with usable generation quality.
Read the full abstract ▾
All authors Donghyun Lee, Jitesh Chavan, Duy Nguyen, Sam Huang, Liming Jiang, Priyadarshini Panda, Timo Mertens, Saurabh Shukla
Computer Vision
MARVEL: Margin-Aware Robust von Mises-Fischer Expert Learning for Long-Tailed Out-of-Distribution Detection
A.S. Anudeep and Vaanathi Sundaresan
For clinical deployment, it is essential that automated diagnostic systems remain reliable when confronted with previously unseen cases, yet deep models routinely misclassify out-of-distribution (OOD) inputs with high confidence, underscoring the need for more robust OOD detection methods. Although substantial effort has been devoted to improving model robustness, most of the existing literature assumes balanced
datasets, evaluates OOD detection on coarse or non-clinical OOD sources, or lacks comprehensive assessment across diverse OOD scenarios. To address the gaps, we propose a novel methodology trained on diverse and imbalanced medical
datasets and evaluated across a clinically reflective OOD spectrum. Our framework comprises three key components: (1) a Nonlinear von Mises-Fisher (NvMF) classifier capable of learning non-linear decision boundaries, with theoretical proof of its asymptotic connection to cosine classifiers; (2) a multi-expert framework in which margin-aware NvMF classifiers specialise in different regions of label distribution to better handle imbalance; and (3) an outlier expert trained explicitly to distinguish inlier from outlier data, thereby strengthening OOD detection. Evaluation on RFMiD, ISIC2019, and NCTCRC
datasets demonstrates consistent improvements over state-of-the-art methods, achieving mean FPR95 reductions of 8.45%, 13.02%, and 36.90% respectively. These gains are further supported by comprehensive ablations that validated the contributions of each component. This enables reliable identification of unfamiliar cases for deferral to clinicians, supporting safer AI-assisted diagnosis in real-world workflows. Our code is available at https://github.com/redboxup/MARVEL.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Self-Auditing Residual Drifting for Pathology-Preserving Accelerated Knee MRI
Qing Lyu, Jianxu Wang, Mohammad Kawas et al. (+2 more)
Accelerated
magnetic resonance imaging reduces acquisition time, but reconstruction from undersampled k-space can blur diagnostically relevant structures or introduce failures that are not captured by global image metrics. We propose SA-RDM-DC, a Self-
Auditing Residual generative Drifting Model with Data Consistency for accelerated knee MRI. The method adapts the newly proposed generative drifting paradigm to accelerated MRI by training a physics-conditioned drift field from the zero-filled reconstruction toward the fully sampled residual correction. It predicts image- and missing-k-space residual corrections, enforces data consistency with acquired k-space, uses frequency-aware and residual drifting supervision to recover fine detail, and produces dense error maps and slice-level risk scores in the same inference pass. We evaluate SA-RDM-DC on multi-coil fastMRI knee data at acceleration factors of 4, 8, and 12, with fastMRI+ pathology annotations for region-level and classifier-based task preservation, and on SKM-TEA for zero-shot and fine-tuned protocol-shift evaluation. Compared with zero-filled reconstruction, UNet-image-SENSE, DC-UNet, Score-
Diffusion, ELF-Diff, SENSE-VarNet, and MoDL baselines, SA-RDM-DC achieves the highest SSIM across fastMRI acceleration factors while retaining subsecond per-slice inference and avoiding the long sampling time of iterative
diffusion baselines. In pathology-aware analysis, SA-RDM-DC preserves lesion-region structural fidelity and reduces meniscus prediction instability. Its self-
auditing scores strongly identify high-error reconstructions on fastMRI and partially transfer as a selective-review signal under SKM-TEA protocol shift. These results support reconstruction evaluation that jointly considers image fidelity, pathology preservation, runtime, and case-specific reliability.
Read the full abstract ▾
All authors Qing Lyu, Jianxu Wang, Mohammad Kawas, Ge Wang, Christopher T. Whitlow
Computer Vision
Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs
Francesca Pistilli, Simone Alberto Peirone and Giuseppe Averta
Understanding human behavior while interacting with the surrounding world is crucial for many applications of
embodied AI. First-person
videos are particularly informative for this problem, as they well capture how activities reshape the scene over time. However, existing approaches often rely on implicit visual or
language-aligned representations, disregarding structured
reasoning over the scene dynamic. We argue that explicit, compositional and editable representations of human-environment interactions can play a crucial role for rich grounded activity understanding. To this end, we introduce SG-Ego, a large scale annotation set extending Ego4D with spatio-temporal scene graphs, where relations triplets are consolidated over time into explicit time-evolving descriptions of the scene state. To reason over this representation, we propose GLEN, a graph-based model that operates over scene graph sequences to both align them with textual actions and model their temporal
evolution. In addition, we formulate the activity-driven graph-edit
forecasting (A-GEF) problem, a novel task that casts scene dynamics as a sequence of structured transformations conditioned on ongoing actions, enabling explicit
reasoning about how scenes change over time. We validate our approach across multiple downstream tasks, spanning
retrieval benchmarks as EgoMCQ and EgoCVR, as well as long-horizon
reasoning benchmarks as EXPLORE-Bench and the newly introduced A-GEF. GLEN achieves strong results compared to raw
video baselines and it excels in
reasoning settings, typically addressed only with MLLMs, while enabling controllable and structured predictions of scene dynamics driven by human activities. We believe our results establish spatio-temporal scene graphs, together with models that reason over them, as strong compositional and interpretable representations for
video understanding and potentially beyond.
Read the full abstract ▾
Computer Vision
Wavelet-Guided Semantic Signal Compensation for Inversion-Free Image Editing
Anqi Tang, Wenhao Sun and Zhaoqiang Liu
Text-guided image editing aims to modify visual content according to a target
prompt while preserving the background. Recent inversion-free image editing frameworks such as FlowEdit have demonstrated strong editing capability without requiring inversion. Empirically, FlowEdit can achieve substantial semantic changes under appropriate hyperparameter settings. However, we observe that under certain global attribute shifts, the editing trajectory may not effectively move away from the source distribution in the early timesteps. Our analysis suggests that in the high-noise regime, the dominant
manifold-seeking flow toward the data
manifold can reduce the influence of the text-conditioned direction, leading to limited global modification while background structures remain only moderately preserved. Inspired by this observation, we propose an inversion-free, frequency-aware semantic compensation strategy that strengthens the effective signal in the early stage of generation, while maintaining structural consistency in the background. The proposed method improves global editing capacity without sacrificing background fidelity.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
LIME: Learning Intent-aware Camera Motion from Egocentric Video
Boyang Sun, Jiajie Li, Yung-Hsu Yang et al. (+6 more)
Autonomous
robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent. While
vision-
language navigation translates instructions to base motion and
vision-
language-action policies map instructions to manipulation actions,
vision-
language conditioned camera motion remains comparatively underexplored as a first-class action. We formulate
vision-
language conditioned camera motion generation: given a current RGB observation and a free-form natural-
language intent, predict a relative target camera pose for the next observation. The challenge is that useful viewpoint changes depend on latent perceptual intent, ranging from coarse spatial moves to fine-grained inspection or occlusion revealing. To model this structure, we mine multi-intent camera-motion supervision from
egocentric video via hindsight labeling, pairing each relative SE(3) target pose with plausible intents and observation-gain descriptions inferred from the reached view. We propose LIME, a
vision-
language camera-motion generator that combines an autoregressive observation-gain output with a continuous flow-matching pose head. This design lets the model jointly predict what the next view should reveal while representing multi-hypothesis target views. Across viewpoint-prediction experiments and downstream robotic tasks, we show that LIME learns intent-conditioned camera motion from passive
egocentric video, turning ordinary human recordings into supervision for a reusable active-perception primitive that supports general-purpose viewpoint selection, manipulation,
embodied question answering, and mid-distance
navigation.
Read the full abstract ▾
All authors Boyang Sun, Jiajie Li, Yung-Hsu Yang, Chenyangguang Zhang, Tim Engelbracht, Sunghwan Hong, Cesar Cadena, Marc Pollefeys, Hermann Blum
Computer Vision · Cross-listed from cs.AI
Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environments
Xianhui Meng, Zirui Song, Yuchen Zhang et al. (+10 more)
Large
Language Models (LLMs) have demonstrated remarkable capabilities in 3D indoor synthesis for Manhattan environments. However, existing methods often fail to capture plausible object layout patterns in non-Manhattan settings, primarily because they struggle to model non-orthogonal spatial relationships, leading to high
geometric violations and low physical fidelity. To address this challenge, we propose SPG-Layout, a novel text-driven framework designed to generate physically plausible indoor scenes within complex non-Manhattan environments. Specifically, we first utilize statistical priors of object distributions to guide the training process, enhancing environmental understanding and fidelity. Furthermore, mirroring human design workflows, we adopt a hierarchical layout strategy that prioritizes the placement of large objects, thereby substantially minimizing layout violations. By synergizing these components, SPG-Layout achieves a balanced optimization of semantic realism and physical plausibility. To evaluate performance in these complex settings, we constructed a new
benchmark comprising 500 diverse non-Manhattan environments. Extensive experiments demonstrate that SPG-Layout consistently and significantly outperforms existing methods across both Manhattan and non-Manhattan environments. The code will be publicly released.
Read the full abstract ▾
All authors Xianhui Meng, Zirui Song, Yuchen Zhang, Li Zhang, Yongxuan Lv, Xiuying Chen, Kun Wang, Yan Luo, Kai Chen, Hangjun Ye, Long Chen, Jun Liu, Xiaoshuai Hao
Computer Vision
Mask-supervised Object-centric Representation Learning with LeJEPA
Jakob Geusen and Ender Konukoglu
Self-supervised image encoders deliver strong features for downstream tasks but need many images for training. A natural remedy to counter this is to make each image count for more. A scene contains many objects, and given masks from human annotators or an off-the-shelf
segmentation model, pre-training can focus on aligning per-object rather than image-wide representations, extracting more signal from every image. Existing mask-supervised methods do this through reconstruction or contrastive losses that leverage negative objects. We instead use two separate projection spaces for the
alignment. In a semantic space, per-object representations from different views are aligned. To avoid collapse, instead of using negative objects, which requires category definitions, we extend the negative-free LeJEPA objective and show that its distributional anti-collapse regularizer ports naturally from whole images to the variable-sized set of objects in a scene. In an instance space, a contrastive loss separates per-object representations from their context and co-occurring instances, including those of the same category. To separate object representations from their context, we copy objects and paste them into other contexts, where each pasted copy serves as an additional view of the original object. Trained on COCO with ground-truth masks, our method outperforms image-level and mask-guided baselines on tracking (DAVIS), classification (ImageNet-1k) and re-identification (NAVI), matches the best of them on semantic
segmentation (ADE20k) and keeps its lead over image-level LeJEPA and a supervision-matched alternative on COCO fractions down to 256 images.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
ACID: Action Consistency via Inverse Dynamics for Planning with World Models
Gawon Seo, Dongwon Kim and Suha Kwak
Decision-time planning with action-conditioned world models has become a popular paradigm for
embodied control. However, the standard planning cost judges a candidate solely by how close its predicted terminal state lies to the goal, leaving the realizability of the intermediate transitions unchecked–a predicted trajectory can look convincing while the environment rollout drifts away from it. In this paper, we propose ACID, a decision-time planning framework that introduces cycle action consistency: the action inferred backward from a predicted transition by an inverse dynamics model should recover the one that was conditioned on. We fold this per-step residual into the planning cost via a scale-invariant adaptive weight. Across four action-conditioned world models and eight tasks encompassing object manipulation and articulated control in simulation, visual
navigation, and real-
robot manipulation, ACID consistently improves planning and matches the baseline's accuracy with substantially less planning compute.
Read the full abstract ▾
Computer Vision
Show Me Examples: Inferring Visual Concepts from Image Sets
Nick Stracke, Kolja Bauer, Stefan Andreas Baumann et al. (+3 more)
Vision-
language models (VLMs) can follow complex textual instructions, yet they struggle to reason from purely visual context. In particular, current models fail to infer shared concepts from sets of example images and apply them to new inputs. We introduce Visual Concept Inference from Sets (VICIS), a task that evaluates this capability. Given a small context set of images sharing a concept and a query image, the model must generate new images that preserve the context-defined concept while remaining consistent with the query. We show that state-of-the-art VLMs perform poorly on this task, often ignoring the visual context or defaulting to biased generations. To address this gap, we propose a training framework and architecture that learn to infer visual concepts from image sets and extract concept-specific
embeddings from queries. Experiments on synthetic data and large-scale ImageNet/WordNet data show that our model generates more accurate and diverse outputs and generalizes to unseen concepts and modalities such as sketches.
Read the full abstract ▾
All authors Nick Stracke, Kolja Bauer, Stefan Andreas Baumann, Miguel Angel Bautista, Josh Susskind, Björn Ommer
Computer Vision
Transformer Geometry Observatory TGO-II: Representational Similarity Observatory
Kaustubh Kapil and Kishor P. Upla
While
Vision Transformers have achieved remarkable success across computer
vision and
language applications, the
geometric evolution of their internal representations throughout training remains insufficiently understood. Existing analyses primarily focus on
attention mechanisms and downstream performance, leaving the
evolution of representation geometry largely unexplored. In this work, we present
Transformer Geometry Observatory-II (TGO-II), a representation geometry analysis framework designed to investigate how
Transformer representations evolve during supervised training. TGO-II analyzes
Vision Transformer (ViT-Small/16) representations using Centered Kernel
Alignment (CKA), Singular Vector Canonical Correlation Analysis (SVCCA), Two-Nearest Neighbor Intrinsic Dimensionality (TwoNN-ID), and
token covariance analysis. Our experiments reveal three key observations. First, both CKA and SVCCA progressively decrease throughout training, indicating increasing representational specialization across
Transformer layers. Second, intrinsic dimensionality consistently increases before stabilizing, suggesting progressive expansion of the representation
manifold into a larger set of locally accessible degrees of freedom. Third,
token covariance and coupling analyses demonstrate that strong
token interaction structure persists throughout training, challenging the hypothesis that increasing representational complexity arises primarily from progressive
token independence. These findings suggest that representation complexity and layer specialization emerge simultaneously during training.
Manifold expansion appears to occur without
token decoupling. Together, these observations motivate a new hypothesis in which
Vision Transformers increase representational complexity through progressively richer transformations while preserving strong
token interaction structure during learning.
Read the full abstract ▾
Computer Vision
Representation Distribution Matching for One-Step Visual Generation
Lan Feng, Wuyang Li, Eloi Zablocki et al. (+2 more)
We elucidate the design space of Representation Distribution Matching (RDM), our name for the paradigm that trains a one-step image generator by matching generated and reference feature distributions under frozen pretrained encoders. We identify two design axes, how the distributions are compared and the representations they are compared in, and controlled studies along them yield three findings. First, the classical MMD, which could not train convincing generators a decade ago, becomes a strong and scalable objective once estimated right. Second, the generated batch is then the operative variable, with an optimum above 2048, far beyond customary batch sizes. Third, any single representation can be gamed, driven below the real score while images stay visibly fake, so we match against a balanced battery of encoders and evaluate with SW_r14, a Sliced-Wasserstein distance over 14 encoders that is independent of the training loss and resists gaming. Combining the preferred choices yields improved RDM (iRDM): it sets the one-step state of the art on ImageNet at SW_r14 1.30, corroborated by PickScore, a human-preference proxy our objective never optimizes, which prefers it over the prior best one-step generator on 71.2% of matched samples. The same recipe post-trains the four-step FLUX.2 [klein] into a one-step generator, surpassing the four-step version on GenEval, 0.826 to 0.794, and on PickScore, 22.76 to 22.58, in 90 H200 GPU-hours. Project page: https://alan-lanfeng.github.io/rdm/.
Read the full abstract ▾
All authors Lan Feng, Wuyang Li, Eloi Zablocki, Matthieu Cord, Alexandre Alahi