Computer Vision
Active Measurement of Two-Point Correlations
Max Hamilton, Daniel Sheldon and Subhransu Maji
Two-point correlation functions (2PCF) are widely used to characterize how points cluster in space. In this work, we study the problem of measuring the 2PCF over a large set of points, restricted to a subset satisfying a property of interest. An example comes from astronomy, where scientists measure the 2PCF of
star clusters, which make up only a tiny subset of possible sources within a
galaxy. This task typically requires careful labeling of sources to construct catalogs, which is time-consuming. We present a human-in-the-loop framework for efficient estimation of 2PCF of target sources. By leveraging a pre-trained classifier to guide sampling, our approach adaptively selects the most informative points for human annotation. After each annotation, it produces unbiased estimates of pair counts across multiple distance bins simultaneously. Compared to simple Monte Carlo approaches, our method achieves substantially lower variance while significantly reducing annotation effort. We introduce a novel unbiased
estimator, sampling strategy, and confidence interval construction that together enable scalable and statistically grounded measurement of two-point correlations in astronomy
datasets.
Read the full abstract ▾
Computer Vision
Hierarchical Mesh Transformers with Topology-Guided Pretraining for Morphometric Analysis of Brain Structures
Yujian Xiong, Mohammad Farazi, Yanxi Chen et al. (+8 more)
Representation learning on large-scale unstructured volumetric and surface meshes poses significant challenges in neuroimaging, especially when models must incorporate diverse vertex-level morphometric descriptors, such as cortical thickness,
curvature, sulcal depth, and myelin content, which carry subtle disease-related signals. Current approaches either ignore these clinically informative features or support only a single mesh
topology, restricting their use across imaging pipelines. We introduce a hierarchical
transformer framework designed for heterogeneous mesh analysis that operates on spatially adaptive tree partitions constructed from simplicial complexes of arbitrary order. This design accommodates both volumetric and surface discretizations within a single architecture, enabling efficient multi-scale
attention without
topology-specific modifications. A feature projection module maps variable-length per-vertex clinical descriptors into the spatial hierarchy, separating
geometric structure from feature dimensionality and allowing seamless integration of different neuroimaging feature sets. Self-supervised pretraining via masked reconstruction of both coordinates and morphometric channels on large unlabeled cohorts yields a transferable encoder backbone applicable to diverse downstream tasks and mesh modalities. We validate our approach on Alzheimer's disease classification and amyloid burden prediction using volumetric
brain meshes from ADNI, as well as focal cortical dysplasia detection on cortical surface meshes from the MELD
dataset, achieving state-of-the-art results across all
benchmarks.
Read the full abstract ▾
All authors Yujian Xiong, Mohammad Farazi, Yanxi Chen, Wenhui Zhu, Xuanzhao Dong, Natasha Lepore, Yi Su, Raza Mushtaq, Stephen Foldes, Andrew Yang, Yalin Wang
Computer Vision
Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D
Daniel DeTone, Tianwei Shen, Fan Zhang et al. (+4 more)
Detecting and localizing objects in space is a fundamental computer
vision problem. While much progress has been made to solve 2D object detection, 3D object localization is much less explored and far from solved, especially for open-world categories. To address this research challenge, we propose Boxer, an
algorithm to estimate static 3D bounding boxes (3DBBs) from 2D open-vocabulary object detections, posed images and optional depth either represented as a sparse point cloud or dense depth. At its core is BoxerNet, a
transformer-based network which lifts 2D bounding box (2DBB) proposals into 3D, followed by multi-view fusion and
geometric filtering to produce globally consistent de-duplicated 3DBBs in metric world space. Boxer leverages the power of existing 2DBB detection
algorithms (e.g. DETIC, OWLv2, SAM3) to localize objects in 2D. This allows the main BoxerNet model to focus on lifting to 3D rather than detecting, ultimately reducing the demand for costly annotated 3DBB training data. Extending the CuTR formulation, we incorporate an aleatoric uncertainty for robust regression, a median depth patch encoding to support sparse depth inputs, and large-scale training with over 1.2 million unique 3DBBs. BoxerNet outperforms state-of-the-art baselines in open-world 3DBB lifting, including CuTR in
egocentric settings without dense depth (0.532 vs. 0.010 mAP) and on CA-1M with dense depth available (0.412 vs. 0.250 mAP).
Read the full abstract ▾
All authors Daniel DeTone, Tianwei Shen, Fan Zhang, Lingni Ma, Julian Straub, Richard Newcombe, Jakob Engel
Computer Vision
Integration of Object Detection and Small VLMs for Construction Safety Hazard Identification
Muhammad Adil, Mehmood Ahmed, Muhammad Aqib et al. (+3 more)
Accurate and timely identification of construction hazards around workers is essential for preventing workplace accidents. While large
vision-
language models (VLMs) demonstrate strong contextual
reasoning capabilities, their high computational requirements limit their applicability in near real-time construction hazard detection. In contrast, small
vision-
language models (sVLMs) with fewer than 4 billion parameters offer improved efficiency but often suffer from reduced accuracy and
hallucination when analyzing complex construction scenes. To address this trade-off, this study proposes a detection-guided sVLM framework that integrates object detection with
multimodal reasoning for contextual hazard identification. The framework first employs a YOLOv11n detector to localize workers and construction machinery within the scene. The detected entities are then embedded into structured
prompts to guide the
reasoning process of sVLMs, enabling spatially grounded hazard assessment. Within this framework, six sVLMs (Gemma-3 4B, Qwen-3-VL 2B/4B, InternVL-3 1B/2B, and SmolVLM-2B) were evaluated in zero-shot settings on a curated
dataset of construction site images with hazard annotations and explanatory rationales. The proposed approach consistently improved hazard detection performance across all models. The best-performing model, Gemma-3 4B, achieved an F1-score of 50.6%, compared to 34.5% in the baseline configuration. Explanation quality also improved significantly, with BERTScore F1 increasing from 0.61 to 0.82. Despite incorporating object detection, the framework introduces minimal overhead, adding only 2.5 ms per image during inference. These results demonstrate that integrating lightweight object detection with small VLM
reasoning provides an effective and efficient solution for context-aware construction safety hazard detection.
Read the full abstract ▾
All authors Muhammad Adil, Mehmood Ahmed, Muhammad Aqib, Vicente A. Gonzalez, Gaang Lee, Qipei Mei
Computer Vision
OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models
Ali Aliev, Kamil Garifullin, Nikolay Yudin et al. (+5 more)
In a rapidly growing field of model training there is a constant practical interest in parameter-efficient fine-tuning and various techniques that use a small amount of training data to adapt the model to a narrow task. However, there is an open question: how to combine several adapters tuned for different tasks into one which is able to yield adequate results on both tasks? Specifically, merging subject and style adapters for generative models remains unresolved. In this paper we seek to show that in the case of orthogonal fine-tuning (OFT), we can use structured orthogonal parametrization and its
geometric properties to get the formulas for training-free adapter merging. In particular, we derive the structure of the
manifold formed by the recently proposed Group-and-Shuffle ($\mathcal{GS}$) orthogonal
matrices, and obtain efficient formulas for the geodesics approximation between two points. Additionally, we propose a $\text{spectra restoration}$ transform that restores spectral properties of the merged adapter for higher-quality fusion. We conduct experiments in subject-driven generation tasks showing that our technique to merge two $\mathcal{GS}$ orthogonal
matrices is capable of uniting concept and style features of different adapters. To the best of our knowledge, this is the first training-free method for merging multiplicative orthogonal adapters. Code is available via the $\href{https://github.com/ControlGenAI/OrthoFuse}{link}$.
Read the full abstract ▾
All authors Ali Aliev, Kamil Garifullin, Nikolay Yudin, Vera Soboleva, Alexander Molozhavenko, Ivan Oseledets, Aibek Alanov, Maxim Rakhuba
Computer Vision
LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows
Zhengqin Li, Cheng Zhang, Jakob Engel et al. (+1 more)
We introduce the Large Sparse Reconstruction Model to study how scaling
transformer context windows affects feed-forward 3D reconstruction. Although recent object-centric feed-forward methods produce robust, high-quality reconstructions, they still lag behind dense-view optimization in recovering fine-grained texture and appearance. We show that expanding the context window – by substantially increasing the number of active object and image
tokens – narrows this gap and enables high-fidelity 3D object reconstruction and inverse rendering. To scale effectively, we adapt native sparse
attention for 3D reconstruction with three key contributions: (1) an efficient coarse-to-fine pipeline that focuses computation on informative regions by predicting sparse high-resolution residuals; (2) a 3D-aware spatial routing mechanism that establishes accurate 2D-3D correspondences using explicit
geometric distances rather than standard
attention scores; and (3) a custom block-aware sequence-parallel strategy with an All-gather-KV protocol to balance dynamic, sparse workloads across GPUs. As a result, LSRM handles 20x more object
tokens and >2x more image
tokens than prior state-of-the-art (SOTA) methods. Extensive evaluations on standard novel-view synthesis
benchmarks show substantial gains over the current SOTA, yielding >2.4dB higher PSNR and >40% lower LPIPS. Furthermore, when extending LSRM to inverse rendering, qualitative and quantitative evaluations on widely used
benchmarks demonstrate consistent improvements in texture and geometry details, achieving an LPIPS that matches or exceeds that of SOTA dense-view optimization methods. Code and model weights are available on our project page.
Read the full abstract ▾
All authors Zhengqin Li, Cheng Zhang, Jakob Engel, Zhao Dong
Computer Vision
MIRAGE: Benchmarking and Aligning Multi-Instance Image Editing
Ziqian Liu and Stephan Alaniz
Instruction-guided image editing has seen remarkable progress with models like FLUX.2 and Qwen-Image-Edit, yet they still struggle with complex scenarios with multiple similar instances each requiring individual edits. We observe that state-of-the-art models suffer from severe over-editing and spatial misalignment when faced with multiple identical instances and composite instructions. To this end, we introduce a comprehensive
benchmark specifically designed to evaluate fine-grained consistency in multi-instance and multi-instruction settings. To address the failures of existing methods observed in our
benchmark, we propose Multi-Instance Regional
Alignment via Guided Editing (MIRAGE), a training-free framework that enables precise, localized editing. By leveraging a
vision-
language model to parse complex instructions into regional subsets, MIRAGE employs a multi-branch parallel denoising strategy. This approach injects latent representations of target regions into the global representation space while maintaining background integrity through a reference trajectory. Extensive evaluations on MIRA-Bench and RefEdit-Bench demonstrate that our framework significantly outperforms existing methods in achieving precise instance-level modifications while preserving background consistency. Our
benchmark and code are available at https://github.com/ZiqianLiu666/MIRAGE.
Read the full abstract ▾
Computer Vision
Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI
Mingjie Li, Edward Kim, Yue Zhao et al. (+2 more)
Learning a robust Variational Autoencoder (VAE) is a fundamental step for many deep learning applications in medical image analysis, such as MRI synthesizes. Existing
brain VAEs predominantly focus on single-modality data (i.e., T1-weighted MRI), overlooking the complementary diagnostic value of other modalities like T2-weighted MRIs. Here, we propose a modality-aware and anatomically grounded 3D vector-quantized VAE (VQ-VAE) for reconstructing multi-modal
brain MRIs. Called NeuroQuant, it first learns a shared latent representation across modalities using factorized multi-axis
attention, which can capture relationships between distant
brain regions. It then employs a dual-stream 3D encoder that explicitly separates the encoding of modality-invariant anatomical structures from modality-dependent appearance. Next, the anatomical encoding is discretized using a shared codebook and combined with modality-specific appearance features via Feature-wise Linear Modulation (FiLM) during the decoding phase. This entire approach is trained using a joint 2D/3D strategy in order to account for the slice-based acquisition of 3D MRI data. Extensive experiments on two multi-modal
brain MRI
datasets demonstrate that NeuroQuant achieves superior reconstruction fidelity compared to existing VAEs, enabling a scalable foundation for downstream generative modeling and cross-modal
brain image analysis.
Read the full abstract ▾
All authors Mingjie Li, Edward Kim, Yue Zhao, Ehsan Adeli, Kilian M. Pohl
Computer Vision
Lightweight True In-Pixel Encryption with FeFET Enabled Pixel Design for Secure Imaging
Md Rahatul Islam Udoy, Diego Ferrer, Wantong Li et al. (+3 more)
Ensuring end-to-end security in image sensors has become essential as visual data can be exposed through multiple stages of the imaging pipeline. Advanced protection requires encryption to occur before pixel values appear on any readout lines. This work introduces a secure pixel sensor (SecurePix), a compact CMOS-compatible pixel architecture that performs true in-pixel encryption using a symmetric key realized through programmable, non-volatile multidomain polarization states of a ferroelectric field-effect transistor. The pixel and array operations are designed and simulated in HSPICE, while a 45 nm CMOS process design kit is used for layout drawing. The resulting layout confirms a pixel pitch of 2.33 x 3.01 um^2. Each pixel's non-volatile programming level defines its analog transfer characteristic, enabling the photodiode voltage to be converted into an encrypted analog output within the pixel. Full-image evaluation shows that ResNet-18 recognition accuracy drops from 99.29 percent to 9.58 percent on MNIST and from 91.33 percent to 6.98 percent on CIFAR-10 after encryption, indicating strong resistance to neural-network-based inference. Lookup-table-based inverse mapping enables recovery for authorized receivers using the same symmetric key. Based on HSPICE simulation, the SecurePix achieves a per-pixel programming power-delay product of 17 uW us and a per-pixel sensing power-delay product of 1.25 uW us, demonstrating low-overhead hardware-level protection.
Read the full abstract ▾
All authors Md Rahatul Islam Udoy, Diego Ferrer, Wantong Li, Kai Ni, Sumeet Kumar Gupta, Ahmedullah Aziz
Computer Vision
Watch Before You Answer: Learning from Visually Grounded Post-Training
Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang et al. (+8 more)
It is critical for
vision-
language models (VLMs) to comprehensively understand visual, temporal, and textual cues. However, despite rapid progress in
multimodal modeling,
video understanding performance still lags behind text-based
reasoning. In this work, we find that progress is even worse than previously assumed: commonly reported long
video understanding
benchmarks contain 40-60% of questions that can be answered using text cues alone. Furthermore, we find that these issues are also pervasive in widely used post-training
datasets, potentially undercutting the ability of post-training to improve VLM
video understanding performance. Guided by this observation, we introduce VidGround as a simple yet effective solution: using only the actual visually grounded questions without any linguistic biases for post-training. When used in tandem with RL-based post-training
algorithms, this simple technique improves performance by up to 6.2 points relative to using the full
dataset, while using only 69.1% of the original post-training data. Moreover, we show that data curation with a simple post-training
algorithm outperforms several more complex post-training techniques, highlighting that data quality is a major bottleneck for improving
video understanding in VLMs. These results underscore the importance of curating post-training data and evaluation
benchmarks that truly require visual grounding to advance the development of more capable VLMs. Project page: http://vidground.etuagi.com.
Read the full abstract ▾
All authors Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang, Penghui Du, Yiming Jia, Dongfu Jiang, Xuan He, Shenhui Zhang, Ping Nie, Peter West, Kelsey R. Allen
Computer Vision
Simultaneous Dual-View Mammogram Synthesis Using Denoising Diffusion Probabilistic Models
Jorge Alberto Garza-Abdala, Gerardo A. Fumagal-González, Eduardo de Avila-Armenta et al. (+5 more)
Breast cancer screening relies heavily on mammography, where the craniocaudal (CC) and mediolateral oblique (MLO) views provide complementary information for diagnosis. However, many
datasets lack complete paired views, limiting the development of
algorithms that depend on cross-view consistency. To address this gap, we propose a three-channel denoising
diffusion probabilistic model capable of simultaneously generating CC and MLO views of a single breast. In this configuration, the two mammographic views are stored in separate channels, while a third channel encodes their absolute difference to guide the model toward learning coherent anatomical relationships between projections. A pretrained DDPM from Hugging Face was fine-tuned on a private screening
dataset and used to synthesize dual-view pairs. Evaluation included
geometric consistency via automated breast mask
segmentation and distributional comparison with real images, along with qualitative inspection of cross-view
alignment. The results show that the difference-based encoding helps preserve the global breast structure across views, producing synthetic CC-MLO pairs that resemble real acquisitions. This work demonstrates the feasibility of simultaneous dual-view mammogram synthesis using a difference-guided DDPM, highlighting its potential for
dataset augmentation and future cross-view-aware AI applications in breast imaging.
Read the full abstract ▾
All authors Jorge Alberto Garza-Abdala, Gerardo A. Fumagal-González, Eduardo de Avila-Armenta, Sadam Hussain, Jasiel H. Toscano-Martínezb, Diana S. M. Rosales Gurmendi, Alma A. Pedro-Pérez, Jose G. Tamez-Pena
Computer Vision
SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaboration
Zhongyu Yang, Zuhao Yang, Shuo Zhan et al. (+3 more)
Video question answering (VideoQA) is a challenging task that requires integrating spatial, temporal, and semantic information to capture the complex dynamics of
video sequences. Although recent advances have introduced various approaches for
video understanding, most existing methods still rely on locating relevant frames to answer questions rather than
reasoning through the evolving storyline as humans do. Humans naturally interpret
videos through coherent storylines, an ability that is crucial for making robust and contextually grounded predictions. To address this gap, we propose SVAgent, a storyline-guided cross-modal multi-
agent framework for VideoQA. The storyline
agent progressively constructs a narrative representation based on frames suggested by a refinement suggestion
agent that analyzes historical failures. In addition, cross-modal decision
agents independently predict answers from visual and textual modalities under the guidance of the evolving storyline. Their outputs are then evaluated by a meta-
agent to align cross-modal predictions and enhance
reasoning robustness and answer consistency. Experimental results demonstrate that SVAgent achieves superior performance and interpretability by emulating human-like storyline
reasoning in
video understanding.
Read the full abstract ▾
All authors Zhongyu Yang, Zuhao Yang, Shuo Zhan, Tan Yue, Wei Pang, Yingfang Yuan
Computer Vision · Cross-listed from cs.AI
Part-Level 3D Gaussian Vehicle Generation with Joint and Hinge Axis Estimation
Shiyao Qian, Yuan Ren, Dongfeng Bai et al. (+1 more)
Simulation is essential for autonomous driving, yet current frameworks often model vehicles as rigid assets and fail to capture part-level articulation. With perception
algorithms increasingly leveraging dynamics such as wheel steering or door opening, realistic simulation requires animatable vehicle representations. Existing CAD-based pipelines are limited by library coverage and fixed templates, preventing faithful reconstruction of in-the-wild instances. We propose a generative framework that, from a single image or sparse multi-view input, synthesizes an animatable 3D Gaussian vehicle. Our method addresses two challenges: (i) large 3D asset generators are optimized for static quality but not articulation, leading to distortions at part boundaries when animated; and (ii)
segmentation alone cannot provide the kinematic parameters required for motion. To overcome this, we introduce a part-edge refinement module that enforces exclusive Gaussian ownership and a kinematic
reasoning head that predicts joint positions and hinge axes of movable parts. Together, these components enable faithful part-aware simulation, bridging the gap between static generation and animatable vehicle models.
Read the full abstract ▾
All authors Shiyao Qian, Yuan Ren, Dongfeng Bai, Bingbing Liu
Computer Vision
R3PM-Net: Real-time, Robust, Real-world Point Matching Network
Yasaman Kashefbahrami, Erkut Akdag, Panagiotis Meletis et al. (+3 more)
Accurate Point Cloud Registration (PCR) is an important task in 3D data processing, involving the estimation of a rigid transformation between two point clouds. While deep-learning methods have addressed key limitations of traditional non-learning approaches, such as sensitivity to noise, outliers, occlusion, and initialization, they are developed and evaluated on clean, dense, synthetic
datasets (limiting their generalizability to real-world industrial scenarios). This paper introduces R3PM-Net, a lightweight, global-aware, object-level point matching network designed to bridge this gap by prioritizing both generalizability and real-time efficiency. To support this transition, two
datasets, Sioux-Cranfield and Sioux-Scans, are proposed. They provide an evaluation ground for registering imperfect photogrammetric and event-camera scans to digital CAD models, and have been made publicly available. Extensive experiments demonstrate that R3PM-Net achieves competitive accuracy with unmatched speed. On ModelNet40, it reaches a perfect fitness score of $1$ and inlier RMSE of $0.029$ cm in only $0.007$s, approximately 7 times faster than the state-of-the-art method RegTR. This performance carries over to the Sioux-Cranfield
dataset, maintaining a fitness of $1$ and inlier RMSE of $0.030$ cm with similarly low latency. Furthermore, on the highly challenging Sioux-Scans
dataset, R3PM-Net successfully resolves edge cases in under 50 ms. These results confirm that R3PM-Net offers a robust, high-speed solution for critical industrial applications, where precision and real-time performance are indispensable. The code and
datasets are available at https://github.com/YasiiKB/R3PM-Net.
Read the full abstract ▾
All authors Yasaman Kashefbahrami, Erkut Akdag, Panagiotis Meletis, Evgeniya Balmashnova, Dip Goswami, Egor Bondarau
Computer Vision
ID-Sim: An Identity-Focused Similarity Metric
Julia Chae, Nicholas Kolkin, Jui-Hsien Wang et al. (+3 more)
Humans have remarkable selective sensitivity to identities – easily distinguishing between highly similar identities, even across significantly different contexts such as diverse viewpoints or lighting.
Vision models have struggled to match this capability, and progress toward identity-focused tasks such as personalized image generation is slowed by a lack of identity-focused evaluation metrics. To help facilitate progress, we propose ID-Sim, a feed-forward metric designed to faithfully reflect human selective sensitivity. To build ID-Sim, we curate a high-quality training set of images spanning diverse real-world domains, augmented with generative synthetic data that provides controlled, fine-grained identity and contextual variations. We evaluate our metric on a new unified evaluation
benchmark for assessing consistency with human annotations across identity-focused recognition,
retrieval, and generative tasks.
Read the full abstract ▾
All authors Julia Chae, Nicholas Kolkin, Jui-Hsien Wang, Richard Zhang, Sara Beery, Cusuh Ham
Computer Vision
Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
Chaoyou Fu, Haozhi Yuan, Yuhao Dong et al. (+16 more)
With the rapid advancement of
video understanding, existing
benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce
Video-MME-v2, a comprehensive
benchmark designed to rigorously evaluate the robustness and faithfulness of
video understanding. To systematically evaluate model capabilities, we design a progressive tri-level hierarchy that incrementally increases the complexity of
video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex
multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a group-based non-linear evaluation strategy that enforces both consistency across related queries and coherence in multi-step
reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid
reasoning. To guarantee data quality,
Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by 3,300 human-hours and up to 5 rounds of quality assurance,
Video-MME-v2 aims to serve as one of the most authoritative
video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level
reasoning. We further find that thinking-based
reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations,
Video-MME-v2 establishes a demanding new testbed for the development of next-generation
video MLLMs.
Read the full abstract ▾
All authors Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang, Yunhang Shen, Xiaoxing Hu, Xueying Li, Jinsen Su, Chengwu Long, Xiaoyao Xie, Yongkang Xie, Xiawu Zheng, Xue Yang, Haoyu Cao, Yunsheng Wu, Ziwei Liu, Xing Sun, Caifeng Shan, Ran He
Computer Vision · Cross-listed from cs.RO
StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing
StarVLA Community
Building generalist
embodied agents requires integrating perception,
language understanding, and action, which are core capabilities addressed by
Vision-
Language-Action (VLA) approaches based on
multimodal foundation models, including recent advances in
vision-
language models and world models. Despite rapid progress, VLA methods remain fragmented across incompatible architectures, codebases, and evaluation protocols, hindering principled comparison and reproducibility. We present StarVLA, an open-source codebase for VLA research. StarVLA addresses these challenges in three aspects. First, it provides a modular backbone–action-head architecture that supports both VLM backbones (e.g., Qwen-VL) and world-model backbones (e.g., Cosmos) alongside representative action-decoding paradigms, all under a shared abstraction in which backbone and action head can each be swapped independently. Second, it provides reusable training strategies, including cross-embodiment learning and
multimodal co-training, that apply consistently across supported paradigms. Third, it integrates major
benchmarks, including LIBERO, SimplerEnv, RoboTwin~2.0, RoboCasa-GR1, and BEHAVIOR-1K, through a unified evaluation interface that supports both simulation and real-
robot deployment. StarVLA also ships simple, fully reproducible single-
benchmark training recipes that, despite minimal data engineering, already match or surpass prior methods on multiple
benchmarks with both VLM and world-model backbones. To our best knowledge, StarVLA is one of the most comprehensive open-source VLA frameworks available, and we expect it to lower the barrier for reproducing existing methods and prototyping new ones. StarVLA is being actively maintained and expanded; we will update this report as the project evolves. The code and documentation are available at https://github.com/starVLA/starVLA.
Read the full abstract ▾
Computer Vision
Vanast: Virtual Try-On with Human Image Animation via Synthetic Triplet Supervision
Hyunsoo Cha, Wonjung Woo, Byungjun Kim et al. (+1 more)
We present Vanast, a unified framework that generates garment-transferred human animation
videos directly from a single human image, garment images, and a pose guidance
video. Conventional two-stage pipelines treat image-based virtual try-on and pose-driven animation as separate processes, which often results in identity drift, garment distortion, and front-back inconsistency. Our model addresses these issues by performing the entire process in a single unified step to achieve coherent synthesis. To enable this setting, we construct large-scale triplet supervision. Our data generation pipeline includes generating identity-preserving human images in alternative outfits that differ from garment catalog images, capturing full upper and lower garment triplets to overcome the single-garment-posed
video pair limitation, and assembling diverse in-the-wild triplets without requiring garment catalog images. We further introduce a Dual Module architecture for
video diffusion transformers to stabilize training, preserve pretrained generative quality, and improve garment accuracy, pose adherence, and identity preservation while supporting zero-shot garment interpolation. Together, these contributions allow Vanast to produce high-fidelity, identity-consistent animation across a wide range of garment types.
Read the full abstract ▾
All authors Hyunsoo Cha, Wonjung Woo, Byungjun Kim, Hanbyul Joo
Computer Vision
PointTPA: Dynamic Network Parameter Adaptation for 3D Scene Understanding
Siyuan Liu, Chaoqun Zheng, Xin Zhou et al. (+3 more)
Scene-level point cloud understanding remains challenging due to diverse geometries, imbalanced category distributions, and highly varied spatial layouts. Existing methods improve object-level performance but rely on static network parameters during inference, limiting their adaptability to dynamic scene data. We propose PointTPA, a Test-time Parameter Adaptation framework that generates input-aware network parameters for scene-level point clouds. PointTPA adopts a Serialization-based Neighborhood Grouping (SNG) to form locally coherent patches and a Dynamic Parameter Projector (DPP) to produce patch-wise adaptive weights, enabling the backbone to adjust its behavior according to scene-specific variations while maintaining a low parameter overhead. Integrated into the PTv3 structure, PointTPA demonstrates strong parameter efficiency by introducing two lightweight modules of less than 2% of the backbone's parameters. Despite this minimal parameter overhead, PointTPA achieves 78.4% mIoU on ScanNet validation, surpassing existing parameter-efficient fine-tuning (PEFT) methods across multiple
benchmarks, highlighting the efficacy of our test-time dynamic network parameter adaptation mechanism in enhancing 3D scene understanding. The code is available at https://github.com/H-EmbodVis/PointTPA.
Read the full abstract ▾
All authors Siyuan Liu, Chaoqun Zheng, Xin Zhou, Tianrui Feng, Dingkang Liang, Xiang Bai
Computer Vision
LoMa: Local Feature Matching Revisited
David Nordström, Johan Edstedt, Georg Bökman et al. (+6 more)
Local feature matching has long been a fundamental component of 3D
vision systems such as Structure-from-Motion (SfM), yet progress has lagged behind the rapid advances of modern data-driven approaches. The newer approaches, such as feed-forward reconstruction models, have benefited extensively from scaling
dataset sizes, whereas local feature matching models are still only trained on a few mid-sized
datasets. In this paper, we revisit local feature matching from a data-driven perspective. In our approach, which we call LoMa, we combine large and diverse data mixtures, modern training recipes, scaled model capacity, and scaled compute, resulting in remarkable gains in performance. Since current standard
benchmarks mainly rely on collecting sparse views from successful 3D reconstructions, the evaluation of progress in feature matching has been limited to relatively easy image pairs. To address the resulting saturation of
benchmarks, we collect 1000 highly challenging image pairs from internet data into a new
dataset called HardMatch. Ground truth correspondences for HardMatch are obtained via manual annotation by the authors. In our extensive benchmarking suite, we find that LoMa makes outstanding progress across the board, outperforming the state-of-the-art method ALIKED+LightGlue by +18.6 mAA on HardMatch, +29.5 mAA on WxBS, +21.4 (1m, 10$^\circ$) on InLoc, +24.2 AUC on RUBIK, and +12.4 mAA on IMC 2022. We release our code and models publicly at https://github.com/davnords/LoMa.
Read the full abstract ▾
All authors David Nordström, Johan Edstedt, Georg Bökman, Jonathan Astermark, Anders Heyden, Viktor Larsson, Mårten Wadenbäck, Michael Felsberg, Fredrik Kahl
Computer Vision
Rethinking Model Efficiency: Multi-Agent Inference with Large Models
Sixun Dong, Juhua Hu, Steven Li et al. (+2 more)
Most
vision-
language models (VLMs) apply a large
language model (LLM) as the decoder, where the response
tokens are generated sequentially through autoregression. Therefore, the number of output
tokens can be the bottleneck of the end-to-end latency. However, different models may require vastly different numbers of output
tokens to achieve comparable performance. In this work, we conduct a comprehensive analysis of the latency across different components of VLMs on simulated data. The experiment shows that a large model with fewer output
tokens can be more efficient than a small model with a long output sequence. The empirical study on diverse real-world
benchmarks confirms the observation that a large model can achieve better or comparable performance as a small model with significantly fewer output
tokens. To leverage the efficiency of large models, we propose a multi-
agent inference framework that keeps large models with short responses but transfers the key
reasoning tokens from the small model when necessary. The comparison on
benchmark tasks demonstrates that by reusing the
reasoning tokens from small models, it can help approach the performance of a large model with its own
reasoning, which confirms the effectiveness of our proposal.
Read the full abstract ▾
All authors Sixun Dong, Juhua Hu, Steven Li, Wei Wen, Qi Qian
Computer Vision
SimpleProc: Fully Procedural Synthetic Data from Simple Rules for Multi-View Stereo
Zeyu Ma, Alexander Raistrick and Jia Deng
Generating procedural synthetic data for multi-view stereo (MVS) usually requires writing complex rules to match the realism of curated
datasets. We demonstrate that we can generate effective training data using SimpleProc: a new, fully procedural generator driven by a very small set of rules based on Non-Uniform Rational Basis Splines (NURBS), as well as simple displacement and texture patterns. At a modest scale of 8,000 images, our approach achieves superior results compared to manually curated images at the same scale sourced from games and real-world objects. When scaled to 352,000 images, our approach achieves similar, and in some
benchmarks, even better results than the current state-of-the-art trained on over 692,000 manually curated images. The source code and the data are available at https://github.com/princeton-vl/SimpleProc.
Read the full abstract ▾
Computer Vision
Your Pre-trained Diffusion Model Secretly Knows Restoration
Sudarshan Rajagopalan and Vishal M. Patel
Pre-trained
diffusion models have enabled significant advancements in All-in-One Restoration (AiOR), offering improved perceptual quality and generalization. However,
diffusion-based restoration methods primarily rely on fine-tuning or Control-Net style modules to leverage the pre-trained
diffusion model's priors for AiOR. In this work, we show that these pre-trained
diffusion models inherently possess restoration behavior, which can be unlocked by directly learning
prompt embeddings at the output of the text encoder. Interestingly, this behavior is largely inaccessible through text
prompts and text-
token embedding optimization. Furthermore, we observe that naive
prompt learning is unstable because the forward noising process using degraded images is misaligned with the reverse sampling trajectory. To resolve this, we train
prompts within a
diffusion bridge formulation that aligns training and inference dynamics, enforcing a coherent denoising path from noisy degraded states to clean images. Building on these insights, we introduce our lightweight learned
prompts on the pre-trained WAN
video model and FLUX image models, converting them into high-performing restoration models. Extensive experiments demonstrate that our approach achieves competitive performance and generalization across diverse degradations, while avoiding fine-tuning and restoration-specific control modules.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
TriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Weian Mao, Xi Lin, Wei Huang et al. (+5 more)
Extended
reasoning in large
language models (LLMs) creates severe KV cache
memory bottlenecks. Leading KV cache compression methods estimate KV importance using
attention scores from recent post-RoPE queries. However, queries rotate with position during RoPE, making representative queries very few, leading to poor top-key selection and unstable
reasoning. To avoid this issue, we turn to the pre-RoPE space, where we observe that Q and K vectors are highly concentrated around fixed non-zero centers and remain stable across positions – Q/K concentration. We show that this concentration causes queries to preferentially attend to keys at specific distances (e.g., nearest keys), with the centers determining which distances are preferred via a trigonometric series. Based on this, we propose TriAttention to estimate key importance by leveraging these centers. Via the trigonometric series, we use the distance preference characterized by these centers to score keys according to their positions, and also leverage Q/K norms as an additional signal for importance estimation. On AIME25 with 32K-
token generation, TriAttention matches Full
Attention reasoning accuracy while achieving 2.5x higher throughput or 10.7x KV
memory reduction, whereas leading baselines achieve only about half the accuracy at the same efficiency. TriAttention enables OpenClaw deployment on a single consumer GPU, where long context would otherwise cause out-of-
memory with Full
Attention.
Read the full abstract ▾
All authors Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, Yukang Chen
Computer Vision
Vero: An Open RL Recipe for General Visual Reasoning
Gabriel Sarch, Linrong Cai, Qunzhong Wang et al. (+3 more)
What does it take to build a visual reasoner that works across charts, science, spatial understanding, and open-ended tasks? The strongest
vision-
language models (VLMs) suggest that broad visual
reasoning is within reach, yet their closed data and reinforcement learning (RL) pipelines make their gains difficult to study, reproduce, or extend. We introduce Vero, a family of fully open VLMs that match or exceed existing open-weight models across diverse visual
reasoning tasks. We scale RL data and rewards across six broad task categories, constructing Vero-600K, a 600K-sample
dataset from 59
datasets, and designing task-routed rewards that handle heterogeneous answers. Across VeroEval, our 30-
benchmark suite, Vero-600K outperforms existing RL
datasets under controlled comparisons. Applied to five starting models, Vero variants gain 2.9-5.4 points on average over their initial models. Notably, Vero-Qwen3I-8B, trained on the Instruct model, surpasses Qwen3-VL-8B-Thinking by 3.8 points on average without additional
distillation. Systematic ablations reveal that different task categories elicit distinct
reasoning patterns and that broad gains depend on learning them jointly rather than in isolation. All data, code, and models are publicly available.
Read the full abstract ▾
All authors Gabriel Sarch, Linrong Cai, Qunzhong Wang, Haoyang Wu, Danqi Chen, Zhuang Liu
Computer Vision
A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
Tommie Kerssies, Gabriele Berton, Ju He et al. (+5 more)
Anticipating diverse future states is a central challenge in
video world modeling. Discriminative world models produce a deterministic prediction that implicitly averages over possible futures, while existing generative world models remain computationally expensive. Recent work demonstrates that predicting the future in the feature space of a
vision foundation model (VFM), rather than a latent space optimized for pixel reconstruction, requires significantly fewer world model parameters. However, most such approaches remain discriminative. In this work, we introduce DeltaTok, a tokenizer that encodes the VFM feature difference between consecutive frames into a single continuous "delta"
token, and DeltaWorld, a generative world model operating on these
tokens to efficiently generate diverse plausible futures. Delta
tokens reduce
video from a three-dimensional spatio-temporal representation to a one-dimensional temporal sequence, for example yielding a 1,024x
token reduction with 512x512 frames. This compact representation enables tractable multi-hypothesis training, where many futures are generated in parallel and only the best is supervised. At inference, this leads to diverse predictions in a single forward pass. Experiments on dense
forecasting tasks demonstrate that DeltaWorld forecasts futures that more closely align with real-world outcomes, while having over 35x fewer parameters and using 2,000x fewer FLOPs than existing generative world models. Code and weights: https://deltatok.github.io.
Read the full abstract ▾
All authors Tommie Kerssies, Gabriele Berton, Ju He, Qihang Yu, Wufei Ma, Daan de Geus, Gijs Dubbelman, Liang-Chieh Chen
Computer Vision
SpatialEdit: Benchmarking Fine-Grained Image Spatial Editing
Yicheng Xiao, Wenhu Zhang, Lin Song et al. (+10 more)
Image spatial editing performs geometry-driven transformations, allowing precise control over object layout and camera viewpoints. Current models are insufficient for fine-grained spatial manipulations, motivating a dedicated assessment suite. Our contributions are listed: (i) We introduce SpatialEdit-Bench, a complete
benchmark that evaluates spatial editing by jointly measuring perceptual plausibility and
geometric fidelity via viewpoint reconstruction and framing analysis. (ii) To address the data bottleneck for scalable training, we construct SpatialEdit-500k, a synthetic
dataset generated with a controllable Blender pipeline that renders objects across diverse backgrounds and systematic camera trajectories, providing precise ground-truth transformations for both object- and camera-centric operations. (iii) Building on this data, we develop SpatialEdit-16B, a baseline model for fine-grained spatial editing. Our method achieves competitive performance on general editing while substantially outperforming prior methods on spatial manipulation tasks. All resources will be made public at https://github.com/EasonXiao-888/SpatialEdit.
Read the full abstract ▾
All authors Yicheng Xiao, Wenhu Zhang, Lin Song, Yukang Chen, Wenbo Li, Nan Jiang, Tianhe Ren, Haokun Lin, Wei Huang, Haoyang Huang, Xiu Li, Nan Duan, Xiaojuan Qi
Computer Vision
ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality
Dawar Khan, Alexandre Kouyoumdjian, Xinyu Liu et al. (+3 more)
We present ClickAIXR, a novel on-device framework for
multimodal vision-
language interaction with objects in extended reality (XR). Unlike prior systems that rely on cloud-based AI (e.g., ChatGPT) or gaze-based selection (e.g., GazePointAR), ClickAIXR integrates an on-device
vision-
language model (VLM) with a controller-based object selection paradigm, enabling users to precisely click on real-world objects in XR. Once selected, the object image is processed locally by the VLM to answer natural
language questions through both text and
speech. This object-centered interaction reduces ambiguity inherent in gaze- or voice-only interfaces and improves transparency by performing all inference on-device, addressing concerns around
privacy and latency. We implemented ClickAIXR in the Magic Leap SDK (C API) with ONNX-based local VLM inference. We conducted a user study comparing ClickAIXR with Gemini 2.5 Flash and ChatGPT 5, evaluating usability, trust, and user satisfaction. Results show that latency is moderate and user experience is acceptable. Our findings demonstrate the potential of click-based object selection combined with on-device AI to advance trustworthy,
privacy-preserving XR interactions. The source code and supplementary materials are available at: nanovis.org/ClickAIXR.html
Read the full abstract ▾
All authors Dawar Khan, Alexandre Kouyoumdjian, Xinyu Liu, Omar Mena, Dominik Engel, Ivan Viola
Computer Vision
FileGram: Grounding Agent Personalization in File-System Behavioral Traces
Shuai Liu, Shulin Tian, Kairui Hu et al. (+6 more)
Coworking AI
agents operating within local file systems are rapidly emerging as a paradigm in human-AI interaction; however, effective personalization remains limited by severe data constraints, as strict
privacy barriers and the difficulty of jointly collecting
multimodal real-world traces prevent scalable training and evaluation, and existing methods remain interaction-centric while overlooking dense behavioral traces in file-system operations; to address this gap, we propose FileGram, a comprehensive framework that grounds
agent memory and personalization in file-system behavioral traces, comprising three core components: (1) FileGramEngine, a scalable persona-driven data engine that simulates realistic workflows and generates fine-grained
multimodal action sequences at scale; (2) FileGramBench, a diagnostic
benchmark grounded in file-system behavioral traces for evaluating
memory systems on profile reconstruction, trace disentanglement, persona drift detection, and
multimodal grounding; and (3) FileGramOS, a bottom-up
memory architecture that builds user profiles directly from atomic actions and content deltas rather than dialogue summaries, encoding these traces into procedural, semantic, and episodic channels with query-time abstraction; extensive experiments show that FileGramBench remains challenging for state-of-the-art
memory systems and that FileGramEngine and FileGramOS are effective, and by open-sourcing the framework, we hope to support future research on personalized
memory-centric file-system
agents.
Read the full abstract ▾
All authors Shuai Liu, Shulin Tian, Kairui Hu, Yuhao Dong, Zhe Yang, Bo Li, Jingkang Yang, Chen Change Loy, Ziwei Liu
Computer Vision
HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes
Mauricio Soroco, Francesco Pittaluga, Zaid Tasneem et al. (+5 more)
Ensuring safety in autonomous driving requires scalable generation of realistic, controllable driving scenes beyond what real-world testing provides. Yet existing instruction guided image editors, trained on object-centric or artistic data, struggle with dense, safety-critical driving layouts. We propose HorizonWeaver, which tackles three fundamental challenges in driving scene editing: (1) multi-level granularity, requiring coherent object- and scene-level edits in dense environments; (2) rich high-level semantics, preserving diverse objects while following detailed instructions; and (3) ubiquitous domain shifts, handling changes in
climate, layout, and traffic across unseen environments. The core of HorizonWeaver is a set of complementary contributions across data, model, and training: (1) Data: Large-scale
dataset generation, where we build a paired real/synthetic
dataset from Boreas, nuScenes, and Argoverse2 to improve generalization; (2) Model:
Language-Guided Masks for fine-grained editing, where semantics-enriched masks and
prompts enable precise,
language-guided edits; and (3) Training: Content preservation and instruction
alignment, where joint losses enforce scene consistency and instruction fidelity. Together, HorizonWeaver provides a scalable framework for photorealistic, instruction-driven editing of complex driving scenes, collecting 255K images across 13 editing categories and outperforming prior methods in L1, CLIP, and DINO metrics, achieving +46.4% user preference and improving BEV
segmentation IoU by +33%. Project page: https://msoroco.github.io/horizonweaver/
Read the full abstract ▾
All authors Mauricio Soroco, Francesco Pittaluga, Zaid Tasneem, Abhishek Aich, Bingbing Zhuang, Wuyang Chen, Manmohan Chandraker, Ziyu Jiang